Mixture of Experts (MoE)
A neural network architecture that routes each input to a subset of specialized sub-networks (experts), achieving the capacity of a very large model while only activating a fraction of parameters per inference.
Mixture of Experts is the architecture behind models like Mixtral and reportedly GPT-4. Instead of passing every input through all parameters, a gating network decides which expert sub-networks are most relevant for each token and routes the computation accordingly. A model with 8 experts might only activate 2 per token, meaning inference uses 25% of total parameters while benefiting from the full model's learned capacity.
The key advantage is the decoupling of model capacity from inference cost. A dense 70B model requires processing 70B parameters for every token. An MoE model with 8x14B experts (112B total parameters) but top-2 routing processes only 28B parameters per token while having access to 112B parameters' worth of knowledge. This makes MoE models significantly faster and cheaper to run than dense models of equivalent quality.
For production deployments, MoE models offer compelling economics. They achieve frontier-level quality at inference costs closer to much smaller models. The trade-off is memory: you still need to store all expert parameters in VRAM, even though only a subset is used per forward pass. This makes MoE models memory-intensive but compute-efficient, favoring deployment on systems with large memory but moderate compute.
Related Terms
RAG (Retrieval-Augmented Generation)
A technique that grounds LLM responses in external data by retrieving relevant documents at query time and injecting them into the prompt context.
Embeddings
Dense vector representations of text, images, or other data that capture semantic meaning in a high-dimensional space, enabling similarity search and clustering.
Vector Database
A specialized database optimized for storing, indexing, and querying high-dimensional vector embeddings with sub-millisecond similarity search.
LLM (Large Language Model)
A neural network trained on massive text corpora that can generate, understand, and transform natural language for tasks like summarization, classification, and conversation.
Fine-Tuning
The process of further training a pre-trained LLM on a domain-specific dataset to specialize its behavior, style, or knowledge for a particular task.
Prompt Engineering
The practice of designing and iterating on LLM input instructions to reliably produce desired outputs for a specific task.