Models
Mixture of ExpertsMoE
Also known as: sparse expert models
A model design that contains many specialised sub-networks ("experts") but uses only a few for each input, giving large capacity at lower running cost.
Mixture of experts is a way to build a very large AI that is still efficient to run. Instead of one giant network where everything is used for every task, the model has many smaller "expert" sub-networks and a router that picks just the right few for each piece of input. It is like a big consultancy: you do not put every specialist on every question, you call in the two or three who are relevant.
A mixture-of-experts (MoE) model splits part of the network into many parallel "expert" sub-networks, plus a router that decides, for each token, which few experts to use. Only those are activated, so the model has a huge total parameter count but only runs a fraction of it per token.
The benefit
- Large capacity, lower cost: more total parameters (more knowledge/skill) without proportionally more computation at inference.
- This is why "total parameters" and "active parameters" can differ a lot for MoE models.
The trade-offs
Routing must be balanced so some experts are not overloaded and others idle; MoE models use more memory (all experts must be stored) and are trickier to train and serve than equally-active dense models.
Mixture-of-experts replaces dense feed-forward layers with a set of expert subnetworks and a gating function that routes each token to top-k experts (commonly k=1–2), yielding conditional computation: FLOPs per token scale with active rather than total parameters (Shazeer et al. 2017). This decouples capacity from per-token compute, enabling very large sparse transformers whose inference cost tracks the active subset.
Engineering concerns dominate: load-balancing losses prevent expert collapse; capacity factors and token-dropping handle routing imbalance; and all experts must reside in memory, so MoE trades compute for memory and communication (expert parallelism across devices). Behaviourally, "active parameters" is the honest indicator of per-inference cost, and headline total-parameter figures can overstate compute. MoE is now common at the frontier as a way to grow capacity economically, at the cost of serving complexity and memory footprint.