DeepSeek V4 carries roughly 1.6 trillion parameters and answers a typical query using a small fraction of them. MiniMax M2.7 holds 230 billion parameters and activates 10 billion, about 4.3 percent, per token. GLM-5.1 runs at 744 billion parameters total. Every one of these numbers only makes sense once you understand the mechanism that makes them possible: a router deciding, one token at a time, which small slice of a much larger network actually gets to do the work.
That mechanism is called Mixture of Experts, and it is now the default architecture for frontier language models. Almost every major release this site has covered in the last year, from DeepSeek’s V3 and V4 lines to MiniMax M2.7, GLM-5.1, and NVIDIA’s Nemotron 3, is a mixture-of-experts model at its core. What almost none of that coverage explains is how the routing itself actually works, why naive versions of it break in a specific, predictable way, and how the current generation of models fixed that problem.
The Basic Swap: One Big Layer Becomes Many Small Ones
A standard transformer layer runs every token through the same feedforward network. A mixture-of-experts layer replaces that single feedforward network with a bank of N smaller feedforward networks, called experts, plus a gating network that looks at each token and decides which experts should process it. Instead of running a token through all N experts, the gate selects the top K, usually a small number like one, two, or eight, and only those experts do any computation for that token. The other experts sit idle for that token entirely.
The gating math is simple by design. For an input token u, the router computes a score for every expert using a learned centroid vector per expert, passes those scores through a gating function, and keeps only the top K scores, zeroing out the rest. The token’s output is the weighted sum of just the selected experts’ outputs, weighted by their gate scores. Everything downstream of that selection, the actual expert computation, is completely ordinary feedforward math. The entire architectural idea lives in which tokens get sent where.
DeepSeek-V3’s own technical report makes a good concrete reference point because its configuration is fully documented. Each of its 61 transformer layers holds 256 routed experts plus 1 shared expert, and the router activates 8 of the 256 routed experts for every single token, alongside the shared expert that runs unconditionally. Out of 671 billion total parameters, roughly 37 billion, about 5.5 percent, actually do any work on a given token. Every other parameter in that layer sat idle for that specific token and will potentially activate on the next one.
This is why total parameter count and active parameter count diverge so sharply in MoE models. A model can hold a huge number of experts, giving it enormous total capacity to store specialized knowledge, while any single token only ever touches a small, fixed number of them. Total parameters set the memory footprint. Active parameters set the compute cost. MoE is the architectural trick that decouples those two numbers, which is exactly why every model chasing lower inference cost at high capability has converged on it.
Where the Idea Actually Came From
The modern version of this traces to a single 2017 paper: Noam Shazeer and coauthors’ “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.” That paper demonstrated MoE could scale a language model to 137 billion parameters at a time when that was an enormous number, using exactly the sparse top-K gating described above. It also documented, candidly and immediately, the problem that has driven most of the subsequent research: left alone, the router does not spread tokens evenly across experts. It collapses onto a small favored subset and starves the rest.
Why Routing Collapses Without Intervention
Expert collapse follows a rich-get-richer dynamic that falls directly out of how gradient descent trains the router and the experts together. An expert that gets picked slightly more often, whether from a lucky initialization or a slightly stronger early match to common inputs, receives more training signal. More training signal makes that expert marginally better at handling the inputs it already sees. A marginally better expert produces higher gating scores for those inputs, which makes the router pick it even more often next time. Nothing in the naive setup corrects this. Left to converge on its own, a router can end up sending the overwhelming majority of tokens to a handful of experts while the rest barely train at all, which defeats the entire point of having many experts and also creates a severe computational bottleneck when experts are spread across different physical devices, since an imbalanced load means some devices sit idle while others are overwhelmed.
The Classic Fix: An Auxiliary Loss
The standard solution, formalized in Google’s GShard and Switch Transformer papers, adds a second loss term during training whose only job is to punish imbalance. For a batch of tokens, the loss multiplies the fraction of tokens actually routed to each expert by that expert’s average gating probability across the batch, sums this product across all experts, and scales the result by a tunable weight. When routing is perfectly even, this term is minimized. When routing collapses onto a few experts, the term grows and pulls training back toward balance. The Switch Transformer paper pushed this to its logical extreme by setting K to one, routing every token to exactly one expert, which maximizes sparsity and keeps the balancing math as simple as possible.
The auxiliary loss works, but it is a blunt instrument. It is added directly to the main training objective, which means its gradient signal competes with the gradient signal actually trying to make the model better at language modeling. Set the balancing weight too low and collapse creeps back in. Set it too high and the model spends training capacity fighting its own router instead of learning the task, a tradeoff every team training a large MoE model has had to tune by hand.
The Modern Fix: Balance Without a Loss Function
DeepSeek-V3 introduced the approach now spreading across the field: loss-free balancing. Instead of adding a penalty term to the training objective, this method attaches a separate bias value to each expert’s routing score, applied after the gate computes its raw scores but before the top-K selection happens. That bias has nothing to do with the training loss and receives no gradient of its own. It updates through a simple, separate rule: after each training step, experts that received more than their fair share of tokens get their bias nudged down, and experts that received less get nudged up. The next step’s routing decision uses the updated bias, so the correction is continuous and automatic without ever touching the loss function the model is actually trying to optimize.
The practical result, reported in DeepSeek’s own research and replicated by independent teams training smaller models from scratch, is better validation loss than auxiliary-loss-controlled training at equivalent balance quality, because the router correction no longer introduces interference gradients into the main objective. This bias-based approach is what DeepSeek used for V3 and continues to use in V4’s architecture, and comparable strategies now show up in OLMoE and DeepSpeed-MoE. A related fix, router z-loss, targets a separate numerical problem: the softmax function inside the gate has a tendency to sharpen into extreme, near-one-hot distributions as training progresses, which independently pushes toward collapse even when load looks balanced on average. Router z-loss penalizes large router logits directly to keep that sharpening in check.
Shared Experts: The Other Half of the Modern Design
Most current frontier MoE models, including DeepSeek’s line, do not route every expert dynamically. They set aside one or more shared experts that process every token unconditionally, alongside the routed experts the gate selects per token. The intuition is straightforward: some knowledge, common grammar, frequent patterns, general world facts, is useful for essentially every token, and forcing the router to rediscover and re-select that knowledge for each input wastes capacity that could go toward genuine specialization in the routed experts. Shared experts absorb the general-purpose load so the routed experts can specialize more sharply, which is part of why fine-grained MoE designs, many small experts rather than a few large ones, have become common in the latest model generation. DeepSeek’s own ablation testing backs this up directly: disabling the shared expert causes a sharp drop in overall capability, while disabling even a small slice of the top routed experts, on the order of one sixteenth of them, produces a measurable performance hit as well. Both halves of the design are load-bearing, not redundant with each other.
What This Architecture Actually Costs
The active-parameter framing that dominates MoE marketing is accurate but incomplete, and the incompleteness matters for anyone actually deploying these models. Active parameters set the compute cost of a forward pass, which is genuinely where MoE saves money against a dense model of equivalent total capacity. They do not set the memory cost. Every expert, used or not on a given token, still has to sit somewhere in VRAM or system memory, because the router’s decision changes token to token and any expert might be needed on the next one. DeepSeek-V3’s 671 billion total parameters all have to be reachable somewhere across a serving cluster even though only 37 billion do any work per token, which is why serving these models remains a multi-GPU infrastructure problem even though the compute-per-token looks modest on paper. Quantization attacks this same memory problem from a completely different angle, shrinking every stored parameter rather than reducing how many activate, and the two techniques stack directly: a quantized MoE model gets both a smaller footprint per parameter and fewer parameters touched per token.
Distributed serving adds a second cost the active-parameter number hides entirely: communication. When experts live on different physical devices, which is unavoidable at the scale of a trillion-parameter model, each token’s routing decision has to actually move that token’s data to whichever device holds its selected experts and bring the result back. This all-to-all communication pattern is a real, measurable share of total inference latency, and it is the reason MoE serving frameworks spend as much engineering effort on expert placement and communication scheduling as on the math of the experts themselves.
What Happens Next
Loss-free balancing is still spreading rather than universally adopted, and teams training smaller MoE models from scratch are actively testing whether its benefits hold at scales far below the trillion-parameter frontier models that popularized it. The fine-grained expert trend, many small specialized experts instead of a handful of large ones, is still an open design question rather than a settled default, and it interacts with the communication cost problem in ways that are not fully worked out: more, smaller experts spread across more devices means more routing decisions crossing device boundaries, not fewer.
The routing decision itself also remains largely a black box. Researchers can measure which experts fire for which tokens after the fact, but predicting or explaining why the router made a specific choice, beyond describing the training dynamics that produced it, is still mostly unsolved. For an architecture that now underpins nearly every frontier model shipping in 2026, the actual decision process at the center of it is one of the least understood parts of how these systems work.
Leave a Reply