Conditional Capacity: How DeepSeek-V3 Spends Parameters
DeepSeek-V3's 671 billion parameters do not all participate in every token computation. The question in this chapter is how the model turns that stored capacity into a conditional path: which experts receive a token, how their outputs are combined, and how the system prevents popular experts from becoming bottlenecks. Understanding that path makes the headline distinction between 671B total parameters and 37B activated parameters meaningful, while also showing why sparse computation creates communication and serving problems of its own.
Capacity that is present but conditional
A standard dense feed-forward layer applies the same large transformation to every token that reaches it. A Mixture-of-Experts, or MoE, layer stores several alternative transformations and uses a router to select a subset for each token. “Expert” does not mean a separate model with a human-assigned specialty. It means a learned feed-forward subnetwork whose parameters can become more useful for some patterns than others. The router makes the choice from the token's current hidden representation.
DeepSeek-V3 keeps the Transformer framework but changes most of its feed-forward layers. The first three layers use ordinary feed-forward networks; the remaining layers use MoE layers. Each of those MoE layers contains one shared expert and 256 routed experts. The shared expert processes every token through a common path. The router then chooses eight of the 256 routed experts for the token. Each routed expert has an intermediate hidden dimension of 2,048, so the layer contains a large pool of conditional capacity even though only a small portion of that pool is used for one token.
The distinction can be pictured as a library. The library may hold hundreds of specialized reference works, but a reader does not carry every book to a desk for every question. A selection system chooses a few books, while a common reference remains available to everyone. The analogy stops there: DeepSeek-V3's experts are not manually labeled, and the router is learned from data. The useful point is that resident capacity and per-token work are different quantities.
For each token, the router computes an affinity score for every routed expert using a sigmoid function. It selects the eight highest-scoring experts and normalizes their selected original scores into gating values. Those values weight the routed experts' outputs before the result is combined with the shared path and passed onward through the Transformer. In the report's configuration, this conditional design, together with 61 layers and a hidden dimension of 7,168, produces 671B total parameters and 37B activated parameters per token.
That accounting is powerful but limited. “Activated” refers to the parameters on the model's selected computational path; it does not mean that a deployment needs to store only 37B parameters. The routed experts must remain available somewhere, and the system must calculate affinities, move the token to the devices that host the selected experts, run the expert transformations, and bring the results back together. Attention and other parts of the Transformer also execute. The 37B figure is therefore best read as a statement about conditional arithmetic and capacity, not as a complete latency or cost estimate.
The conditional path is also repeated through depth, rather than chosen once for the whole model. The first three layers are dense, and each later MoE layer makes its own routing decision. A token that favors one group of experts at one layer may be sent to a different group at another layer because its hidden representation has changed. This gives the model more opportunities to use specialized transformations, but it also means that the routing fabric is part of the repeated forward pass. The exact active-parameter total is not obtained by taking eight out of 256 and declaring the model equivalent to a dense small model: shared experts, the dense layers, attention, and the rest of the Transformer still contribute work.
That repeated route creates a specific interface between learned behavior and physical execution. The router receives a hidden state and returns a sparse execution plan. The plan determines both which transformations are applied and which devices must participate. The dispatch system then has to preserve the token's identity while sending it to selected experts, and the combine step has to return weighted outputs to the continuing Transformer computation. A routing decision that looks sensible from the model's point of view can still be expensive if it scatters work across too many machines. Conversely, a communication-friendly restriction can reduce the choices available to the router. DeepSeek-V3's architecture is therefore not merely “many experts plus a selector”; it is a contract between representation and network.
The router's decision is constrained by geography. DeepSeek-V3 limits each token to at most four nodes. This node-limited routing keeps an individual token from creating an unrestricted cross-cluster search for experts. It reduces the communication surface that later runtime systems must handle, but it also restricts the set of experts that can be selected together. The model's architecture is already making a promise to the infrastructure: conditional capacity is useful only within a route the network can deliver.
The report says that DeepSeek-V3 does not drop tokens during training or inference. Token dropping is a common emergency response when an expert receives more work than its capacity can handle: excess tokens are discarded or sent through a fallback path. Avoiding it means that the balancing design must keep the selected experts within workable loads while preserving the intended computation. That is a stronger operational requirement than simply reporting a low average number of active parameters. It also creates a useful distinction between a route that is sparse and a route that is reliable. Sparse selection reduces the number of expert computations, but reliability requires enough capacity at the destinations for every selected token to complete its path.
The diagram separates two decisions that are easy to conflate. The biased score determines which experts are selected and where the token is sent. The original affinity determines how the selected expert outputs are weighted. That separation is the basis of DeepSeek-V3's approach to load balancing.
Balancing without forcing sameness
An MoE router has two competing objectives. It should select the experts that best fit a token, but it should not send so many tokens to the same experts that some devices sit idle while others become overloaded. Conventional MoE systems often add an auxiliary loss that rewards equal utilization. That loss can stabilize the hardware workload, yet a sufficiently large penalty changes the model's optimization target: the router is pushed toward balance even when unequal use would represent useful specialization. The DeepSeek-V3 report identifies this performance cost as the reason to seek another trade-off.
DeepSeek-V3 gives every routed expert a bias term used only during top-K selection. At the end of each training step, the system examines expert load across the whole batch. If an expert is overloaded, its bias is decreased; if it is underloaded, its bias is increased. On a later routing decision, the adjusted scores affect which experts enter the top eight. The update is a control loop around routing rather than a penalty added directly to the model's main learning objective.
The timing and scope of that loop are important. It observes the aggregate load produced by a training step and adjusts the next routing decisions; it is not a rule that can guarantee equal use for every individual sequence in advance. A batch can contain many kinds of tokens, so an expert's low share in one sequence may be balanced by its high share in another. The feedback controller is consequently trying to stabilize the workload at the scale where expert parallelism operates, while leaving more local variation available to the learned router. The report sets the bias update speed to 0.001 for the first 14.3 trillion tokens and then sets it to zero for the final 500 billion. Whatever the reason for that schedule, the described run does not keep changing this control parameter through its last phase.
The distinction between selection and contribution matters. Suppose an expert is attractive to many tokens because its learned transformation fits them well. Its bias can be lowered so that fewer tokens select it, relieving the computational hotspot. But for a token that still selects that expert, the gating value is derived from the original affinity rather than the adjusted score. The balancing mechanism changes access to the expert without directly rewriting the model's learned judgment about how strongly that expert should contribute. It is an attempt to protect the route without flattening the representation.
“Auxiliary-loss-free” should not be read as “balancing-free.” DeepSeek-V3 keeps a complementary sequence-wise balance loss with an extremely small weight. Its purpose is narrower: prevent an individual sequence from becoming extremely imbalanced even when batch-wide routing remains flexible. The reported settings make the distinction concrete. The bias update speed is 0.001 through the first 14.3 trillion training tokens and is set to zero during the final 500 billion; the sequence-wise balance factor is 0.0001. The large-scale balancing work is carried by the routing feedback, while the residual loss guards against a local failure mode.
This remaining loss is not a contradiction in the design; it marks two different failure surfaces. Batch-wide imbalance can waste the cluster by overloading some experts. Sequence-wise imbalance can make one particular example depend too heavily on a narrow part of the network, even when the overall batch looks healthy. DeepSeek-V3 gives the first problem to the bias controller and uses the small complementary term to discourage the second. The claim is therefore more precise as “auxiliary-loss-free primary balancing,” not the elimination of every balancing signal.
This scope difference explains why batch-wise balance can support specialization. If every sequence must use experts in nearly equal proportions, a sequence about mathematics and a sequence about code are both pushed toward the same local distribution. If balance is assessed across a large batch, the model has more freedom for different groups of tokens to favor different experts, provided the aggregate load remains manageable. The report's analysis of smaller models finds more visible domain specialization in the auxiliary-loss-free setup. That is an observation about the resulting routing patterns, not proof that every expert has one clean, human-readable role.
The ablations support the performance side of the trade-off, with auxiliary-loss-free models generally improving on most reported benchmarks compared with matched baselines that use purely auxiliary-loss-based balancing. But the scale and scope matter. The comparisons use 15.7B- and 228.7B-parameter MoE baselines and the authors' evaluation framework; they are not an independent replication of every condition in the 671B model. The report also notes that a batch-wise auxiliary loss can reach similar validation losses in smaller experiments. That suggests the advantage may come partly from relaxing sequence-wise constraints, not necessarily from the absence of a loss term as a magical property.
Flexibility creates new operating conditions. A large training batch can absorb local variation because an expert that is lightly used by one sequence may be heavily used by another. A small batch has less opportunity for that averaging. At inference time, the distribution of user prompts can shift away from the training distribution, producing a different expert-load pattern even if average training balance was excellent. The report addresses this serving problem with redundant-expert deployment: high-load experts can be duplicated and placed so that traffic is spread across devices. The architectural choice therefore continues into operations.
The full design is a chain of interfaces. The hidden state crosses into the router, where affinity and a routing-only bias determine a limited set of destinations. The token crosses the network to those experts, then the expert outputs cross back to a combine step. The balance controller owns neither the model's entire representation nor the expert computation; it adjusts the routing decision from observed batch load. The gating values remain part of the model's forward calculation. This division of responsibility lets DeepSeek-V3 pursue specialization without ignoring the physical workload that specialization creates.
DeepSeek-V3 thus spends parameters conditionally, but conditional does not mean costless. Its shared expert provides a common transformation, its routed experts provide a large pool of optional capacity, and its router selects eight experts under a four-node communication limit. The balancing loop tries to keep that selection useful to both the model and the cluster, while the tiny sequence-wise loss and later redundant serving strategy acknowledge the cases where batch-level control is not enough. Once the model has made this arithmetic conditional, attention must still retain a state for generation, and training must decide whether richer future-token supervision can make each active computation more valuable.