Checkpoint: Where the First Savings Actually Come From

DeepSeek-V3's first architectural choices now support a more useful judgment than “the model is sparse.” Conditional experts, MLA, MTP, and load balancing act on different bottlenecks, and their benefits are not interchangeable. The question here is what each mechanism saves, what it shifts elsewhere, and why the combined answer points directly to communication as the unresolved systems constraint. Seeing those relationships lets us evaluate the design as an operating model rather than as a list of innovations.

Three bottlenecks, three kinds of saving

The simplest way to connect the mechanisms is to ask what must happen repeatedly for a token to become the next output. DeepSeek-V3's routed feed-forward layers determine which learned transformations are active. MLA determines how much attention information remains available from earlier positions. MTP changes what the model is asked to represent during training and can later provide an additional proposal during decoding. These are three different locations in the computation: active arithmetic, persistent state, and training signal.

Conditional experts reduce the breadth of feed-forward work. DeepSeek-V3 has a shared expert and 256 routed experts in its MoE layers, but selects eight routed experts for a token. The total model can therefore contain a large pool of capacity without executing every routed transformation for every position. The saving is conditional: it exists when the selected path is smaller than the full pool and when the runtime can execute that path efficiently.

MLA addresses a different burden. Generation repeatedly reuses the weights, but it also carries a KV cache containing attention information from earlier positions. MLA stores a compressed key-value latent and a separate positional key rather than the expanded content keys and values. The saved resource is persistent memory per position. It does not reduce the number of expert transformations selected by the router, and it does not make the attention calculation disappear. It changes the size and form of the state that follows the generation loop.

MTP spends extra training structure to shape the main model. In DeepSeek-V3's one-depth configuration, the main model predicts the next token and an additional module predicts one more future token through a sequential causal chain. The extra module can be discarded at inference, so the matched ablation compares models with the same ordinary inference cost. The training benefit is therefore not the same kind of saving as sparse arithmetic or compressed cache state. It is an attempt to make each learned representation carry more useful predictive information.

These mechanisms can reinforce one another, but they cannot substitute for one another. A smaller active expert path does not reduce the cache held for a long context. A smaller cache does not balance expert traffic. A richer training target does not automatically make either the router or the attention state cheaper. The fact that all three appear in one model is evidence of a design that treats cost as a set of bottlenecks, not evidence that one optimization has made the others unnecessary.