The State and Signal Problem: MLA and Multi-Token Prediction
DeepSeek-V3's sparse experts reduce how much feed-forward capacity a token activates, but they do not solve every cost of autoregressive generation. The model still has to remember enough of the preceding context to produce the next token, and it still has to learn from a training signal that normally focuses on only one token ahead. This chapter examines two different responses: Multi-head Latent Attention, which compresses the state retained during generation, and Multi-Token Prediction, which adds a structured future-token signal during training and can optionally assist speculative decoding. The distinction matters because one mechanism primarily changes memory, while the other primarily changes supervision.
MLA compresses what generation must remember
At each step of text generation, a language model uses the new token's representation as a query against representations of earlier tokens. Attention needs keys to decide which earlier positions matter and values to retrieve the corresponding information. Once a token has been processed, those keys and values can be retained so the model does not recompute the entire preceding context at every step. This retained collection is usually called the key-value, or KV, cache.
The cache is a different resource from the model's parameters. The weights can stay fixed and be reused, while the cached state grows as the conversation or document grows. A longer context means more earlier positions whose attention information must be available. The cache must also be held close enough to the computation, or moved efficiently enough, to meet a serving latency target. That is why a model with a modest active path can still face a serious memory problem during long generation.
DeepSeek-V3 uses Multi-head Latent Attention, or MLA, to change the representation stored for each position. Instead of treating every full key and value vector as the state that must be retained, MLA applies a low-rank joint compression to the keys and values. For a token, it first produces a compressed key-value latent vector. The attention computation can use that latent to reconstruct the content portions of the per-head keys and values when needed.
Position information needs separate treatment. DeepSeek-V3 keeps a decoupled key component that carries Rotary Positional Embedding, commonly abbreviated RoPE. The cached state therefore consists of the compressed key-value latent and this separate RoPE-bearing key component. The full up-projected content keys and values do not need to be retained in the cache in their expanded form. This is the central operational change: MLA reduces the representation that persists across generation while preserving a path for both content and position.
The published configuration makes the design concrete. MLA uses 128 attention heads with 128 dimensions per head. The key-value compression dimension is 512, the query compression dimension is 1,536, and the decoupled positional component has 64 dimensions per head. These dimensions are not a universal promise of a fixed cache multiplier; the actual memory behavior also depends on sequence length, batch size, data types, and the rest of the serving implementation. They do show that DeepSeek-V3 is deliberately moving attention state into a lower-dimensional latent space rather than merely hoping that sparse experts will solve long-context cost.
The query side has a different job. MLA also applies low-rank compression to queries, which reduces activation memory during training. That benefit should not be folded into the KV-cache claim. The KV path concerns the state retained for autoregressive inference; the query path concerns temporary information needed while training the model. Both use compression, but they protect different phases and different memory surfaces.
MLA also preserves a distinction between compression and quality. The report says that the reduced cache representation maintains performance comparable to standard multi-head attention. That is a first-party description of the design, not a reason to attach one exact savings percentage to every deployment. A serving system may be limited by attention arithmetic, expert communication, memory bandwidth, or batching before the theoretical cache reduction becomes the dominant gain. The safe conclusion is that MLA makes a large-context state cheaper to retain, not that it makes long-context inference cheap in every circumstance.
The interface is straightforward even if the underlying projections are mathematically elaborate. The current hidden state enters MLA. MLA produces a query for the current step and a compressed key-value representation for the new position, while the positional component is formed separately. The cache owns the retained latent and decoupled key. On a later step, MLA reads those compact representations, performs the attention calculation, and writes the state for the newly processed token. The cache persists across the generation loop; it is not part of the model's learned weights and it does not need to be rebuilt from scratch for every output token.
The low-rank representation changes where information is reconstructed. In ordinary multi-head attention, the cache can be understood as holding the per-head objects that later attention will read. In MLA, the cache holds a shared compressed description for the key/value content and a separate positional description; the attention computation expands or combines what it needs at the point of use. The model is not discarding the distinction between content and position. It is storing them in forms chosen for persistence, then paying the projection work when attention needs the expanded view.
That exchange is useful because persistent memory and temporary arithmetic have different scaling behavior. A representation held for every earlier position occupies cache capacity for the entire continuation. A projection performed during a step is temporary and can be scheduled with the rest of that step's computation. MLA therefore moves part of the burden from long-lived storage into on-demand computation. Whether that trade is favorable depends on the serving system: a memory-constrained deployment may value the smaller cache highly, while a compute-saturated deployment may see less benefit. The report establishes the intended cache reduction, not a single answer for every bottleneck.
The separate RoPE component also protects an important invariant. Attention needs to know not only what a prior token represented but where it occurred in the sequence. Combining positional information with the compressed content path without accounting for how it transforms could damage long-context behavior. DeepSeek-V3 instead keeps the positional key decoupled, so the cache reduction does not require treating location as ordinary content. This is why the mechanism is more than simply lowering the width of every attention vector.
The diagram also shows why MLA and MTP should not be described as one combined compression trick. MLA changes the state carried through the ordinary generation loop. MTP adds an optional participant after the main model has produced a representation; it can be absent from ordinary inference altogether.
MTP makes the training target look farther ahead
The usual language-model objective asks the system to predict the next token from the current context. That objective is powerful, but it delivers one immediate prediction at each position. DeepSeek-V3 investigates Multi-Token Prediction, or MTP, to extend the prediction scope to future tokens at each position. The intended benefit is not simply to make the model emit several tokens at once. A richer target can densify the training signal and encourage representations that remain useful for what comes next rather than only for the nearest prediction.
DeepSeek-V3's implementation predicts additional tokens sequentially. Its published depth is one: besides the exact next token predicted by the main model, each position predicts one additional token. The additional module keeps the causal chain intact. At the extra depth, it combines the representation produced at the preceding depth with the embedding of the future token and passes that combination through a Transformer block before using the shared output head to produce the next prediction. The result is a second prediction connected to the first, not a collection of independent guesses made from the same untouched representation.
During training, the target sequence supplies the future-token information needed to construct this additional prediction. The module can therefore be trained against a known next-next token while the main model remains responsible for the ordinary next-token path. The two losses receive related but not identical information: one asks for immediate correctness, and the other asks whether the representation can support a further step in the causal chain. At inference, the future token is not known in advance, which is why the MTP path changes from a supervised training branch into a proposal mechanism that the main model must verify.
Two forms of sharing keep the extra objective from becoming an entirely separate model. Each MTP module shares the embedding layer with the main model and shares the output head used to turn a representation into vocabulary probabilities. The module has its own transformation for the additional prediction, but its inputs and output interface remain tied to the main model. The shared components also support the memory-saving arrangement described in the report: the shallow embedding side and the deep output side can be placed so that the main model and MTP path physically share parameters and gradients during training.
The MTP losses are added to the main training objective with a changing weight. The report uses a weight of 0.3 for the first 10 trillion training tokens and 0.1 for the remaining 4.8 trillion. The schedule makes MTP an auxiliary training signal rather than a replacement for next-token prediction. The main objective remains responsible for the model's ordinary language-model path; the extra prediction asks the internal representation to support a slightly longer causal horizon.
That distinction is important when reading the ablation. The authors attach a one-depth MTP module to small and large MoE baselines while keeping the training data and other architecture the same. During inference, they discard the MTP module, so the compared models have the same inference cost. MTP improves performance on most of the reported benchmarks in those experiments. This supports the claim that the additional training target can improve the main model's learned representations even when the extra module is not carried into ordinary serving.
The reason this can happen is representational rather than magical. A next-token objective tells the model what must be correct immediately. The extra target supplies another constraint on the same evolving representation: it must preserve enough information to support the following step as well. That pressure may make some features more useful across a short causal chain. It does not guarantee better performance on every task, and the ablations are evidence from matched smaller-scale models rather than an independent decomposition of every gain in the full system.
MTP can also be reused at inference, but that is a separate operating mode. In speculative decoding, a smaller or auxiliary path proposes future tokens, and the main model verifies them. If the proposals are accepted, the service can advance more than one token's worth of output for roughly the cost of a verification step; if they are rejected, the main model remains the authority. DeepSeek-V3's MTP module can serve as the proposal mechanism, while the main model can be run normally when that extra complexity is not worthwhile.
The report gives two concrete observations about this optional path. The acceptance rate of the second-token prediction is reported at 85–90% across various generation topics, and the authors report a 1.8-times tokens-per-second decoding speed. Those figures make the idea operationally relevant, but they do not mean that every request gains 1.8 times the speed. Acceptance depends on the topic and serving workload, while throughput also depends on hardware, batching, verification strategy, and the cost of running the MTP module. The report does not specify one universal multiplier for all deployments.
Acceptance is the key mediator between a proposal and a realized speedup. A proposal that is usually correct can let the verifier commit more output per expensive main-model step. A proposal that is often wrong still incurs the cost of generating and checking it, while returning the system to the ordinary path. This is why the acceptance percentage and the tokens-per-second figure answer different questions: the first describes how often the extra prediction agrees with the verifier in the tested topics, while the second combines that behavior with a particular implementation and workload. The high reported acceptance rate is evidence that the second-token head can be useful; it is not a guarantee that adding the module will improve every service's latency.
The separation between training and inference is the core of the design. The MTP loss is paid during training to shape the main model. The MTP modules can then be discarded, leaving ordinary inference unchanged, or retained as an optional speculative tool. This gives DeepSeek-V3 a way to spend extra training computation for a possible representation benefit without requiring every user request to execute the extra path. The choice is not free in development: the additional modules, loss, and shared pipeline still consume training resources. But the matched ablation shows why their value cannot be judged only by counting the modules present at serving time.
MLA and MTP therefore answer different questions. MLA asks: what is the smallest faithful state that generation must carry from one position to the next? MTP asks: how can training make each representation useful for more than the immediately next target, and can that future prediction later help the decoder? The first targets persistence in inference. The second targets information density in training, with an optional latency benefit. Keeping those roles distinct prevents a common mistake: treating every efficiency technique as a reduction in active parameters.
DeepSeek-V3's cost model now has three separate locations. Conditional experts reduce the breadth of feed-forward computation for a token. MLA reduces the size of the attention state that persists across tokens. MTP increases the information requested from training and may reduce serving steps when its proposals are accepted. None of these changes erases the work around it. Routing still needs a network, compressed state still needs attention computation, and speculative proposals still need verification. The value of the design lies in making those trade-offs explicit rather than hiding them behind one parameter count.