Overview

DeepSeek-V3 becomes a usable assistant through staged training, not pretraining alone. Its 14.8T-token corpus, fill-in-the-middle data, 4K pretraining, and two YaRN phases extend context to 32K and 128K before 1.5M instruction-tuning instances and reinforcement learning. This chapter explains how rule-based and model-based rewards, GRPO, and R1-derived reasoning distillation shape behavior, while longer reasoning can increase output cost. The reported long-context results and distillation gains remain tied to their tests, ablations, and data-generation pipeline.

Learning outcomes

DeepSeek-V3's 14.8T-token pretraining produces a 4K base model; 32K and 128K context behavior requires two additional YaRN phases.
Rule-based rewards suit externally verifiable tasks, while model-based rewards handle open-ended responses; both depend on the feedback they provide.
A DeepSeek-V2.5 ablation found reasoning distillation improved scores while lengthening responses, so capability gains can increase decoding work.

Keywords

DeepSeek-V3pretrainingYaRNGRPOdistillationcontext

From Training Tokens to a Usable Assistant

DeepSeek-V3 is not finished when pretraining stops. The model must first extract useful signal from its corpus, then acquire long-context behavior, instruction following, and reasoning patterns through additional stages. This chapter traces that lifecycle and asks what changes at each transition: what the model learns, what information or behavior is preserved, what new cost is paid, and where feedback becomes less certain. The result is a more complete view of efficiency, because a cheap base-model run is valuable only if it becomes a capable and usable system.

Signal first: pretraining and context extension

DeepSeek-V3's pretraining corpus contains 14.8 trillion tokens. The report says it increased the share of mathematics and programming data, expanded multilingual coverage beyond English and Chinese, and refined its processing pipeline to reduce redundancy while preserving diversity. These choices shape the value of the compute budget. A token is not useful merely because it occupies a position in a batch; its domain, language, repetition, and structure affect what the model can learn from processing it.

The data pipeline also includes fill-in-the-middle training. Using a Prefix-Suffix-Middle format, a document can present a prefix and suffix around a missing span and ask the model to generate the middle. DeepSeek-V3 applies this strategy at a rate of 0.1 during document-level pre-packing. The purpose is to add a code- and document-relevant prediction situation without abandoning ordinary next-token prediction. The report treats the strategy as compatible with that ordinary capability, so it becomes an additional way to use the same training tokens rather than a separate model objective that replaces the main one.

Document packing increases the amount of useful material placed into each training sequence, but it creates an interface question: should examples packed next to one another be allowed to attend across their boundaries? The report says it uses document packing without cross-sample attention masking. That is an exact implementation choice, not a reason to assume that every packed example behaves like an isolated document. It illustrates the broader lifecycle trade-off: improving hardware utilization can change the informational relationship among samples, so data layout is part of model behavior rather than an invisible preprocessing step.

Tokenization creates another boundary. DeepSeek-V3 uses byte-level BPE with a 128,000-token vocabulary. Its pretokenizer combines some punctuation and line breaks to improve multilingual compression efficiency. The authors note a possible token-boundary bias when a multiline prompt does not end with a terminal line break. To expose the model to more variants, they randomly split a proportion of those combined tokens during training. The intervention does not make tokenization neutral; it reduces one known mismatch between how text is presented during training and how some prompts appear during evaluation or use.

The pretraining schedule is similarly staged. The maximum sequence length is 4K. The learning rate warms up to 2.2 × 10^-4, remains constant until the model has consumed 10 trillion tokens, then follows a cosine decay across 4.3 trillion more tokens.

Published by DeepMox

FAQ

How does DeepSeek-V3 gain 128K context?

It extends the 4K pretraining context through two 1,000-step YaRN phases, first to 32K and then to 128K, with a separate context-extension budget.

How are DeepSeek-V3's reasoning and non-reasoning data prepared?

Reasoning data uses a DeepSeek-R1-derived expert pipeline, while creative, role-play, and straightforward data uses DeepSeek-V2.5 responses verified by human annotators.

What is the serving cost of longer reasoning in DeepSeek-V3?

Longer reasoning can improve reported ablation scores but requires more decoding steps; a DeepSeek-V2.5 ablation linked higher scores with longer average responses, so the extra quality is not free to serve.

Previous10 minDeepSeek-V3: How Selective FP8 Stays Stable

Next14 minDeepSeek-V3: What Its Cost Story Proves

DeepSeek-V3: From Pretraining to Usable Behavior

From Training Tokens to a Usable Assistant

Signal first: pretraining and context extension

FAQ