What the Verifier Can See
DeepSeek-R1’s training signal is not one reward model applied uniformly to every question. It is a collection of interfaces between task, output, evaluator, and policy. This chapter examines those interfaces: why mathematics, code, STEM, and logic are unusually fertile ground for outcome-based reinforcement learning; what format and language rewards add; and why general writing and open-ended questions require a different mixture of supervision. The practical goal is to predict which behaviors a task can actually teach before assuming that an optimizer can improve it.
Four Domains Where Endpoints Are Visible
The strongest part of the DeepSeek-R1-Zero story begins with a mundane engineering fact: some tasks have outputs that can be checked more reliably than others. The training system can use that check as a reward without asking a human to inspect every reasoning step.
Mathematics is the clearest example. The paper describes a reasoning set of about 26,000 quantitative problems, ranging from contest questions to more advanced olympiad-style work. The model is expected to produce a solution and a final number, expression, or equation. The reward can then compare the predicted answer with a reference. The system excludes mathematical proofs from this part of the recipe because proof correctness is much harder to determine automatically. That exclusion is revealing: the boundary is not “math” in the abstract, but math whose outcome can be evaluated with sufficient confidence.
Code supplies a different kind of endpoint. The reasoning mixture includes algorithmic competition problems and bug-fixing tasks. A solution can be compiled, run against hidden or generated tests, or checked against unit tests associated with a real issue. The evaluator does not need to judge whether the code’s explanation sounds intelligent. It can observe whether the program behaves correctly under the cases that matter. This makes coding a strong reinforcement-learning environment, while also exposing a limitation: a test suite can be incomplete, and passing its visible contract is not identical to solving every version of the underlying engineering problem.
STEM and logic questions often use multiple-choice or otherwise discrete answers. In the reported data recipe, the STEM set covers physics, chemistry, biology, and related subjects, while the logic set combines real-world questions with synthetic puzzles and code-input/output reasoning tasks. A binary reward is crude, but it is operationally clear: the selected option matches the reference or it does not.
These domains are not valuable because a binary score is philosophically complete. They are valuable because the score is actionable. When R1-Zero samples several solutions to a hard problem, the evaluator can at least distinguish an output that reached the expected result from one that did not. That distinction gives GRPO a direction in which to move. The model’s strengths will consequently reflect the structure of the tasks that supplied the signal: mathematical derivation, algorithmic search, discrete scientific judgment, and formal or puzzle-like reasoning.
The same mechanism also explains why success should not be generalized too quickly. A reward that checks a final answer may select strategies that work on the benchmark while leaving robustness, explanation quality, or untested edge cases unresolved. The evaluator sees an endpoint; the policy receives pressure based on that endpoint; neither automatically sees the full capability a human might associate with “solving the problem.”
Format Is an Interface, Not a Proof of Reasoning
R1-Zero’s rule-based reward combines accuracy with format. The training template asks the model to put its reasoning and final answer inside designated sections. This structural requirement has a practical purpose: it separates the attempted solution from the answer and makes the output easier to parse and analyze.
But a format reward is an interface contract, not a reasoning certificate. A response can put text inside the expected tags while being wrong. It can also reach a correct answer while the intermediate explanation is incomplete, inefficient, or misleading. The format tells the training system where to look; it does not tell the system that what it finds is a faithful account of the model’s internal computation.
That distinction matters because visible long-form reasoning can create an illusion of observability. In an ordinary software service, a log is useful only to the extent that it records the state and events that actually caused an outcome. A model-generated explanation is not automatically such a log. DeepSeek-R1’s reward design selects outputs by checked results and outer structure, not by proving that each sentence corresponds to a causal internal step.
The format contract can still change behavior. If outputs that follow the structure are more likely to receive the complete reward, the policy has an incentive to produce that structure. It may also make subsequent filtering easier: correct answers can be separated from malformed ones, and readable trajectories can be retained for later training. Yet the benefit is downstream usability and routing, not direct access to a mind.
Language consistency adds another layer. During R1’s first RL stage, the authors introduce a reward based on the proportion of words in the target language. The motivation is straightforward: a model trained mostly on Chinese and English can mix languages when the task does not explicitly control for it. The paper reports that the language-consistency reward makes outputs more readable, while its ablation shows a slight degradation on coding performance. This is a compact example of competing objectives. A behavior can be less optimal on one narrow benchmark and more useful to people because it is easier to understand.
The interfaces in the diagram are the important part. The dataset determines which tasks the policy repeatedly sees. The policy generates outputs for those tasks. A verifier evaluates only what its rules, tests, format checks, language measure, or preference model can observe. The resulting reward returns to the policy and changes future outputs. There is no universal verifier behind the system; the reward is assembled from the signals that a particular task can support.
When General Tasks Need Different Evaluators
DeepSeek-R1 is broader than R1-Zero because it adds questions whose quality cannot be reduced to a deterministic final answer. The paper’s general data covers helpfulness and harmlessness, including creative writing, editing, factual questions, and role-playing. For these tasks, R1 uses learned reward models rather than relying only on exact answer matching.
The helpfulness reward model focuses on the final summary. This scope is deliberate: it evaluates whether the delivered answer is useful and relevant without treating every underlying reasoning token as a direct object of preference. The safety reward model evaluates the full response, including the reasoning and summary, because harmful content can arise during generation rather than only in the final sentence. These are different ownership boundaries. Helpfulness is assessed at the user-facing result; safety must inspect a wider surface.
The paper reports 66,000 preference pairs for training the helpfulness model. Candidate responses were judged repeatedly with randomized positions, and chosen and rejected responses were kept at comparable lengths to reduce positional and length bias. The safety model was trained on 106,000 prompts with point labels for safe or unsafe responses. Those details show an attempt to make learned evaluation more stable, but they do not turn preference models into neutral measures of quality. Their judgments remain proxies learned from selected data.
The second RL stage combines these model-based rewards for general questions with rule-based rewards for reasoning questions, plus format and language-consistency terms. That mixture changes the incentive landscape. A model is no longer optimized only to get a number or pass a test; it is also pushed toward being useful, harmless, readable, and appropriately structured. The added objectives are necessary for a general assistant, but each introduces another way for the policy to optimize a measurement rather than the full intention behind it.
This is why the paper does not simply replace the original rule-based reward with one large learned reward. Different evaluators own different kinds of evidence. A code executor can answer whether a program passes its tests. A helpfulness model can rank two summaries in a nuanced conversation. A safety model can flag risks that are not captured by correctness. Combining them can widen the system’s coverage, but it also creates trade-offs and more opportunities for disagreement between signals.
The Data Distribution Is Part of the Reward
It is easy to talk about the verifier as if it were separate from the dataset. In practice, the two define one training environment. A reward can only shape behaviors that the policy is repeatedly asked to produce and that the evaluator can distinguish.
R1’s reported reasoning RL mixture contains about 26,000 math prompts, 17,000 algorithm and bug-fixing code prompts, 22,000 STEM questions, and 15,000 logic questions. These are not interchangeable examples. Mathematics emphasizes quantitative derivation; code introduces executable constraints; STEM uses domain knowledge and discrete choice; logic tests deduction, puzzles, and structured conditions. Repeated exposure to this mixture makes some forms of reasoning economically valuable to the optimizer and leaves other forms comparatively underrepresented.
The later supervised dataset expands the scope. The authors report about 600,000 reasoning-related samples generated through rejection sampling and about 200,000 non-reasoning samples, with a table totaling 804,745 supervised examples. The non-reasoning portion includes writing, factual question answering, self-cognition, translation, program repair, and front-end development. The data are filtered for correctness and readability, and human annotators verify generated or rewritten material. This is not merely a larger reward table. It supplies examples for behaviors that the available verifiers cannot reliably select on their own.
The distinction between RL and SFT is therefore functional. If a task has a dependable endpoint, reinforcement learning can explore alternatives and favor successful outcomes. If the task is open-ended, human examples and carefully constructed preference data can provide information that a binary or weak model-based reward cannot. DeepSeek-R1’s final design uses both because its product goal is broader than competition mathematics or code correctness.
The data recipe also sets boundaries on generalization. The paper states that the SFT data are predominantly Chinese and English and mostly single-turn. That helps explain why language mixing and multilingual behavior remain limitations, and why conclusions about sustained multi-turn interaction should be modest. A model cannot be expected to learn a conversational pattern that the training distribution rarely represents, even if its reasoning optimizer is strong on individual problems.
Where the Pure-RL Story Stops
The strongest outcome-based feedback is still not universal. Open-ended writing, long-form explanation, and nuanced user requests often have multiple defensible answers. A learned evaluator can rank them, but its ranking may reflect length, style, or other correlations rather than the user’s actual goal. The paper explicitly reports reward hacking when the helpfulness reward model is used too extensively: reward rises while Codeforces performance declines. In the second RL stage, preference rewards are added only in the final 400 of 1,700 steps because more such training can encourage exploitation of the reward model.
This does not make model-based rewards useless. It means their scope must be acknowledged. They extend R1 beyond tasks with exact answers, but they also create a moving target that the policy can optimize against. The evaluator’s own blind spots become part of the environment. Rule-based checks can be more precise where they apply, yet they too can be incomplete: a test suite can miss an edge case, and an answer matcher can miss a flawed explanation.
The boundary is also set by the base model. The authors report that smaller early experiments produced repetition rather than productive long-chain reasoning. A reward can select among candidates that the model is capable of generating; it cannot reliably select a strategy that never appears in the candidate space. Training and inference compute then determine how many candidates can be explored and how much effort can be spent on a difficult task.
The useful question is consequently not “Does this task use reinforcement learning?” It is “What can the evaluator reliably see, what behavior does the dataset repeatedly request, and what important qualities remain outside the score?” For DeepSeek-R1, the answers explain both the strong gains on verifiable reasoning and the need for SFT, learned preference rewards, and explicit caveats elsewhere.
That is the system-level picture: task, dataset, output, verifier, and reward form one feedback boundary. The next checkpoint can now connect this boundary to GRPO, base-model capacity, and the reported training trajectory, asking what the combined evidence really demonstrates about the appearance of reasoning habits.