The Feedback Contract Behind DeepSeek-R1
DeepSeek-R1 and its predecessor, DeepSeek-R1-Zero, are a useful case for asking a precise question about artificial reasoning: can a model learn better ways to solve problems without being shown a complete script for how to reason? This chapter introduces the two models, the feedback contract that connects them to reinforcement learning, and the limits of what their visible reasoning traces can prove. The aim is to give the reader a working model for judging when rewards can shape useful reasoning behavior—and when they merely reward a proxy.
Two Models, One Training Question
DeepSeek-R1-Zero is the cleaner experiment. It starts from DeepSeek-V3-Base and goes directly into reinforcement learning, without a supervised fine-tuning stage containing human-written reasoning examples. DeepSeek-R1 is the more usable successor. It retains the reasoning capability developed through the zero variant, then adds conversational “cold-start” data, rejection sampling, supervised fine-tuning, and further reinforcement learning on reasoning and general-purpose tasks.
That distinction matters because the models answer different parts of the same question. R1-Zero asks whether a capable base model can discover effective solution patterns when the training system supplies problems and outcome feedback, rather than demonstrations of every desired step. R1 asks how to keep that discovery while repairing the behaviors that make a raw reasoning model difficult to use: poor readability, mixed languages, weak open-domain answers, and limited general writing.
The paper positions this design against a familiar post-training sequence: first teach a model with supervised examples, then optimize it with reinforcement learning. Supervised fine-tuning is valuable because it gives the model explicit input-output examples. But the authors argue that human reasoning traces can also become a ceiling. They may omit reflection or verification, carry human preferences about how a solution should look, or simply be less effective than a strategy the model could discover for itself. R1-Zero therefore removes that initial reasoning SFT stage as an experiment in exploration.
This is not an argument that supervised learning is obsolete. The final R1 pipeline uses it extensively for human-readable reasoning, general questions, and tasks for which correctness is difficult to define. The real contrast is between two kinds of guidance: examples that prescribe a form of response, and feedback that selects among behaviors according to whether they work.
What the Reward Sees
A reward is not a complete definition of reasoning. It is an observation boundary. It tells the optimizer which aspects of an output are visible and valuable enough to improve.
For R1-Zero’s reasoning tasks, the main feedback is rule-based. In mathematics, the final answer can be matched against a reference answer. In programming, a compiler and test cases can evaluate whether a solution works. STEM and logic questions can often be checked against a correct option or answer. A separate format reward encourages the model to place its reasoning and final answer inside designated tags. Accuracy and format are combined, while the authors deliberately avoid neural reward models for this pure reasoning stage because learned evaluators can be exploited and are expensive to retrain.
This arrangement leaves an important part of the behavior unspecified. The verifier can see whether the answer is correct, whether a program passes its tests, or whether a response follows the required outer structure. It does not automatically establish that every intermediate statement is true, that the path was efficient, or that the visible chain of thought is a faithful report of an internal process. A model may reach a correct endpoint through a messy route. It may also learn to satisfy the format without acquiring the kind of reasoning a reader imagines from the tags.
The distinction is easiest to see in code. A test suite can establish that a program produces the right outputs on the cases it contains. It cannot, by itself, explain which reasoning steps led there or guarantee behavior on every untested case. The same logic applies to a mathematical answer checker: it verifies the endpoint more readily than the route. Outcome rewards are powerful partly because they avoid trying to score every step, but that convenience is also their boundary.
The diagram captures the central idea without implying that the verifier understands a mind. A capable base model produces candidate solutions; feedback changes which kinds of candidates become more likely. The behavior that emerges is constrained by what the evaluator can distinguish, by what the base model can already represent, and by the compute available to search and train. Safety controls belong on the boundary too: a reasoning capability that is useful in a benchmark is not automatically safe in an open deployment.
Why Behavior Can Change Without a Reasoning Script
Reinforcement learning gives the model repeated opportunities to try. If several candidate answers are generated for a problem, and some receive better outcomes than others, the training update can increase the likelihood of the higher-reward patterns. The system does not need a human label for every sentence if the final task supplies enough signal to distinguish success from failure.
That mechanism makes a particular kind of self-improvement possible. A base model already contains language, mathematics, code, and problem-solving patterns from pretraining. Reinforcement learning does not create those ingredients from nothing. It changes which paths the model tends to explore and retain when it is trying to solve a difficult, checkable problem. A longer attempt that verifies an intermediate result may survive more often than a short guess. A response that revisits a contradiction may have a better chance of reaching a correct endpoint. Over many updates, those differences can become more common even though nobody wrote a rule saying “now reflect.”
The paper reports this pattern in R1-Zero. On AIME 2024, its pass@1 accuracy rises from 15.6% at the start of training to 77.9%. With self-consistency over 16 samples, the reported accuracy reaches 86.7%. At the same time, the average response length grows, even though the training setup does not explicitly prescribe a target length. The authors also report more frequent use of words associated with reflection—such as “wait,” “mistake,” “verify,” and “check”—and a late increase in the word “wait.”
These observations support a restrained conclusion: the training process selected outputs that were longer, more accurate, and more likely to contain recognizable revision or verification patterns. They do not establish that the model has acquired human consciousness, an inner monologue, or a transparent account of its own computation. The paper itself makes this caution explicit when it describes first-person, vivid reasoning styles in the cold-start data as engineered heuristics rather than evidence of human-like intelligence.
The word “emergence” is therefore useful only if it is kept operational. It describes a behavior that was not directly specified in the reward rules but became more frequent during optimization. It does not mean that the behavior appeared without a substrate, without selection pressure, or without measurement choices. The base model supplied a space of possible continuations; the task and verifier changed which continuations were repeatedly reinforced.
The Conditions Behind the Claim
The phrase “reinforcement learning can create reasoning” is too broad for what this case supports. The paper’s own discussion identifies several prerequisites.
First, the base model must be capable enough to make long exploration productive. The authors report that early experiments with smaller 7-billion-parameter dense and 16-billion-parameter mixture-of-experts models tended to repeat themselves as responses grew longer. Those models did not effectively turn extra tokens into better AIME performance. More capable 32-billion-parameter, 230-billion-parameter, and 671-billion-parameter bases produced the substantial pure-RL gains reported in the paper. This is not a systematic scaling law, but it is a warning against treating RL as a substitute for a useful starting representation.
Second, the questions must be difficult enough to reward improvement and structured enough to be checked. A verifier that cannot distinguish a correct solution from a persuasive-looking one gives the optimizer little reason to learn the intended capability. This is why mathematics, algorithmic coding, STEM, and logic are central to R1-Zero, while open-ended writing remains harder to optimize directly.
Third, the system needs enough compute to sample alternatives, evaluate them, and update the policy across many iterations. Longer reasoning can improve difficult-task performance, but it also consumes training and inference resources. The paper’s reported behavior is not a free change in style; it is a change in how much computation the model is willing to spend before committing to an answer.
Finally, the evaluation must be interpreted with care. The reported results come from the paper’s benchmarks, prompts, sampling procedures, and decontamination process. The authors acknowledge that n-gram filtering cannot rule out paraphrased contamination on older tests. The evidence is strong enough to show a substantial capability change in the reported setting, but not to establish general intelligence or a universal recipe for arbitrary tasks.
The Question This Case Leaves Open
DeepSeek-R1-Zero makes the reward-centered idea visible, but it also exposes why DeepSeek-R1 needs more than pure RL. A model that learns to solve checkable problems can still be hard to read, mix languages, overthink easy questions, struggle with tools, or perform unevenly outside the domains that supplied reliable feedback. R1 addresses some of those problems by adding supervised examples and preference-oriented rewards. That repair does not invalidate the zero variant; it shows that exploration and usability are different objectives.
The central lesson is consequently narrower and more useful than “models learn to think.” DeepSeek-R1 shows that optimization can select more reflective and effective problem-solving behavior when a capable model receives hard tasks and feedback that reliably distinguishes better outcomes. The same process cannot see beyond its contract. If the evaluator measures only an endpoint, the path remains partly unknown. If the objective is subjective, a proxy can be hacked. If the model is deployed in a risky setting, benchmark success does not supply safety controls.
The next question is mechanical: how does R1 turn several sampled answers into a training signal without a separate label for every intermediate step? The answer lies in Group Relative Policy Optimization, which makes the feedback contract operational by comparing outputs against one another and using those comparisons to push the policy.