Checkpoint: From Correct Answers to Reasoning Habits

DeepSeek-R1-Zero gives us three ideas that now need to be held together: GRPO turns group-scored outcomes into policy pressure, verifiers expose only selected parts of an answer, and a capable base model supplies the candidate behaviors that reinforcement learning can select. This checkpoint connects those ideas to the paper’s reported training trajectory. The result is a conditional model of “self-evolution”: what was measured, what was inferred, and what must be true before a reward can select a useful reasoning habit.

The Causal Chain Is Conditional

Start with the base checkpoint. DeepSeek-R1-Zero is not an empty learner asked to invent mathematics or programming from first principles. It begins with DeepSeek-V3-Base, a large language model already able to generate candidate solutions, language, code, and familiar problem-solving patterns. The authors’ account of reinforcement learning is therefore a selection story. The model supplies a space of possible continuations; the training loop changes which continuations become more probable.

The second link is the task distribution. The questions are hard enough that a short guess will often fail, but structured enough that a checker can identify at least some successful outcomes. Mathematics offers reference answers, code offers compilation and tests, and STEM and logic often offer discrete correct options. These tasks create repeated opportunities for the policy to produce multiple attempts and discover that some routes work better than others.

The third link is group-relative pressure. GRPO samples several outputs for a question, compares their rewards within the group, and increases the policy’s tendency toward outputs that performed relatively well. It does not require a human to mark every intermediate sentence. It does require the group to contain enough variation and the evaluator to distinguish success from failure with enough fidelity to make the comparison meaningful.

The fourth link is behavior selection. If a trajectory that pauses, checks a result, revises a contradiction, or tries an alternative route is more likely to pass the evaluator, the policy can gradually favor continuations with those properties. No explicit reward term needs to say “reflection is good” if reflection is part of a family of outputs that repeatedly produces better checked results.

This chain explains both the appeal and the restraint of the R1-Zero result. Reinforcement learning can select a behavior that was not directly demonstrated in a human-written trajectory. But it does so through existing model capacity, a task distribution, and a feedback boundary. The behavior is not unconstrained self-invention, and the optimizer is not discovering a general theory of reasoning detached from the problems it is asked to solve.

flowchart LR capacity["Capable base model"] --> tasks["Hard, verifiable tasks"] tasks --> candidates["Sampled solution candidates"] candidates --> rewards["Checked outcomes and format"] rewards --> relative["Group-relative policy pressure"] relative --> behavior["Selected reflection, verification, longer effort"] behavior -. "output trace is not proof" .-> interpretation["Cautious interpretation"]

The chain also clarifies where a failure can occur. A model may lack useful candidates.