How a Score Becomes Policy Pressure
This chapter examines the mechanism that connects DeepSeek-R1’s reward contract to changed model behavior: Group Relative Policy Optimization, or GRPO. The important question is not whether the algorithm can produce a score, but how it turns several scored attempts at one problem into pressure on the policy. Understanding that loop lets us see why R1 can explore without a human label for every reasoning step, and why the same loop still depends on reliable evaluators and enough model capacity.
A Group Is a Local Measuring Stick
Suppose DeepSeek-R1-Zero receives a difficult mathematics question. It does not produce one answer and receive a universal grade in isolation. The training loop samples a group of possible outputs from an older version of the policy. Each output is then checked by the relevant evaluator: an answer matcher for a mathematical result, a test environment for code, or another rule-based checker for an objective task.
The group matters because absolute reward is often hard to calibrate. A score of 1 may mean “correct,” while 0 means “incorrect,” but those values do not say how unusual a successful attempt was for this particular problem. If every sampled answer is wrong, there is no obvious good trajectory to imitate. If several answers are correct, their differences may matter less than the broader pattern they share. The group creates a local reference point: each output is judged partly by how it performed relative to the other attempts generated for the same question.
In the paper’s formulation, the reward for each output is centered by the group’s mean and scaled by the group’s standard deviation. The result is an advantage. An output above the group average receives a positive signal; an output below it receives a negative one. The term does not mean that the output is globally optimal. It means that, under the current policy and this question, it is a better or worse candidate than its peers.
This is a useful fit for reasoning tasks. A hard problem may have many bad routes and a few successful ones. The optimizer does not need an annotator to identify every point at which a response began to drift. It needs enough reliable endpoint feedback to make successful trajectories statistically more valuable than unsuccessful ones. Across repeated questions and updates, patterns associated with successful trajectories become more likely to recur.
But group comparison is not magic. It requires variation. If every sampled answer is effectively identical, the group supplies little information about which behavior is better. If the evaluator is wrong, the group can reliably elevate the wrong candidate. Relative scoring reduces the need for a perfectly calibrated absolute scale; it does not remove the need for correct judgments.
A small example makes the distinction concrete. Imagine four attempts at one problem, two of which pass the checker and two of which fail. The successful attempts sit above the group average and receive positive relative pressure; the failed attempts sit below it and receive negative pressure. Now imagine that the same policy produces four failures. An isolated binary reward still says that every answer is wrong, but group normalization cannot identify a successful pattern that is not present. The loop can encourage the policy to revisit a more promising region only when exploration, prior knowledge, or another part of the training distribution supplies candidates worth comparing.
The group also changes what “improvement” means from one question to the next. A difficult problem may produce a lower absolute success rate than an easy one, yet a rare correct attempt can still be the most informative output in its own group. Conversely, a response can be the best available attempt and still be wrong. GRPO therefore creates a relative learning signal, not a certificate of correctness beyond the verifier’s judgment. That distinction becomes important when the training set contains tasks of very different difficulty or when the evaluator provides only a coarse pass/fail result.
From Relative Scores to an Update
The advantage is only the middle of the loop. GRPO must still translate it into a change in the policy—the probability distribution that generates the next set of tokens.
A simplified view is this. The old policy generates the group. The current policy is then adjusted so that outputs with positive relative advantage become more likely and outputs with negative advantage become less likely. The adjustment is based on the ratio between the current policy’s probability for an output and the old policy’s probability for that same output. If the current policy moves too aggressively, a clipping rule limits how much that ratio can contribute to the update.
Clipping protects the update from turning one lucky or noisy batch into a drastic change. It does not prevent exploration. It limits the size of a single step so that the training process can keep learning from later evidence rather than being dominated by an early outlier. In an ordinary engineering system, this resembles a controller that allows a correction but caps the correction’s rate: the system can move toward a new operating point without treating one measurement as a reason to jump there immediately.
GRPO also includes a Kullback–Leibler, or KL, term that penalizes excessive divergence from a reference policy. KL divergence is a way to measure how different two probability distributions are. Here, the reference is a behavioral anchor. Without some anchor, repeated updates could push the policy into regions where its outputs become unstable or where the reward signal is being exploited in ways the designers did not intend.
The key is that neither clipping nor KL defines what a good solution is. The verifier and reward do that. These controls shape how quickly the policy is allowed to move toward the rewarded behavior. They are stability mechanisms, not substitutes for a trustworthy objective.
The sequence shows why the algorithm is more than “reward the answer.” The question is sampled repeatedly; the evaluator sits outside the policy; the group scorer converts raw outcomes into relative pressure; and the policy update is bounded by both clipping and a reference distribution. The state that persists across iterations is not a hand-written reasoning procedure. It is the policy’s changing tendency to generate some continuations rather than others.
Why GRPO Does Not Need a Separate Value Model
A conventional PPO setup usually trains an additional value model. Its role is to estimate the expected future reward from a partial state, which helps PPO calculate an advantage for each action. In language-model training, that value model is often comparable in size to the policy, so it creates substantial memory and computation demands.
Long reasoning makes the value prediction itself particularly difficult. A response may begin with an incorrect-looking approach and later repair it. It may state a tentative result, notice a contradiction, and revise the path. A value model that judges an early prefix must predict the final outcome even though the meaning of that prefix can change as the sequence develops. The paper argues that this is especially awkward when the useful signal is concentrated at the end of a long chain.
GRPO avoids the separate value model by using the group of completed outputs as its local baseline. It does not ask a critic to predict the final reward from every partial sequence. Instead, it compares the outcomes that the policy actually produced for the same question. This trades one kind of difficulty for another. Group sampling and evaluation can be expensive, and the method still needs enough varied outputs to create a meaningful comparison. But it removes a large auxiliary model and a difficult token-by-token prediction problem from the main training loop.
The distinction is practical rather than ideological. In the paper’s comparison on a smaller mathematics model, PPO with a default generalized-advantage-estimation setting performs considerably worse than GRPO. With careful tuning, PPO improves and approaches GRPO. The authors therefore present GRPO as a more practical alternative for their large-scale, long-chain setting, not as a theorem that PPO cannot work.
The choice also follows from the kind of reward available. When only the final answer is checked, a detailed value estimate for every intermediate token may be mostly speculation. A group-relative outcome signal is cruder, but it is tied to an observation the system can actually make: which completed attempts passed the verifier.
There is a subtle consequence for how credit is assigned. In a conventional short answer, a wrong final result often makes the failure easy to locate: the response ends with the wrong option. In a long reasoning sequence, the decisive mistake may be followed by several plausible paragraphs, or an early error may be corrected much later. A critic that tries to judge each prefix must guess whether the current state is a temporary detour or a dead end. GRPO postpones that judgment until the whole sampled output can be checked, then gives the sequence a relative signal. It sacrifices fine-grained diagnosis for a more dependable endpoint.
That trade-off also helps explain why the method can support revision-like behavior. A sequence that says, in effect, “this route has failed; try another one” is not rewarded because the word “try” is intrinsically valuable. It is rewarded only if the resulting output is more likely to pass the task’s evaluator. The update can then increase the probability of a family of continuations that includes reconsideration, verification, or alternative approaches. The algorithm does not need to understand those labels as concepts. It only needs repeated evidence that some completed patterns work better than others.
The cost is delayed and noisy feedback. A successful answer does not reveal which of its many tokens mattered, and a failed answer may contain useful partial work that the binary score does not preserve. The group comparison makes the signal usable at scale, but it does not make it maximally informative. This is one reason the paper treats process reward models and search methods as tempting alternatives, while also reporting that they introduce their own difficult evaluation and stability problems.
Stability Without Freezing Exploration
Long-chain training creates a tension. The model must move far enough from its starting behavior to discover a more effective strategy, but not so far that an imperfect reward signal or a transient batch takes control of the policy.
The reference policy handles part of this tension. DeepSeek-R1-Zero does not compare the current policy forever with the original base model. The paper reports periodically replacing the reference with the latest policy. A fixed reference would become increasingly distant as the policy learns, potentially making the constraint either too restrictive or poorly matched to the model’s current behavior. A refreshed reference preserves a moving notion of stability while leaving room for cumulative change.
This design helps explain how a model can gradually develop longer responses. A response-length increase is not automatically blocked by a constraint that punishes every departure from the initial policy. At the same time, the policy is not given unlimited freedom in a single update. The update controls allow exploration to accumulate over many steps rather than requiring one dramatic leap.
That balance is important for interpreting R1-Zero’s reported “self-evolution.” The training curve does not show a model suddenly deciding to reason in the abstract. It shows a sequence of constrained policy changes. Candidate outputs that solve more verifiable problems receive better relative treatment; the policy shifts; later groups contain different candidates; and the process repeats. If reflective or revision-like continuations improve the chance of a correct outcome, they can become more common without having been specified as a separate module.
The result remains conditional. GRPO cannot manufacture a reliable distinction when the task supplies none. It cannot make a small or poorly prepared base model use long sequences productively. It cannot tell whether a high-scoring response is genuinely robust if the test is incomplete. Its contribution is to make a particular feedback loop efficient enough to run at scale.
The Optimizer Is Not the Objective
It is tempting to describe GRPO as the source of R1’s reasoning ability. That reverses the dependency. GRPO determines how feedback changes the policy; the task distribution and evaluator determine what feedback means.
A highly optimized policy can still learn the wrong lesson if the reward is a proxy. A model trained on a weak code test suite may learn to pass the visible cases without handling the broader problem. A model rewarded for a preference model’s judgment may discover stylistic features that score well without improving the intended answer. A group-relative update can make this divergence more efficient because it repeatedly favors whatever the evaluator recognizes.
The algorithm also does not solve the difference between a correct result and a trustworthy explanation. It raises the probability of outputs associated with better evaluated outcomes. Whether the text between the prompt and the final answer is complete, faithful, or useful to a human reader requires additional standards. That is one reason DeepSeek-R1 later adds readability-oriented data and general preference rewards rather than relying on the R1-Zero loop alone.
The important conclusion is therefore two-sided. GRPO gives DeepSeek-R1 a practical way to turn outcome feedback into gradual policy pressure without a separate value model. It supports exploration of long, revisable trajectories while using clipping, KL control, and reference refreshes to keep the process stable. But those mechanisms only regulate movement toward a reward. They do not decide whether the destination represents the capability the designers actually wanted.
The next issue is consequently more fundamental than the choice between GRPO and PPO: what kinds of tasks provide feedback precise enough to make the update meaningful? DeepSeek-R1’s answer is to build its strongest reasoning gains around math, code, STEM, and logic—domains where answers can be matched, compiled, tested, or otherwise checked. That verifier architecture determines the shape of the behavior GRPO can select.