From a Group of Answers to a Policy Update
For DAPO, the central engineering question is not simply how to reward a correct answer. It is how a group of sampled answers becomes a policy update without losing the information that makes the comparison useful. This chapter reconstructs that path—from PPO’s value estimate to GRPO’s group-relative signal and then to DAPO’s modified objective—so that later changes to exploration, sampling, token credit, and length reward have a concrete place to act.
Why GRPO replaces a value model with a group comparison
Proximal Policy Optimization, or PPO, needs an estimate of how valuable each partial sequence is. Its value model predicts expected future reward from the state represented by the tokens generated so far. PPO then uses that estimate, commonly through Generalized Advantage Estimation, to decide whether a token sequence performed better or worse than expected. This is a familiar division of labor: one model produces behavior, and another estimates the value of being in each intermediate state.
That architecture becomes awkward for long reasoning responses. In a math problem, the decisive signal may arrive only in the final answer. Early tokens can contain a false start that is later corrected, or a promising line that is abandoned. Predicting the final outcome from every partial prefix is difficult when the response is long enough to include reflection and revision. The value model also has a cost: at large scale, it is often comparable in size to the policy model, adding memory and computation to a training loop already dominated by rollout and forward passes.
Consider two partial responses that begin with the same algebraic mistake. In one, the model notices the error, backs up, and reaches the correct result. In the other, it repeats the mistake until the answer is wrong. Before the ending is available, those prefixes can look similar while their eventual returns differ. A value model must learn to forecast that difference from incomplete evidence. The longer the reasoning trace and the more often it revises itself, the more difficult that forecast becomes. GRPO avoids asking a separate model to make this prediction at every position; it waits for several complete attempts and lets their outcomes establish a local comparison. That saves a model-sized component, but it also means the comparison can fail when the sampled attempts are too similar.
GRPO takes a different route. For one question, the old policy samples a group of responses rather than asking a separate value model to judge every state. Each response receives a reward, and the group supplies its own baseline. If response i receives reward Rᵢ, its relative advantage is calculated from how far that reward lies above or below the group mean, scaled by the group’s standard deviation. A response that does better than its peers receives a positive direction; one that does worse receives a negative direction.
A small example makes the dependency visible. Suppose four attempts receive outcomes corresponding to correct, correct, incorrect, and incorrect. The group mean sits between the two outcomes, so the correct attempts receive positive relative advantages and the incorrect attempts receive negative ones. The update can reinforce the token patterns that appeared in the successful attempts and reduce the probability of patterns associated with the unsuccessful ones, although it cannot know that every token in a successful trace was necessary. Now change the group so that all four attempts are correct. The verifier is still doing its job, but the group no longer says which successful variation was better than another. Normalization removes the common level, leaving little directional information. Group-relative RL therefore needs not only reliable labels but also the right distribution of labels within each local comparison.
This is a useful exchange. GRPO avoids the value model and uses information that is naturally available when several attempts answer the same question. But it makes group composition part of the algorithm’s state. The same response can carry a different update direction depending on what else was sampled. A group containing both correct and incorrect answers has contrast. A group in which every answer receives the same outcome does not. The reward may be perfectly accurate and still fail to produce a useful relative signal if there is no variation to normalize.
The distinction is narrower than “PPO is old and GRPO is new.” PPO and GRPO make different bets about where advantage information should come from. PPO invests in a learned value estimate for intermediate states. GRPO invests in repeated responses to the same question. DAPO keeps the second bet because it is practical for large language models, then adds controls for the ways group-relative learning can degrade in long-chain-of-thought training.
How DAPO turns an outcome into a controlled update
The DAPO loop begins with a question and an old policy. The policy samples several responses for that question. A verifier evaluates the final answer of each response. In the mathematical setting described by the paper, an equivalent answer receives a positive correctness reward and an incorrect one a negative reward. These outcomes are normalized within the group, producing the relative advantages that tell the policy which sampled behavior should become more or less likely.
The policy does not simply copy the best response. For each generated token, DAPO compares the token’s probability under the current policy with its probability under the old policy that produced the sample. That ratio measures how far the update is trying to move from the behavior used for rollout. Clipping limits the contribution of ratios that move too far in either direction. The purpose is not to freeze the policy; it is to keep one update from making the sampled evidence unusable before the next round.
The sign of the advantage determines what the ratio is trying to do. With a positive advantage, the update wants the current policy to assign more probability to the token sequence; with a negative advantage, it wants less. A clipped surrogate objective says that once the intended movement has reached a boundary, pushing farther should no longer increase that token's contribution to the objective. This protects against a large update being driven by a small batch of sampled outcomes. But a single boundary can have different practical effects at different starting probabilities. Increasing a token from a very small probability may require a substantial relative change before it becomes a meaningful alternative, while a common token can reach a high absolute probability with a similar ratio limit. DAPO's later asymmetric clipping is therefore a change to the geometry of this trust region, not a removal of the idea of a trust region.
DAPO changes the shape of that control in four connected ways. First, it decouples the lower and upper clipping ranges. The lower side can continue to suppress a token when the evidence is negative, while the upper side can leave more room for a rare token whose probability should increase. The exact reason for that asymmetry becomes clearer when examining entropy collapse, but the important point here is ownership: clipping controls probability movement, not whether a prompt enters the batch.
Second, DAPO imposes a mixed-correctness condition on each retained group. The number of correct responses must be greater than zero and less than the group size. This is not a claim that incorrect responses are desirable in the final model. It is a statement about learning information: during exploration, a group needs at least two outcome levels for a relative comparison to point anywhere. The condition later becomes Dynamic Sampling, which keeps drawing until the update buffer has enough groups that satisfy it.
Third, DAPO aggregates policy-gradient contributions at token level rather than giving every sampled response the same final weight after averaging within that response. This matters because a long response contains more token decisions and can contain both valuable reasoning and low-quality repetition. The change does not create a new correctness label for each token. It changes how the outcome signal is distributed across the tokens already present in the sampled group.
That distinction prevents a common misunderstanding. If a response earns the positive outcome, DAPO does not thereby prove that every token in it was a good reasoning decision. It applies the response-level evidence through the policy-gradient terms for the tokens that were generated. The reduction determines how much a long sequence can influence the aggregate update relative to a short sequence; it does not turn an outcome verifier into a step-by-step judge. The same limitation applies on the negative side: a wrong final answer does not identify one uniquely responsible token. Token-level loss is a more deliberate weighting rule, not a hidden process-reward model.
Fourth, the paper removes the KL penalty from the proposed DAPO objective. A KL term ordinarily discourages the online policy from moving too far from a frozen reference policy. In long-CoT training, the authors argue that the policy may need to diverge substantially from its initial distribution to discover useful reasoning behavior, making that restriction unnecessarily tight. This is a trade-off, not a declaration that divergence is harmless. Once KL is removed, clipping, reward quality, exploration behavior, and training diagnostics carry more responsibility for keeping the update usable.
The reference point and the rollout policy also have different jobs. The old policy records the distribution that generated the current batch, so its probabilities are needed to form the update ratio. A reference policy in a KL term acts as a longer-lived anchor. Confusing those roles makes it difficult to reason about what “stability” means: staying close to the sampler that produced the evidence is not the same as staying close to the initial model. DAPO keeps the former comparison while dropping the latter penalty in its proposed objective. The update can therefore move through successive local steps without assuming that the initial model already contains the ideal reasoning distribution.
This sequence has two important boundaries. The verifier owns the outcome judgment; the policy update owns how that judgment changes probabilities. A rule-based checker can say whether an answer is equivalent to the target, but it does not explain which intermediate step deserves credit. Conversely, a better loss reduction cannot repair a reward that systematically labels good behavior as bad. DAPO’s design keeps these responsibilities distinct while making their interaction visible.
Imagine the loop handling one geometry problem. The old policy produces four attempts: one reaches the right answer directly, one reaches it after checking a construction, and two end with different wrong answers. The verifier need not rank the two correct attempts or understand the geometry; it only supplies the outcome signal. Group normalization creates a positive side and a negative side. The policy update then examines the token probabilities of all four sampled traces, uses the relative advantages to determine direction, and clips the proposed movement. After the update, the next rollout is not required to reproduce the best-looking solution. It is required to sample from a changed distribution in which behaviors associated with the positive evidence have more opportunity to appear.
That last point matters for reasoning systems. Reinforcement learning is not supervised copying of a single reference answer. The response that receives a positive outcome may contain an idiosyncratic but valid path, and another sampled path may later prove more useful. A group-relative objective can preserve that possibility because it compares outcomes rather than forcing one canonical derivation. It can also amplify an accidental correlation if the group is small, the verifier is noisy, or the policy has already lost diversity. The controls around the objective exist because both outcomes are possible.
It also explains why a group with no correctness contrast is structurally problematic. With identical rewards, the group mean equals each member’s reward, so the normalized relative advantages collapse toward zero in the described setup. The resulting update has little or no directional information from that group. This claim is specific to the group-relative objective and reward normalization here; it is not a universal statement that every reinforcement-learning objective produces exactly zero gradient whenever a batch has identical outcomes.
A final comparison helps locate DAPO in its lineage. The related DeepSeek-R1 report describes a GRPO setup that retains a KL coefficient and periodically refreshes its reference policy. DAPO’s preliminary discussion instead removes the KL term for its long-CoT objective. The two choices reflect different balances between staying close to a reference distribution and giving exploration room to discover behavior absent from that reference. They should not be merged into one generic “GRPO recipe.”
Once the loop is visible, the later stability problems become specific. If clipping constrains rare exploratory tokens too tightly, the policy can become repetitive even while the update remains formally bounded. If the policy becomes accurate enough that every sampled response is correct, group normalization loses contrast. If a response is long, sample-level averaging can distort token credit. And if the response crosses a hard length limit, the verifier and the length rule can send conflicting signals. The next question is the first of these: why can a safeguard designed for stability also starve exploration?