When Correctness Becomes a Dead End: Dynamic Sampling

DAPO’s Dynamic Sampling addresses a paradox in group-relative reinforcement learning: as the policy becomes better at a question, the resulting batch can become less useful for learning. This chapter explains why all-correct and all-incorrect groups contain little relative information, how DAPO oversamples and filters them through a dynamic buffer, and why the extra rollout work can still support faster convergence under the system conditions described in the paper. The important judgment is not that more sampling is always efficient; it is that useful update information, rather than nominal batch size, is the resource being protected.

Why a perfect group can provide no direction

GRPO does not judge a response in isolation. It compares the rewards of several responses generated for the same question, then normalizes each reward relative to that group. The comparison is useful when the responses occupy different outcome levels. A correct response can receive a positive relative advantage when its peers are wrong; an incorrect response can receive a negative one. The update can then increase the probability of patterns associated with the successful side and decrease the probability of patterns associated with the unsuccessful side.

Now consider a group in which every response is correct. The verifier has delivered good news, but it has not distinguished the members of the group. Their rewards share the same level, so subtracting the group mean removes the common signal. In the described group-relative setup, the normalized advantages become zero or collapse toward zero, and that group contributes no useful policy gradient. An all-incorrect group has the same structural problem in the other direction: it shows that the sampled set failed, but not which alternative should be preferred.

This is not a claim that all-correct and all-incorrect groups have the same value in every learning system. An all-correct group may be valuable for evaluation, and an all-incorrect group may reveal that a prompt is difficult or that the policy needs broader exploration. The narrower point is about this particular update: a common outcome within a group does not provide the relative contrast the objective uses to decide how token probabilities should move.

The problem grows as the model improves. Early in training, many prompts produce mixed groups because the policy solves some attempts and misses others. As successful behavior is reinforced, the fraction of prompts whose sampled responses are all correct rises. The nominal batch may still contain the same number of questions, but fewer of those questions produce a directional comparison. The effective prompt count shrinks. The remaining gradient is calculated from a smaller and more variable set of informative groups, so its magnitude can decrease and its sensitivity to noise can increase.

This is why benchmark accuracy and learning signal can move in opposite directions. A policy that solves more sampled attempts may look healthier by the outcome metric while supplying fewer mixed groups for the next update. The training process has not necessarily run out of correct answers; it has run short of contrasts between answers. If the update keeps treating saturated groups as ordinary data, the batch size conceals a gradual loss of information density.

The DAPO paper tracks this saturation directly through the proportion of samples whose group accuracy is one. As that proportion grows, the nominal number of prompts can stay constant while the number of prompts capable of producing a nonzero relative comparison falls.