The Stability Map: From Exploration to Informative Learning

DAPO’s first problem is now clear: a clipped policy can become so concentrated that it stops testing useful alternatives. The checkpoint here connects that exploration diagnosis to the larger learning loop. The practical judgment is not “make entropy higher.” It is to distinguish a healthy policy distribution from a useful update: the first keeps alternatives alive, while the second depends on what those alternatives reveal when they are sampled and compared.

A policy distribution is a search surface, not a verdict

A policy distribution determines which continuations can enter the next rollout. The rollout then turns probabilities into actual responses, and the verifier turns those responses into outcomes. Group-relative normalization compares the outcomes before the policy update changes the distribution again. The loop is therefore circular: policy shape affects the evidence available to the group, group evidence affects the update, and the update changes the policy shape that will generate the next evidence.

That loop gives exploration a precise role. A rare continuation does not matter because rarity is inherently good. It matters because a continuation that is rarely sampled cannot accumulate outcome evidence. If it appears in a successful reasoning path, the learner needs enough upward room for the path—or at least its useful token choices—to become more likely in later rollouts. If it appears in an unsuccessful path, the update needs a way to reduce it without erasing unrelated alternatives. Clip-Higher changes that probability movement at the policy boundary.

But probability movement is only the first link. Once a group has been sampled, the system needs a comparison that says something about the candidates. A group of distinct-looking responses can still be uninformative if their outcomes are indistinguishable or if the verifier is not reliable. Conversely, a group with a clear outcome difference can provide a strong relative signal even when the responses share much of their structure. Diversity is therefore a condition for search, not a substitute for evidence.

Consider two hypothetical snapshots of the same DAPO run. In the first, entropy has fallen, the responses repeat the same opening, and validation accuracy stops improving. The policy distribution is failing before the verifier can see enough alternatives. In the second, entropy is higher and the responses take visibly different routes, but every route receives the same outcome on the sampled prompts. The search surface is healthier, yet the group-relative update still has little to distinguish. The two snapshots can look similar if the only reported number is “training reward,” but they require different questions. The first asks whether probability movement has collapsed exploration; the second asks whether the batch contains useful outcome contrast.

This distinction also prevents a misleading use of the word diversity. Different strings are not necessarily different reasoning strategies. A group can vary in wording while making the same mathematical mistake, or repeat a common template while containing one meaningful alternative. The verifier and the group comparison determine whether variation becomes learning evidence.