Who Gets Credit for a Long Answer? Token-Level Loss

Dynamic Sampling protects the group-level contrast that an update needs. DAPO’s Token-Level Policy Gradient Loss addresses the next boundary: once a group is informative, how much influence should each response and each token have? This chapter compares GRPO’s sample-level reduction with DAPO’s token-level aggregation, using the two opposite length failures the paper reports—useful long reasoning being diluted and low-quality long detours being under-penalized. The goal is to understand why healthier length is a stability result, not a license for unlimited verbosity or a substitute for process supervision.

The hidden weighting inside sample-level averaging

A policy-gradient loss is built from terms for the tokens generated in each response. The original GRPO formulation first averages those terms within each sample and then averages the samples in the group. That reduction looks balanced because every response receives one place in the final average. But equal response weight is not equal token weight.

Imagine a group with one response containing 100 tokens and another containing 1,000. Under sample-level reduction, the 100-token response and the 1,000-token response can each contribute one averaged sample loss. The tokens in the long response therefore carry a much smaller share of the final loss than the tokens in the short response. This may be appropriate if every response should count as one indivisible observation. It becomes questionable when the long response contains additional reasoning decisions that the policy must learn from, or additional low-quality behavior that should be suppressed.

The issue is not simply that long answers are longer. Long-chain-of-thought responses often contain internal structure: an initial approach, a check, a correction, a backtrack, or a repeated phrase. Treating the whole response as one equally weighted sample hides the fact that the model made many more policy decisions inside it. A useful long solution can contain patterns that deserve reinforcement but receive diluted influence because the response is long. A pathological response can occupy many tokens with gibberish or repetition while its average loss remains only one sample’s share of the update.

That creates two opposite distortions. First, the system can underlearn reasoning-relevant patterns in a long correct response. The outcome says the trajectory reached the right answer, but sample-level averaging prevents its many token-level decisions from having proportional influence. Second, the system can fail to punish undesirable long behavior strongly enough. Repetition increases the response length, but the response still enters the final loss as one sample, so the aggregate update may not push back against every repeated stretch with enough force.

These distortions can reinforce one another. If useful long responses are underlearned, the policy may keep relying on familiar short or incomplete paths. If pathological long responses are under-penalized, the model may discover that additional tokens provide room to continue searching without paying a commensurate training cost. The result can be an unhealthy increase in both entropy and response length: more tokens, but not necessarily more reliable reasoning.

The distinction from Dynamic Sampling is important.