Overview

This chapter follows how InstructGPT’s PPO stage changes the SFT policy under a learned reward. In a one-response bandit setup, the policy samples an answer, receives reward-model feedback, and pays a per-token KL penalty for departing from SFT. PPO raises the likelihood of high-scoring outputs, while PPO-ptx adds gradients from the pretraining distribution to reduce the alignment tax—the regression of some public NLP capabilities after optimizing the API-oriented objective. In reported experiments, PPO-ptx greatly reduced those regressions without sacrificing labeler preference, but recovery remained incomplete and increasing KL was not equivalent. The trade-off is deliberate rather than solved: reward, SFT proximity, and capability preservation can pull in different directions, and the score remains a proxy.

Learning outcomes

PPO changes SFT by increasing the likelihood of responses favored by the reward model while charging for token-level drift from SFT.
PPO-ptx directly preserves pressure from the pretraining distribution, unlike simply increasing the KL coefficient.
PPO-ptx reduced—but did not eliminate—reported public-task regressions, so alignment and capability preservation remain competing objectives.

Keywords

InstructGPTPPOPPO-ptxKL divergenceAlignment taxPretraining

The Policy Learns to Chase the Score—and Stay Near Home

InstructGPT’s reward model supplies a direction, but it does not change the policy by itself. This chapter follows the final training step: how PPO uses the learned score to alter the supervised policy, why the update is restrained by a KL penalty, and why the authors add pretraining gradients in PPO-ptx. The key judgment is that alignment is a competing-objective problem. Improving the behavior that people prefer can also damage capabilities that the feedback process did not explicitly protect.

Optimizing a learned score without losing the starting policy

The reinforcement-learning environment in the study is deliberately simple. It presents a prompt, allows the policy to generate a response, gives that prompt-response pair to the reward model, and ends the episode after the response. This is a bandit setting: the policy does not receive a long sequence of intermediate rewards while carrying out a multi-step task. It receives one learned evaluation of the completed answer, with an additional penalty applied across the response tokens.

The policy being updated is initialized from the supervised model. For each prompt, it samples a response and receives a scalar from the reward model. PPO then changes the policy so that responses with higher predicted reward become more likely. In ordinary language, the policy is being taught to choose outputs that the learned evaluator would prefer. In optimization language, it is no longer merely imitating the labeler demonstrations; it is searching for a distribution of responses that scores well under the proxy.

That search needs a reference point. The reward model was trained on a finite comparison dataset, so it is most trustworthy near the kinds of prompts and responses it has seen. If the policy is free to change dramatically, it may find strange responses that receive high reward because the evaluator interprets them incorrectly. The policy could also drift away from the language patterns and broad capabilities it inherited from pretraining. A high score would then mean only that the policy had learned to satisfy the evaluator, not that it had become a better assistant.

InstructGPT controls this drift with a per-token Kullback–Leibler, or KL, penalty measured against the SFT policy. KL divergence can be understood here as a cost for moving the new policy’s token probabilities too far from the supervised model’s probabilities. The penalty does not prohibit change. It makes change expensive when the new policy departs sharply from the behavior that produced the candidate responses and served as the comparison process’s starting point.

The resulting objective has two immediate forces. The reward-model term pulls the policy toward outputs that labelers were predicted to prefer. The KL term pulls it back toward the SFT distribution. The coefficient on the KL term controls the balance. With a weak penalty, the policy can pursue the score aggressively but may leave the region where the reward model is reliable.

Published by DeepMox

FAQ

What is the KL penalty doing in InstructGPT?

The KL penalty makes it costly for the updated policy’s token probabilities to move too far from SFT, restraining exploration without forbidding change. It is a guardrail, not proof that the policy cannot exploit the reward model.

What is the alignment tax in InstructGPT?

The alignment tax is the loss of performance on capabilities measured by public NLP benchmarks after the policy is optimized for the API-oriented feedback objective. It can occur even while human preference rises.

How does PPO-ptx differ from increasing KL?

PPO-ptx directly trains on the original pretraining distribution, whereas increasing KL mainly keeps the policy closer to SFT. In the reported analysis, the pretraining mix produced a better balance for the tested settings.

Previous12 minInstructGPT: Preference Rankings and Reward Models

Next10 minInstructGPT: Control, Truthfulness, and Safety

InstructGPT: PPO, KL, and the Alignment Tax

The Policy Learns to Chase the Score—and Stay Near Home

Optimizing a learned score without losing the starting policy

FAQ