Whose Intent Is Being Optimized?

InstructGPT answers the opening question in two parts. Human feedback can make a pretrained language model much better at following instructions, but the resulting behavior is not aligned with an abstract, universal humanity. It is the output of a chain that selects prompts, elicits demonstrations, ranks responses, fits a reward proxy, updates a policy, and evaluates selected consequences. The final judgment of this series is therefore neither that human feedback solves alignment nor that it is merely cosmetic: it is a powerful way to change an existing model, whose meaning depends on the process that supplies and constrains the feedback.

One feedback loop, several narrowing choices

The full InstructGPT pipeline is easiest to understand as a sequence of translations. Pretraining gives GPT-3 a broad ability to model language and many patterns in the world, but it does not directly specify how an assistant should respond to a user’s intent. The researchers then choose a field of prompts from labelers and customers. That choice turns “what people may want from a language model” into a distribution of tasks that the training process can actually see.

Labeler-written demonstrations make the desired relationship between prompt and answer concrete. Supervised fine-tuning turns those examples into an initial policy that is more likely to treat a request as an instruction. The policy then generates multiple candidates for the same prompt. Labelers rank those candidates, converting a judgment about relative quality into comparison data. A reward model compresses the comparisons into a scalar score. PPO uses the score to change the policy, while a KL penalty and, in PPO-ptx, a pretraining objective restrain the update and protect capabilities outside the narrow feedback target.

Each translation makes optimization possible and also discards something. A customer’s real context may be absent from the prompt. A labeler’s reasons for preferring one answer may be reduced to an ordering. The ordering becomes a scalar whose meaning depends on the prompt and the examples. The scalar becomes a gradient that changes probabilities over future text. By the time the final policy is evaluated, the system no longer has direct access to the original human situation; it has a learned behavioral tendency produced by these intermediate representations.

flowchart LR A["Prompt and stakeholder selection"] --> B["Demonstrations and SFT policy"] B --> C["Comparisons and reward proxy"] C --> D["PPO with KL and pretraining preservation"] D --> E["Observed gains and alignment tax"] E --> F["Failure and misuse surfaces"] A --> G["Representation and governance questions"] C --> G F --> G

This is why the reward model should not be imagined as a small container holding human values. It is a model of a particular comparison process. Its score is useful because it lets the policy improve responses that labelers would likely prefer, including responses no labeler wrote verbatim. Its limitation is the same one that gives it power: the policy can optimize patterns in the score more aggressively than a human can inspect each answer. A reward that correlates with truthfulness in common cases may still reward hedging, confident style, or direct obedience in cases where those traits are wrong.

The alignment tax makes the chain visible from the other side. PPO improved the target behavior but reduced performance on several public NLP tasks. PPO-ptx recovered much of that capability without sacrificing labeler preference, but not all of it. The result shows that an objective is not a neutral description of quality. It allocates pressure. More pressure on instruction-following behavior can mean less pressure on another capability unless the training process explicitly preserves both.