InstructGPT and the Gap Between Prediction and Intention
When predicting the next word is not enough
InstructGPT is a useful case for understanding a deceptively difficult question: why can a language model that is very good at producing plausible text still be bad at doing what a user means? The paper studies GPT-3 models after they are fine-tuned with human demonstrations and preferences. Its focus is not simply whether the model can produce fluent language, but whether training can change the objective that guides its behavior. That distinction supports the central judgment of this series: human feedback can make a model more responsive to users, but the feedback becomes an optimization target through a designed process, and therefore remains narrower than human intent itself.
A pretrained language model is trained mainly to predict the next token from text. That objective gives it broad linguistic and factual patterns, and it lets the model adapt to many tasks through prompting. But the text it has seen does not tell it, in a reliable and explicit way, what a particular user wants from a particular request. A prompt may ask for a summary, a decision, a story, or an answer to a question. It may also contain an implicit constraint: stay within the supplied evidence, correct a false premise, avoid harmful advice, or write in a specified form. A continuation can be statistically plausible while missing the task. It can sound confident while inventing a fact, or follow a harmful request too readily.
InstructGPT treats this as an objective mismatch. The target is not merely “produce likely text,” but behave in a way that is useful to the person making the request. The paper describes that behavior through three practical dimensions. Helpfulness means trying to solve the user’s task and infer the intended request when it is expressed indirectly. Truthfulness means avoiding fabricated or misleading information. Harmlessness means avoiding outputs that can cause physical, psychological, social, environmental, or institutional harm. These dimensions are related, but they are not interchangeable. A model can be obedient but untruthful, cautious but unhelpful, or responsive to a user while making a dangerous request easier to carry out.
That is why making the model larger is not the same as making it better aligned. The paper compares GPT-3 models with InstructGPT models built from the same general architecture. On the study’s API prompt distribution, a 1.3-billion-parameter InstructGPT model was preferred to a 175-billion-parameter GPT-3 baseline by the human evaluators. This is a striking result, but its meaning is specific: changing post-training behavior produced a larger preference gain on those tasks than the parameter-count difference produced. It is not a general law that small fine-tuned models always beat large pretrained models, nor does it show that InstructGPT is universally aligned.
The feedback loop changes the target
The important innovation in the paper is not a single algorithm. It is the way several forms of human judgment are connected into a training loop. First, labelers write examples of responses that would be useful for a wide range of prompts. GPT-3 is fine-tuned on those examples to create a supervised policy, usually called SFT. This gives the model an initial ability to treat a natural-language request as an instruction rather than merely as the beginning of text.
Second, the researchers show model outputs to labelers and ask them to rank alternatives for the same prompt. Those comparisons are different from demonstrations. A demonstration says, in effect, “here is one response that illustrates the desired behavior.” A ranking says, “between these possible behaviors, this one is better.” That relative judgment can cover qualities that are difficult to enumerate in a fixed answer: relevance, clarity, restraint, truthfulness, and how well the response respects the request’s constraints.
The rankings then train a reward model. The reward model receives a prompt and a response and produces a scalar score. It is trained so that a response preferred by the labelers receives a higher score than the response they ranked lower. At this point, a qualitative judgment has been translated into a numerical signal that another model can optimize. The scalar is not a direct measurement of human values. It is a prediction of the preferences represented in the comparison data.
Third, the supervised policy is fine-tuned with reinforcement learning. InstructGPT uses proximal policy optimization, or PPO, to produce responses that score well under the reward model. The policy is not being taught only by copying a fixed set of answers. It is being adjusted so that its future outputs receive higher predicted reward. A penalty based on divergence from the supervised policy helps restrain this search, while the PPO-ptx variant also mixes in updates from the pretraining distribution to preserve capabilities that preference optimization might damage.
The sequence matters because each stage changes what the next stage can do. The prompt distribution determines which tasks enter the process. Demonstrations create a workable starting policy. Comparisons define which kinds of responses are preferred. The reward model compresses those comparisons into a score. PPO then searches for policy changes that improve the score. In this sense, human feedback becomes an optimization objective only after it has passed through several interfaces: a prompt-selection process, a labeling rubric, a comparison format, a reward model, and a policy-update algorithm.
That chain also explains why the phrase “human preferences” needs care. The model does not receive an unfiltered expression of humanity’s wishes. It receives data generated by particular labelers, under instructions written by researchers, while responding to prompts from a particular customer population. The labels are therefore part of an engineered measurement system. Changing the people, the instructions, the interface, or the prompt distribution could change the behavior that the reward model learns to favor.
What the result does—and does not—show
The paper evaluates InstructGPT at several distances from the training objective. The closest measure is human preference on held-out prompts from the API distribution. The researchers also ask labelers to score overall quality and mark more specific behaviors, such as whether the model attempted the correct instruction, followed an explicit constraint, used language appropriate for a customer assistant, or hallucinated information in a closed-domain task. These measures make the headline result more concrete: InstructGPT is not merely preferred as a style; it is more controllable in several task-relevant ways.
The evaluation then moves farther from ordinary preference judgments. Truthfulness is examined through TruthfulQA and through the rate at which a model adds information not present in a closed-domain input. Toxicity is assessed with both automatic scores and human judgments on RealToxicityPrompts. Bias is probed with Winogender and CrowS-Pairs. Traditional language benchmarks test whether fine-tuning has damaged capabilities that were not the main target of the feedback process.
These evaluations are useful precisely because they are not all the same. A labeler’s preference tells us how a response compares under a rubric. A hallucination label tests a narrower failure mode. TruthfulQA asks about another kind of behavior, and toxicity measures depend on how the prompt is framed. The paper finds that InstructGPT improves truthfulness and reduces hallucination on the reported tasks. It finds a toxicity advantage when the model is explicitly instructed to be respectful, but that advantage largely disappears without the respectful instruction. It does not find a reliable improvement on the selected bias measures. A model can become more certain and more responsive without becoming less biased.
The conditions are part of the result. InstructGPT is better at following instructions, and that is beneficial when the instruction asks for a respectful, accurate, constrained answer. The same capability can make the model more effective at producing toxic output when the prompt explicitly asks for biased or offensive language. The paper’s training setup also allowed the model to comply with potentially harmful requests rather than imposing a universal refusal policy. Improved instruction following is therefore not identical to harmlessness. It is a capability whose consequences depend on the request and the deployment context.
The same caution applies to the labelers. The study reports that held-out labelers, who did not generate the training data, preferred InstructGPT over the GPT-3 baselines at similar rates. That result is evidence that the models did not simply memorize the judgments of individual workers. It is not evidence that the model represents all users, all cultures, or all people affected by its outputs. Agreement within a selected evaluation process is a useful generalization check, not a universal consent mechanism.
The most useful way to read InstructGPT, then, is as a controlled demonstration of objective translation. The paper shows that a broad pretrained model can be made more helpful and more controllable by collecting examples and comparisons, learning a reward proxy, and optimizing a policy against it. It also shows that the proxy has boundaries. Its gains are measured on particular distributions. Its safety improvements are conditional. Its benchmark regressions require a second objective to mitigate them. Its failures include accepting false premises, hedging when a direct answer is warranted, and degrading when several constraints must be satisfied at once.
Those boundaries do not make the result unimportant. They identify what the result actually contributes. InstructGPT offers evidence that alignment work can improve an existing model without waiting for a larger model, and that some instruction-following behavior can generalize to tasks such as code questions and non-English prompts even when direct supervision is sparse. But it does not turn a learned reward into a complete account of what people want. The unresolved question is not whether human feedback matters. It is how faithfully a particular feedback process captures the intent, risks, and interests that matter in the situations where the model will be used.
The next question follows directly from this boundary. Before preferences can be optimized, the system must decide which prompts and tasks will be placed in front of the labelers and the model, and it must create examples that make the desired behavior learnable. In InstructGPT, the quality and coverage of that starting material are not peripheral data decisions. They are the first part of the objective.