What Improved: Control, Truthfulness, and the Uneven Safety Story
With the PPO training loop in place, InstructGPT can be judged by what people and benchmarks actually observe rather than by the elegance of its objective. This chapter separates three questions that are often collapsed into one: did the model become easier to control, did it become more truthful, and did it become safer? The paper reports meaningful gains, but each gain belongs to a particular prompt distribution and measurement method. Reading the results well means preserving those conditions instead of turning them into one broad claim of alignment.
The staged gains are mainly gains in control
The most direct evidence comes from human preference evaluations on held-out prompts. The study compares several stages of the pipeline: the original GPT-3 models, GPT-3 with a carefully chosen few-shot prefix, supervised fine-tuning, PPO, and PPO-ptx. Preference improves in that sequence. Prompting helps the base model enter an instruction-following mode; demonstrations make that behavior more systematic; optimizing comparisons improves it further. The stages therefore provide more than a leaderboard. They connect changes in the observed behavior to changes in the training signal.
The size of the reported difference is substantial. On the study’s API prompt distribution, outputs from the 175B InstructGPT model were preferred to outputs from 175B GPT-3 85 ± 3% of the time. They were preferred to outputs from few-shot prompted 175B GPT-3 71 ± 4% of the time. The comparison indicates that the human-feedback stages changed how the same general model family responded to requests. It does not mean that every response improved, that a smaller model will always defeat a larger one, or that the comparison applies to prompts outside the study.
The researchers looked beyond the overall preference ranking. Compared with GPT-3, the PPO models were more often judged appropriate in the context of a customer assistant. They attempted the correct instruction more often, followed explicit constraints more reliably, and hallucinated less often on closed-domain tasks, where the answer should not add information absent from the input. These measures turn “better” into recognizable control properties. A model that produces elegant prose but ignores “use two paragraphs” has not fully followed the request; a model that summarizes by adding plausible but unsupported facts has not respected the task boundary.
The individual measurements also help explain why preference is an informative but composite outcome. A labeler’s overall score may reflect several features at once: relevance, clarity, compliance with the requested form, factual restraint, and the impression that the response would be appropriate from a customer assistant. The metadata labels separate some of those features after the fact. If the overall preference rises together with explicit-constraint satisfaction and lower closed-domain hallucination, the interpretation that the policy became easier to control is stronger than if only a stylistic preference changed.