Why Language Models Need a Feedback Signal
For Constitutional AI, the prerequisite problem is simple to state but easy to miss: a language model can be highly capable at producing text without being trained to pursue the user’s actual intent. This chapter focuses on the feedback architecture that turns a pretrained model into an instruction-following assistant. By the end, the reader should be able to trace the route from a prompt to a supervised policy, a reward model, and a constrained policy update—and see why that route represents a particular group’s judgments rather than “human values” in the abstract.
From next-token prediction to user intent
A pretrained language model is optimized to predict what text is likely to come next. The training material may contain instructions, explanations, arguments, stories, warnings, and countless examples of how people talk to one another. From that material, the model can acquire broad capabilities and can often imitate the style of an assistant. But the objective itself does not ask a crucial question: what response would best serve this person, in this situation, with acceptable consequences?
That missing question creates a gap between capability and alignment. A model may complete a prompt fluently while ignoring an explicit constraint, answering a different question, inventing information, or producing language that is toxic or dangerous. Scale can improve the model’s ability to represent patterns, but it does not by itself tell the model which patterns should govern a deployed interaction. The InstructGPT paper makes this point directly: making language models larger does not inherently make them better at following a user’s intent.
The papers make that intent more concrete with three practical dimensions. A helpful assistant should follow the task the user is trying to accomplish, including an intention implied by examples rather than stated as a formal instruction. An honest assistant should not fabricate facts or mislead. Because a model’s private “beliefs” are not directly observable, the InstructGPT work measures parts of honesty through truthfulness: whether answers are accurate, whether a closed-domain summary adds information not present in its input, and whether the system avoids confidently repeating false premises. A harmless assistant should avoid causing physical, psychological, or social harm, as well as encouraging illegal or dangerous activity.
These dimensions can conflict. A model that obeys every request may be more responsive but may also help a user carry out a harmful plan. A model that refuses every sensitive request may be harmless in a narrow sense but useless, opaque, and unable to provide a safer alternative. The training problem is therefore not just to add a safety filter. It is to shape behavior under competing objectives while preserving enough of the model’s general capability to make the intervention worthwhile.
This is where feedback enters. Instead of relying only on the statistical patterns absorbed during pretraining, the developer supplies examples of desired behavior and judgments about competing outputs. Those judgments become an additional signal that can be learned and optimized. Reinforcement learning from human feedback, or RLHF, is the baseline architecture used in the companion InstructGPT study and the structure that Constitutional AI later adapts for harmlessness.
How demonstrations become a constrained reward
The RLHF pipeline has three connected stages. First, human labelers write demonstrations: a prompt is paired with an answer that is intended to help the user. A pretrained model is fine-tuned on these examples using supervised learning. The result is a supervised policy, often called an SFT model, that has a more reliable starting behavior than the raw pretrained model.
The demonstrations do more than provide isolated good answers. They define the initial distribution from which the later training process samples responses. That matters because reinforcement learning does not search over every conceivable piece of language equally. It starts from a policy with its own habits, tendencies, and errors. A supervised model that more often attempts the requested task gives the next stage a more useful surface to improve. In CAI, this role becomes especially important: the supervised constitutional stage is intended to move the model “on-distribution” before reinforcement learning refines it.
The second stage collects comparisons. Given one prompt, the supervised policy generates multiple candidate responses. Human labelers rank those responses from best to worst. The comparisons are then used to train a reward model, which receives a prompt and a response and outputs a scalar score. The model is not told a universal measure of goodness. It learns to assign higher scores to responses that resemble the choices made by the labelers.
The comparison format is valuable because it asks for a relative judgment. A labeler may find it difficult to assign an absolute number to the quality of an answer, but can often say that one of two answers is more useful, more accurate, or less harmful. In the reward-model formulation, the difference between two predicted scores corresponds to the odds that one completion will be preferred over the other. The reward model thereby compresses many local comparisons into a function that can score new samples.
This compression also determines what the system can see. A labeler compares the responses that happen to be presented for one prompt; the reward model generalizes from those examples to responses it has not seen. If the candidate set rarely contains a useful alternative, the model receives little evidence about what “better” would look like. If the prompt leaves the user’s intent ambiguous, a ranking may reflect the labeler’s interpretation rather than the user’s. The learned score is therefore a map of judgments under a sampling and instruction regime, not a direct measurement of value in the world.
That compression is the key systems boundary. Human judgment enters as demonstrations and rankings; the reward model owns the learned approximation; the policy optimizer then treats that approximation as a training signal. Each handoff changes the object being passed along. A ranking is not the same thing as a written explanation. A scalar score is not the same thing as a ranking. And maximizing a scalar score is not the same thing as satisfying the original human intention in every context.
The third stage uses the reward model in reinforcement learning. In the InstructGPT setup, the policy produces an answer to a prompt, receives a reward from the reward model, and is updated with proximal policy optimization, or PPO. The episode is short—a prompt and response—so the system resembles a bandit: the model chooses a completion, receives an evaluation, and the interaction ends. Repeating this process shifts the policy toward outputs that the reward model scores more highly.
The policy is not supposed to move without restraint. PPO includes a per-token Kullback–Leibler, or KL, penalty relative to the supervised policy. In plain terms, the new policy pays a cost for drifting too far from the behavior that supplied its starting point. This anchor is a guard against exploiting quirks in the reward model so aggressively that the assistant loses the qualities learned from demonstrations. It does not make the reward model correct; it limits how quickly its errors can reshape the policy.
The researchers also experimented with mixing ordinary pretraining updates into PPO. They call the resulting models PPO-ptx. The purpose is to reduce an “alignment tax”: a loss in performance on capabilities that matter but are not well represented in the preference data. In the reported InstructGPT results, standard PPO showed regressions on some public tasks, including question answering, reading comprehension, and translation. The pretraining mix substantially reduced many of those regressions, although it did not remove all of them. Increasing the KL coefficient alone was not as effective in the cited experiments.
It helps to read the PPO objective as a negotiation among three forces. The reward model pulls the policy toward outputs that resemble preferred answers. The KL term resists abrupt departure from the supervised policy. The pretraining term protects patterns of general competence that may be invisible to the preference data. None of these forces is the original user intention. The first is a learned approximation, the second is a stabilizer, and the third is a preservation term. Their coefficients determine which errors are easier for the policy to make: it may stay too close to an inadequate starting behavior, optimize a narrow reward, or preserve undesirable patterns that were present in pretraining. The architecture is useful precisely because these pressures can be adjusted, but that adjustability is not the same as knowing the correct setting in advance.
The overall architecture can be read as a controlled flow of state and authority:
The diagram also shows why the later Constitutional AI substitution is precise. CAI does not discard the whole architecture. It replaces the source of direct harmlessness comparisons: instead of asking a human to rank two responses for harmlessness, it asks a feedback model to choose under a constitutional principle. The preference model and PPO stages remain recognizably similar. To understand what changes, it is necessary to keep the original boundaries in view.
The boundary is also empirical. InstructGPT was evaluated mainly on held-out prompts drawn from related API distributions, supplemented by public datasets and targeted tests. That design can reveal whether the training signal improves the kinds of tasks it was built around, but it cannot guarantee the same behavior in an unfamiliar application. The companion paper notes that public NLP benchmarks do not fully represent real API use, which includes a large share of open-ended generation and brainstorming. A feedback system is only as broad as the situations used to collect and test its judgments.
A baseline aligned to a reference group
The word “human” in RLHF can conceal several design choices. InstructGPT’s data came from a group of roughly 40 contractors, selected and trained for the task. Researchers wrote the instructions that guided demonstrations and rankings, answered questions about edge cases, and selected the prompt distribution. Much of that distribution came from customers using an API. The resulting policy was therefore aligned to a particular combination of labeler judgments, researcher guidance, and customer use—not to a single, universally agreed human objective.
The study reports substantial agreement among its labelers, around 73 percent, and finds that the trained model generalized reasonably well to held-out labelers. Those are useful signs that the system learned more than one individual’s idiosyncrasies. They do not erase disagreement or establish that the reference group represents everyone affected by deployment. The authors explicitly distinguish their operational target from broader claims about human values. Customers may want one behavior while end users want another; people affected by a model’s output may not be present in the labeling process at all.
That distinction will matter when principles replace direct harmlessness labels. A constitution can make the intended direction more inspectable and can reduce the number of examples humans must label. It cannot decide, by itself, which harms count most, how conflicting values should be traded off, or whether a response is safe in a deployment context the feedback process never saw. Feedback is a mechanism for transmitting judgment, not evidence that judgment has disappeared.
The RLHF baseline thus gives Constitutional AI both its opportunity and its constraint. The opportunity is to use a model’s capabilities to produce some of the evaluation and training material that humans would otherwise have to create. The constraint is that every learned reward remains a proxy, anchored to a starting policy and a data distribution. The next question is whether a written principle can first help the model correct its own responses, creating a safer and more useful starting point before any comparison-based reward is optimized.