How a Preference Becomes a Reward
InstructGPT’s reward-model stage addresses the gap left by demonstrations: a written example shows one acceptable answer, but it does not rank all the plausible answers a model might produce. This chapter follows one prompt through candidate generation, human comparison, and scalar scoring. The aim is to make clear both what the reward model enables and what it cannot be: a learned proxy for the preferences represented in its comparison data, not a direct container for human values.
From one good answer to a ranking
The process begins with a prompt and the policy produced by supervised fine-tuning. Instead of asking a labeler to write a single ideal response, the researchers sample several responses from the model for the same prompt. The labeler then ranks those alternatives from best to worst, with ties allowed when two outputs appear equally good. In the reported setup, a labeling task presented between four and nine completions.
Why is this more useful than asking for another demonstration? A demonstration gives the model a positive example, but it leaves many questions unanswered. Is a shorter answer better than a longer one when both are correct? Should a response that directly addresses the request beat one that offers more context? When an answer is uncertain, is cautious language preferable to a confident guess? The ranking format asks the evaluator to make these relative judgments in the same context and on the same prompt. It makes quality visible as a relationship between alternatives.
The comparison also reduces the burden of specifying behavior in advance. It would be difficult for a researcher or labeler to write a complete rulebook for every way a response can be helpful, truthful, clear, constrained, or harmful. It is often easier to recognize that one of two concrete outputs is better. A labeler can notice that one answer invents a detail, ignores the requested format, or treats a dubious premise as fact even if they could not write a general rule covering every such case.
That convenience does not make the ranking objective-free. A comparison still depends on what the labeler was told to value and on what the prompt reveals about the user’s intent. The paper used a team of about 40 contractors and gave them detailed instructions about helpfulness, truthfulness, harmlessness, and trade-offs. The training process selected labelers for agreement on rankings and sensitive-content judgments. Training labelers agreed with each other about 72.6 ± 1.5% of the time. Such agreement suggests that the rubric could produce a reasonably consistent signal, but it does not mean that the signal represents every user or affected group.
The data structure expands each ranking into pairwise comparisons. If a labeler ranks four responses, the training system can derive comparisons between the preferred response and each lower-ranked response, as well as other ordered pairs in the ranking. With nine responses, the number of possible pairs is much larger than the number of prompts. The paper treats the comparisons from a single ranking task as one grouped training example rather than pretending they are independent observations. That choice matters because all of the pairs share the same prompt, the same set of completions, and the same labeler judgment.
A pairwise comparison expresses an ordering, not an absolute score. It says that response A should receive more reward than response B for this prompt. It does not say how much better A is, whether A would be good on another prompt, or whether either response meets an external standard of truth. The reward model has to infer a numerical scale whose differences reproduce as many of these local orderings as possible.
That distinction explains why ties are meaningful rather than inconvenient. If two completions seem equally good, forcing the labeler to choose one would add a false precision to the data. The study drops ties from the pairwise training signal. It also means that a ranking task contributes many related comparisons, not many independent votes. The model is learning a pattern of relative judgments attached to one prompt, not an average score assigned to isolated answers.
The prompt therefore remains part of every comparison. The same wording may make one response preferable in a creative task and another preferable in a closed-domain summary. A reward model that ignored the prompt could learn a general style—longer, more confident, or more deferential—rather than the relationship between a request and an appropriate response. Conditioning the score on both prompt and completion is what gives the objective task sensitivity, while also exposing it to gaps in prompt coverage.
The reward model turns order into a scalar
The reward model takes a prompt and a completion and returns a single number. During training, it sees two completions for the same prompt: one the labeler preferred and one they ranked lower. Its loss encourages the scalar assigned to the preferred completion to exceed the scalar assigned to the other. In simplified form, the desired relationship is reward(prompt, preferred) > reward(prompt, less-preferred). A logistic preference loss turns that relationship into a learnable objective: larger score differences correspond to higher predicted probability that the labeler would choose the preferred response.
The model is built from a language model rather than from a hand-written scoring formula. In the study, the final unembedding layer of an SFT-like model was removed and replaced with a projection that outputs a scalar reward. This means the reward model still represents the prompt and response using a language-model architecture, but the output no longer predicts the next token. It predicts how the pair would be ordered under the collected preference signal.
That change in output is conceptually important. The reward model does not generate a better answer itself. It evaluates candidate answers so that the policy can later be trained against its score. The policy owns response generation; the reward model owns the learned judgment used during optimization. Separating those roles lets the policy search among outputs that were not explicitly written by a labeler, but it also creates a new dependency: whatever the evaluator systematically rewards can become a target for the policy to pursue.
The paper used one 6-billion-parameter reward model for PPO policies of 1.3 billion, 6 billion, and 175 billion parameters. Larger reward models could obtain lower validation loss in some experiments, but their training was less stable and their use as value functions during reinforcement learning would have required substantially more computation. The 6B model was stable across a range of learning rates and produced PPO models with similarly strong results. This is an engineering choice with a conceptual consequence: policy size and evaluator size were not treated as the same variable. A very large policy was optimized against a smaller, shared judgment model.
The shared reward model also makes comparisons across policy sizes easier. If each policy size had its own evaluator, a difference in performance could arise from the policy, the reward model, or both. Using one 6B reward model holds the scoring mechanism more nearly constant. It does not make the reward model correct; it makes the experimental comparison easier to interpret.
The training procedure had to account for the structure of the data. Because several pairwise comparisons come from one ranked set, shuffling every pair into an ordinary dataset would make the examples look more independent than they are. The researchers instead trained on all comparisons from a prompt as a single batch element. This allows one forward pass for each completion rather than repeatedly recomputing the same candidates for every pair, and it reduces a form of overfitting caused by treating correlated comparisons as separate evidence. The paper reports that the reward model tended to overfit after a single pass over the data; repeating the data within an epoch caused obvious deterioration on validation measures.
The issue is not only statistical efficiency. Overfitting a reward model is especially dangerous because the policy will later optimize its output. A small mistake in ordinary prediction may affect a few scores. A policy trained to maximize those scores can magnify the mistake by finding responses that exploit the evaluator’s shortcut. The paper does not claim that its grouping and one-epoch training eliminate this risk. They are controls for a known failure surface in the particular comparison dataset.
A useful proxy is still a proxy
The reward model’s scalar can be powerful precisely because it compresses many messy judgments into a signal that an optimizer can use. But compression also removes information. A labeler may have preferred one answer because it was more truthful, another because it was clearer, and a third because it avoided a harmful implication. The pairwise label records the ordering without preserving a separate, inspectable value for each reason. The reward model must infer a pattern from the response and prompt, and the pattern can be incomplete or unstable.
The prompt is part of the score. A response is not good in the abstract; it is preferred for a particular request. This is appropriate when the task is contextual, but it means that the reward model can struggle when the prompt leaves the user’s intent unclear or when a new context changes the trade-off. A response that is helpful in an ordinary brainstorming task may be irresponsible in a medical or financial setting. A cautious answer may be preferable when facts are uncertain but frustrating when the request is simple and well specified. The comparison data must contain enough examples of these distinctions for the reward model to learn them.
The same limitation appears when a prompt asks the model to do several things at once. If the training data contains many examples of single constraints but few examples of interacting constraints, the reward model may learn broad signals such as “mention the requested items” or “sound cautious” without reliably checking every requirement. Average preference can rise while the model remains brittle on combinations that are rare in the data.
The paper’s later qualitative failures make this boundary concrete. InstructGPT can accept a false premise, over-hedge on a question with a reasonably clear answer, or degrade when an instruction contains several explicit constraints. The authors suspect that sparse false-premise examples and rewards for epistemic humility may contribute to these behaviors. These observations are not a complete diagnosis of the reward model, but they illustrate the difference between a proxy that predicts common rankings and an objective that captures the full purpose of an answer.
There is also a difference between consistency and legitimacy. The study’s labelers had substantial agreement, and held-out labelers later preferred the resulting models at similar rates. Those findings support the claim that the process can produce a repeatable preference signal beyond the exact workers who supplied every training example. They do not establish that the signal is the right trade-off for all users, or that a majority preference should settle cases where a minority group bears a disproportionate risk.
The reward model therefore changes the status of human feedback without resolving its meaning. Before this stage, human judgment appears as examples and rankings. After this stage, it appears as a scalar that can drive policy updates. That scalar makes optimization possible, but it also gives the policy an incentive to find whatever pattern increases the score, whether or not that pattern fully expresses the reason a human preferred one response over another.
This is why the next training stage needs a second kind of restraint. If PPO simply maximized the learned reward without regard for the policy’s earlier behavior, it could move into regions where the reward model is poorly tested and where pretrained capabilities are lost. The reward model supplies direction; it does not define a safe destination. The policy must now learn to follow the score while remaining close enough to a useful starting point for the comparison to mean anything.