Before Reward, There Must Be an Example
InstructGPT’s first post-training step answers a practical question: what does a model need to see before it can be optimized for human preferences? This chapter focuses on the prompt distribution and the supervised fine-tuning stage that turns prompts plus labeler-written responses into an initial instruction-following policy. Understanding that starting point matters because the later reward model can only improve behavior that the data-collection process has made visible and the policy has learned to produce.
The prompt distribution is part of the objective
The training data came from two connected sources. One consisted of prompts written by the labelers themselves. The other consisted mainly of prompts that customers had submitted through an earlier version of the OpenAI API Playground, where they interacted with an early supervised version of the model. The two sources served different purposes. Customer prompts brought in tasks that people actually wanted to perform. Labeler-written prompts supplied instruction-like examples that were not common enough in ordinary GPT-3 usage to bootstrap the first model.
That distinction is easy to underestimate. If the researchers had relied only on text that users had previously sent to a general language model, much of the material would have represented continuation or completion rather than a clear request to an assistant. To teach a model to follow instructions, they needed inputs that made the task legible: write a story, answer a question, rewrite a passage, summarize supplied material, classify an item, or carry out a constraint. Labelers were asked to create plain prompts, few-shot prompts with examples, and prompts based on intended use cases. This was not merely a way of increasing the dataset. It changed the type of behavior the model was invited to learn.
The resulting distribution was broad but not neutral. It included generation, open and closed question answering, brainstorming, chat, rewriting, summarization, extraction, classification, and other language tasks. Generation and brainstorming made up a much larger share of the API-oriented material than the kinds of classification and question-answering tasks that dominate many public NLP benchmarks. The paper later reports that classification and question answering accounted for about 18% of its labeled API prompt distribution, while open-ended generation and brainstorming accounted for about 57%. Those proportions help explain why a model trained on real user prompts could outperform models fine-tuned on large collections of public task datasets when evaluated by API users’ apparent needs.
The point is not that one distribution is intrinsically correct. It is that the distribution declares what the system is being prepared to do. A model trained mostly on multiple-choice questions will receive a different behavioral education from one trained mostly on open-ended writing and brainstorming. The first may have clearer automatic targets; the second requires judgments about usefulness, tone, constraints, and context. InstructGPT’s authors argue that the broadest instruction-following system would combine both kinds of data. Their comparison does not make public benchmarks irrelevant. It shows that benchmark coverage and user coverage are different forms of coverage.
The difference also affects what counts as a good answer. In a closed question-answering task, the input often supplies a relatively clear criterion for correctness. In brainstorming or creative generation, the response may be judged by whether it develops the request, respects an implied audience, and offers useful variety without becoming repetitive. Summarization sits between these cases: it is open-ended in wording but should remain faithful to the supplied text. A single model can encounter all of these forms, but the feedback process has to expose their differences or the reward signal will flatten them into a vague preference for whatever style appears most often.
This is why “more data” is not enough to describe the design. The labeler-written prompts are not simply extra rows added to the customer prompts. They make certain kinds of intent explicit and create examples for tasks that the early model would otherwise see only rarely. Customer prompts, in turn, keep the dataset tied to actual usage rather than to an imagined curriculum. The two sources meet in the training pipeline, but they do not mean the same thing: one bootstraps the concept of an instruction, while the other supplies evidence about the kinds of work customers were trying to do.
The collection process also changed what could be learned and how the results could be interpreted. Prompts were heuristically deduplicated by looking for long common prefixes, and the number of prompts contributed by a single user was limited. Training, validation, and test splits were made by user ID, so that the held-out sets did not contain users represented in training. That design is important: a model might appear to generalize if it merely recognizes a recurring user’s style or subject matter. Splitting by user creates a stronger test of whether the behavior transfers to new customers within the study’s distribution.
The researchers filtered personally identifiable information from the training split. This is both a privacy measure and a boundary on the examples the model receives. The policy is trained on a curated view of requests, not on an unmodified stream of every interaction. The data was also overwhelmingly English—over 96% by the paper’s language classifier—and came from a Playground context rather than from every setting in which a language model might be deployed. The distribution was diverse enough to support the study’s claims about broad instruction following, but it was not a representative sample of all users, languages, domains, or risks.
The user-based split is especially important for reading the later comparisons. If the same person’s prompts appeared in both training and testing, a preference gain could partly reflect familiarity with that person’s recurring topics or wording. Holding out customers does not remove every form of overlap, and it does not test deployment in a new domain, but it makes the reported result a more meaningful test of transfer within the chosen API population. The split is therefore part of the evidence design, not merely a bookkeeping decision.
That qualification belongs at the center of the mechanism. A prompt distribution is not just an input file waiting for an algorithm. It determines which intents are common, which constraints appear often, which harms are visible to labelers, and which capabilities are treated as useful. When the later policy is praised for following instructions, it is following instructions drawn from this chosen field of view. Data curation therefore enters the optimization objective before a reward model exists.
Demonstrations create the first policy
Once a prompt is selected, the first training signal is a demonstration of the response someone wants the model to produce. The labeler’s task is not limited to answering the literal surface form. A request may state its goal directly, imply it through a few examples, or provide the beginning of a passage that the model is expected to continue. Labelers were asked to infer the user’s intention and to skip inputs that were too unclear. They were also asked to account for implicit expectations such as truthfulness and avoidance of harmful, biased, or toxic language.
This makes a demonstration more informative than a bare target string. It contains a judgment about how to interpret the request. Consider a prompt asking for a summary. A useful demonstration should stay within the supplied material rather than add plausible details from elsewhere. A prompt asking for advice may require an answer that is clear without pretending to know facts that were not provided. A request with an explicit format constraint should demonstrate that the constraint matters. The response teaches the model not only what words can follow the prompt, but what kind of relationship should hold between the request and the answer.
The researchers fine-tuned GPT-3 on these prompt-response pairs using supervised learning. They call the resulting model SFT, for supervised fine-tuning. The paper reports roughly 13,000 SFT training prompts in total. In the training split, 11,295 prompts were labeler-written and 1,430 came from customers. The imbalance reflects the bootstrapping role of the labeler interface: labelers were asked not only to answer prompts but, early in the project, to create instruction templates and few-shot examples from which multiple training examples could be constructed.
SFT changes the starting behavior in a direct way. Instead of asking the model to continue text in whatever manner its pretraining makes likely, it rewards responses that resemble the demonstrations. This is why it can produce a substantial improvement before reinforcement learning begins. It gives the policy a vocabulary of task relationships: an instruction is something to execute, an input passage can constrain a summary, and a requested form can govern the answer.
The distinction from pretraining is not that SFT gives the model an entirely new language ability. GPT-3 already contains patterns for stories, explanations, questions, and many other forms of text. SFT changes which pattern is selected in response to a user request and how directly the model treats the prompt as an instruction. A capable completion model may need a carefully designed prefix to enter that mode; the supervised policy is trained to enter it more reliably from the request itself. The gain is behavioral control, not a replacement for the knowledge and linguistic structure acquired earlier.
But SFT is still behavior cloning. It learns from selected examples, not from a complete ordering of all the responses that might be possible. A demonstration can show one good answer without explaining why it is preferable to a shorter answer, a more cautious answer, or an answer that corrects the premise instead of accepting it. It may be difficult to write an ideal response for every prompt, especially when helpfulness, truthfulness, and harmlessness pull in different directions. The model can imitate the visible response while missing the principle that made the response appropriate.
The limitation becomes clearer when a prompt leaves room for judgment. Suppose two answers both satisfy a requested format. One is concise and accurate; the other is longer but includes an unsupported detail that sounds plausible. A demonstration of the first answer shows the desired result, but it does not tell the training process how strongly to prefer it over the second answer, or how to behave when the trade-off changes. Likewise, a refusal may be appropriate for one harmful request and unnecessarily obstructive for another. Written examples can encode these judgments, but they do so sparsely and unevenly. They establish a starting policy; they do not exhaust the space of comparisons that the policy will face.
This is also where the human data-collection system begins to shape the policy. InstructGPT used about 40 contractors for demonstrations, comparisons, and evaluations. The team screened them for agreement with researcher judgments on sensitive speech and response rankings, and gave them detailed instructions. The rubric described useful behavior in terms of clarity, task completion, truthfulness, and harm avoidance. These instructions were not incidental notes around the dataset; they helped determine what a demonstration counted as good.
The priorities could also differ by stage. During training, the labelers were instructed to prioritize helpfulness to the user above truthfulness and harmlessness. In final evaluations, they were asked to prioritize truthfulness and harmlessness. This does not invalidate the results, but it means that the policy and the evaluation were not produced by exactly the same ordering of values. A response can be judged in one context as a helpful attempt and in another as an unsafe or misleading answer. The distinction becomes especially important once the model is optimized to exploit regularities in the judgments.
The SFT policy is therefore an interface between the data distribution and the later reward stage. It is not just a passive record of what labelers wrote. It is the first model that decides which kinds of responses to generate when the comparison process begins. If it produces only bland, short, or obviously defective candidates, later rankings will teach the system little about the difficult boundary between good and better. If it produces a wide range of plausible alternatives, comparisons can reveal more of the trade-off. The quality of the initial policy consequently affects the information available to the next stage.
The policy is capable enough to generate candidate responses that can be compared, yet still retains the limitations of its demonstrations. The next step does not discard those examples. It uses the policy to create alternatives and asks humans to say which alternative is better. That comparative signal can express distinctions that a single demonstration cannot. It is the point at which “this is an acceptable answer” becomes “this response is preferable to that one,” and the preference can begin to function as an objective.
The central lesson of this stage is simple but consequential: InstructGPT did not begin by optimizing an abstract definition of human intent. It began by selecting a field of tasks and teaching a policy through examples written and judged by a particular group. The policy’s later strengths—and some of its blind spots—are already latent in those choices. Before a preference can become a reward, there must be an example of what the system is being asked to improve.