From Next-Token Prediction to Predictable Scaling
To interpret GPT-4’s later exam and benchmark results, it helps to separate two questions: how the model acquires broad learned behavior, and how its developers learned to anticipate some changes as training scale increased. This chapter follows GPT-4 from next-token pretraining through post-training and evaluation. The practical goal is not to reconstruct undisclosed architecture, but to understand which transition produces which kind of evidence—and why a successful forecast for one metric does not make every capability predictable.
The documented starting point: predict the next token
The technical report describes GPT-4 as a Transformer-style model pretrained to predict the next token in a document. A token is a unit of text used by the model’s processing system; the training objective asks it to estimate what unit is likely to follow the preceding context. Repeated across a very large collection of text, this objective exposes the model to patterns in language, facts, styles, relationships, and code. The report does not disclose the full dataset construction or the precise architecture, so the safe conclusion is limited: pretraining establishes a broad base of learned behavior, but the report does not let readers explain every capability through a fully visible mechanism.
A continuation objective also explains why the same underlying model can support many apparently different tasks. The input context establishes a local problem, and the model generates a continuation that fits the patterns it has learned. In one setting that continuation may look like an answer; in another it may look like a program, a summary, or a description of an image. The observable task depends on the prompt, the input representation, the sampling procedure, and the instructions around the model. The report’s wording supports a model capable of many such behaviors, but it does not license the stronger claim that every behavior reflects a stable, human-like skill.
That distinction matters because “predict the next token” describes the optimization target, not the whole way users encounter GPT-4. A model trained on continuation can be prompted to answer a question, write an explanation, solve a problem, or analyze an image when the surrounding interface and later training shape the interaction. The target is a foundation from which many behaviors can be elicited; it is not itself a guarantee that an answer will be accurate, useful, or safe. A fluent continuation may fit the immediate context while still containing a factual or reasoning error. The training objective and the user’s desired outcome are related, but they are not identical measures.
The report also places an explicit boundary around what can be inferred from this lifecycle. OpenAI says that it does not provide model size, hardware, training compute, dataset construction, training method details, or similar information. Those omissions prevent a reader from independently reproducing the run or attributing a particular result to a specific architectural component. What remains available for analysis is the relationship between training stages and observed behavior: broad capability appears in the pretrained model, while post-training changes how that capability is expressed in response to instructions and safety-sensitive requests.
The transition from base capability to directed behavior
The first important state is the pretrained base model. It is the result of next-token training before the behavior-shaping process described as RLHF. This state matters because it provides a comparison point: the report can ask whether a post-trained model performs differently from the base model on a task, and the System Card can distinguish latent capability from the probability that the launch system will expose it in a given interaction.
The next state is post-training. RLHF uses human feedback to steer responses toward desired behavior, including following user intent and handling sensitive or disallowed requests. The report’s evidence suggests that this transition does not simply create all of GPT-4’s capabilities from scratch. On the multiple-choice portions of its exam evaluations, the base model averaged 73.7% and the post-RLHF model averaged 74.0%, a small difference in that particular comparison. The authors therefore conclude that post-training did not substantially alter base capability on those tests. They also caution that free-response comparisons are harder because the sampling method favors the instruction-following ability of the post-trained model.
The comparison has an important asymmetry. Multiple-choice questions make it relatively easy to hold the answer space constant and compare whether the base and post-trained systems select the same correct option. Free-response tasks expose another property: whether the model can understand the instruction, choose an appropriate form, sustain an answer, and stop at a usable point. A post-trained model may look more capable in such a setting partly because it is better at converting the task into a response that a grader can evaluate. That does not make the result invalid; it means the evaluation is measuring the interaction between knowledge and directed behavior rather than knowledge alone.
The implication is not that RLHF is unimportant. It changes the response policy: whether the model follows an instruction, refuses a harmful request, hedges, or presents an answer in a form a user can use. Those are deployment-critical properties even when a multiple-choice score remains similar. A base model and a post-trained model may contain comparable task knowledge while differing in how readily that knowledge is made available, how they respond to ambiguity, and how they handle safety constraints. Capability and behavior are related states, not interchangeable labels.
The System Card makes the distinction sharper by comparing GPT-4-early with GPT-4-launch. The early version was used to surface risks with minimal safety mitigation. The launch version incorporated further changes intended to improve helpfulness and harmlessness. A harmful completion from the early version is evidence that a risk surface can be elicited under that setup; it is not a direct estimate of how often the launch model will produce the same completion. Conversely, a launch refusal shows that the later behavior is safer on the tested request, not that the pretrained capability has been erased. Fine-tuning can steer behavior, but it does not turn every possible response into an impossible response.
These version labels also tell us what persists across a transition. The underlying trained model remains the source of the broad behavior being directed. The post-training stage changes preferences and response tendencies, and the launch system adds further controls around use. What is lost or changed is not necessarily knowledge; it may be access, presentation, refusal, or the likelihood of a particular completion. That is why a behavioral comparison should say whether it is testing what the model can produce under pressure or what the deployed interface ordinarily returns.
This is a lifecycle rather than a simple before-and-after story. At each transition, some properties persist and others change. Broad learned capability persists as the foundation. Instruction-following and refusal behavior are shaped after pretraining. The evaluation protocol then selects which part of the system becomes visible. If a reader collapses these states into one thing called “GPT-4,” they can easily attribute a benchmark result to the wrong process or treat one safety example as a property of every version.
The diagram shows two related paths. Training stages produce the model state that will answer a prompt. Smaller runs provide information for forecasting selected outcomes before or during the final run. Neither path gives direct access to the full internal mechanism, and neither forecast substitutes for evaluating the final behavior.
Why predictable scaling changes the engineering decision
Training a very large model is expensive and difficult to adjust after the run begins. If developers can learn in advance how a target metric will change with training scale, they can make better decisions about evaluation, alignment, and deployment before committing to the largest run. GPT-4’s report presents predictable scaling as an engineering achievement for that reason, not merely as an observation about curves on a graph.
For final loss, the team fitted a power law with an irreducible loss term to smaller models trained with the same methodology. The fit used runs with as much as 10,000 times less compute than GPT-4 and predicted the final loss on an internal codebase dataset with high accuracy. The timing of the prediction is important: it was made shortly after the run began and did not use partial results from the final run. This reduces one obvious way of making a forecast look accurate after the fact.
Loss is useful because it is relatively smooth and less noisy than many task metrics. It provides a compact signal about how well the model predicts held-out material in the tested setting. But a lower prediction error on code tokens is not the same as a prediction of how a user will experience the model. A smooth aggregate can coexist with uneven performance on individual tasks, and an internal codebase measure is not itself a direct measure of professional coding or safe software production. The report therefore treats loss as an engineering diagnostic and supplements it with more interpretable capability measures.
The report’s second example is HumanEval, a coding benchmark that measures whether the model can synthesize Python functions. The researchers registered predictions before training completed and extrapolated from models using up to 1,000 times less compute. For most of the difficulty buckets they analyzed, the predictions were accurate enough to demonstrate that at least some interpretable capability metrics can be forecast from smaller runs. Registering the prediction in advance matters because it makes the forecast a testable expectation rather than a curve fitted after the final score is known.
Advance knowledge of a capability can affect the order of operations. If a small-scale trend indicates that a final model may cross a meaningful performance level, developers can expand safety testing before release rather than discover the change only after deployment. If the trend is uncertain, they can reserve more evaluation capacity for the final model. In this sense, scaling forecasts connect measurement to governance: they can inform when to test and what to prepare for, even though they do not establish that a model is ready to use.
The forecast also has a more subtle value. It gives decision-makers a way to distinguish a capability that was surprising only in the public narrative from one that was genuinely difficult to anticipate during development. A result that can be predicted from smaller runs may still be socially important, but it is different operationally from a capability that appears abruptly or behaves irregularly. The report’s aspiration is to register predictions across more capabilities before large training runs begin, making the field more accountable to forecasts rather than retrospective explanation.
Where extrapolation stops working
The report is careful not to turn successful scaling results into a universal law. On individual HumanEval problems, performance can occasionally worsen as scale increases. The predictions were less successful on the easiest difficulty bucket, where GPT-4 underperformed the forecast. The authors also examine an inverse-scaling task called Hindsight Neglect. Earlier models showed a pattern in which performance declined with scale, but GPT-4 reversed that trend. A relationship that looked stable across smaller systems did not remain stable at the next scale.
These exceptions matter because safety-relevant capabilities may not behave like average loss. A mean or aggregate curve can conceal a rare task that suddenly becomes easier, a behavior that becomes more persuasive, or an interaction that only appears when the model is connected to tools. Even within a single benchmark, a model can improve on the average while regressing on a particular problem. The report’s positive forecast is therefore evidence about selected metrics and difficulty groups, not evidence that unusual behavior is absent.
The distinction between forecasting and evaluation is operational as well as epistemic. Forecasting tells a team where a result may land before the final model exists. Evaluation tests what the completed model actually does, including cases that the forecasting family did not represent. A responsible process needs both: prediction can create time to prepare, while targeted testing can search for non-monotonic or emergent behavior that the prediction did not capture.
The right mental model is therefore conditional extrapolation. A forecast says, in effect, “given this training method, this family of smaller runs, this metric, and this range of scales, we expect this outcome.” It does not say, “all important properties of GPT-4 will change monotonically and predictably.” The exceptions are not a failure of the entire approach; they identify where targeted evaluation remains necessary.
GPT-4 enters an evaluation with both a history and a boundary. Its pretraining supplies broad learned capability, post-training shapes how that capability is expressed, and scaling work can provide advance warning for selected outcomes. The next question is how those outcomes were measured. Once the model is placed in an exam or benchmark, the prompt, snapshot, scoring method, and contamination status become part of the result itself.