The First Checkpoint: A Conditional Picture of Capability
The evidence about GPT-4 now supports something more useful than a single verdict. It supports a method for deciding what a result means: identify the task, preserve the conditions that produced the observation, state how far the result can transfer, and then ask whether the model is reliable and appropriate for the intended use. This checkpoint connects GPT-4’s scaling forecasts, benchmark results, protocol caveats, and reliability limits into that method before the analysis turns to concrete safety risks.
Four questions that travel with every result
The first question is the narrowest: what task did GPT-4 perform? A simulated Uniform Bar Examination score of 298 out of 400, reported around the 90th percentile, is evidence about performance on that examination under its scoring rules. A Codeforces rating below the 5th percentile is evidence about a different task involving repeated competitive programming contests. These observations can both be true because they probe different combinations of knowledge, response format, sampling conditions, sustained implementation, and error tolerance. Neither result needs to be discarded to make the other seem plausible.
The contrast is useful because the two tests reward different forms of success. A formal exam can present a finite question set and an established rubric. A programming contest requires a sequence of executable solutions, exposes failures that cannot be repaired by persuasive prose, and aggregates performance across many problems and attempts. The report’s result does not tell us that one test is more “real” than the other. It tells us that a capability claim is always indexed to a task construct. If the intended use resembles a contest, a bar-exam percentile may be relevant background but insufficient direct evidence.
The second question is how the observation was produced. The report’s scaling work used smaller runs to predict selected outcomes, while its exams and benchmarks used particular snapshots, prompts, sampling procedures, graders, and contamination checks. A benchmark score is therefore not a property that floats free of its measurement system. It is the output of a model in a defined state placed inside a defined evaluation. Changing the prompt, the input modality, the shot count, the scoring rule, or the relationship between training and test material can change what the score represents.
The third question is the transfer boundary. A result may support a claim about a formal task without supporting a claim about an open-ended occupation or current-world problem. GPT-4’s strong scores on MMLU and HumanEval establish performance on those datasets and procedures; they do not establish that every professional judgment or software task will be completed with the same reliability. The multilingual MMLU results support substantial performance on translated questions, not equal quality in every language. A visual demonstration supports the ability to answer a particular image-and-text prompt, not comprehensive visual competence or safety.
Transfer is not all-or-nothing. Some parts of a tested skill may carry into a new setting while other requirements do not. A model that can explain a small code sample may help an engineer inspect a larger system, but the report’s observation does not establish that it will identify every vulnerability or produce secure code. A model that answers an exam question may help a student explore a concept, but that answer does not establish that it knows the latest developments in the field. A narrow transfer claim can be reasonable; an unqualified substitution claim is not.
The fourth question is decision relevance. Even a well-measured capability may be unsuitable for a use if errors are difficult to detect, facts change, affected people cannot consent, or the output controls an important decision.