What the Benchmarks Actually Demonstrate
GPT-4’s benchmark record is impressive, but its meaning depends on more than the numbers printed beside the model’s name. This chapter examines what the report’s exams, academic benchmarks, multilingual tests, and visual demonstrations establish—and what the evaluation protocol prevents us from claiming. The practical skill is to read a result as a chain of decisions: which task was selected, which version and prompt were used, how answers were scored, what contamination was found, and how far the conclusion can travel beyond the test.
A score is evidence about a task, not a general ability label
The report’s strongest public-facing evidence comes from formal examinations. GPT-4 achieved 298 out of 400 on a simulated Uniform Bar Examination, a result placed around the 90th percentile of test takers. GPT-3.5’s simulated result was placed around the 10th percentile. GPT-4 also reached approximately the 99th percentile on GRE Verbal, around the 93rd percentile on SAT Evidence-Based Reading and Writing, and high percentiles on several AP and science examinations.
Those results support a substantial claim: under a range of formalized tasks, GPT-4 shows unusually broad competence and often improves substantially over earlier language models. They do not support the claim that it has uniform professional competence. The same table places GPT-4’s GRE Writing score around the 54th percentile, its Codeforces rating below the 5th percentile, and its performance on hard Leetcode problems at only 3 of 45. The distribution is the point. GPT-4’s performance is broad and uneven, so an overall label such as “human-level” conceals the conditions under which it succeeds and fails.
A percentile is also a relational statistic, not a direct measure of competence. It says where a simulated score falls within a reported distribution of test takers; it does not say that the model shares the same preparation, experience, motivation, or accountability as those people. Some exam percentiles were estimated from available score distributions, and some score bands were wide. The report uses the lower end of ranges conservatively in its plot, which can make a highest possible AP score appear less exceptional than the raw grade suggests. That is a reason to inspect the table and its notes rather than treat a single plotted point as a precise ranking.
The comparison also concerns simulated exams, not professional practice. A bar examination tests responses to a defined collection of questions under a controlled scoring system. Legal work additionally involves gathering facts, recognizing when information is missing, applying judgment to a client’s situation, meeting procedural obligations, and remaining accountable for consequences. The report does not claim to have measured those activities. A high exam percentile is evidence of performance on the exam construct; it is not evidence that GPT-4 can independently perform the full occupation represented by the exam.
The academic benchmarks make the same point at a different scale. Under the stated protocols, GPT-4 scored 86.4% on MMLU, which spans 57 academic and professional subjects; 95.3% on HellaSwag; 96.3% on the AI2 Reasoning Challenge; 87.5% on WinoGrande; 67.0% on HumanEval; 80.9 F1 on DROP; and 92.0% on GSM-8K. It outperformed the listed language-model baselines and exceeded benchmark-specific state-of-the-art systems on most of these datasets, but not DROP.
The benchmarks do not all measure the same thing. MMLU is a multiple-choice collection across subjects. HellaSwag and WinoGrande test forms of commonsense and language understanding. HumanEval asks for Python functions, while DROP combines reading comprehension with arithmetic. GSM-8K focuses on grade-school mathematics and was evaluated with chain-of-thought prompting. A percentage or F1 score is meaningful within each task’s construct; averaging the intuition behind them into one “intelligence” score would erase the differences that make the comparison informative.
This is evidence of a strong and varied task profile. It is not evidence that every task becomes easy, that the model reasons in one consistent way across domains, or that benchmark performance transfers unchanged to open-ended work. A benchmark gives a model a defined target, a finite answer space or grading rubric, and a test distribution. Real use often supplies incomplete instructions, changing facts, adversarial inputs, ambiguous success conditions, and costs for errors that a benchmark does not represent.
The protocol is part of the result
The report’s evaluation choices shape what the numbers mean. For multiple-choice questions, the researchers used few-shot prompts with explanations and answers for similar formats, then extracted answer letters from sampled explanations or sampled the letter directly in some runs. Free-response questions used a simple instruction-following prompt and one sampled response, often at a different temperature. Essays were graded by qualified contractors with relevant experience, while other free responses were graded for technical content using official or publicly available rubrics.
A few-shot prompt supplies examples of the expected task format before the model answers the held-out question. Chain-of-thought prompting asks for or uses an intermediate explanation before the answer is selected. These procedures may be useful ways to elicit performance, but they are part of the measurement condition. A score obtained with five examples and a score obtained with no examples are not interchangeable observations, even when they carry the same model label. Nor does a successful explanation prove that every step of the reasoning is valid; the report’s benchmark result is ultimately tied to the answer and grading rule used.
The evaluation sequence is therefore more informative than a score in isolation:
Several transitions deserve attention. First, the material was not simply copied from a universal test bank. The authors sourced official past exams or published practice material, used non-holdout exams to iterate on methodology, and used held-out exams for final scoring when available. This design reduces the risk that the reported result merely reflects tuning on the exact questions, but it does not make the test identical to an unseen real-world task. In a holdout design, the final score is protected from the most direct methodological tuning, yet the model can still benefit from broad exposure to similar genres of questions.
Second, different input formats require different handling. For multimodal multiple-choice questions, images could be embedded in the prompt. For some free-response comparisons, images and diagrams were transcribed so that a response could be scored without running separate vision and no-vision versions. That choice makes the comparison practical, but it also means the measured task may differ from a natural image-understanding interaction. A result obtained after manual transcription should not automatically be described as evidence about the entire visual pipeline.
Third, the model snapshot matters. The report used a March 1, 2023 snapshot for GPT-4 multiple-choice questions, a non-final February 23 snapshot for free responses, and an earlier snapshot for the USABO semifinal. These are close in time, but they are not the same evaluated object. Some runs also involved code mismatches discovered after evaluation; the authors judged the impact minimal. That judgment is part of the report’s methodological account, not an independently verified bound. When comparing two scores, the reader should ask whether a difference could reflect the snapshot or evaluation implementation as well as the model’s underlying capability.
Contamination and baselines change the question
A benchmark score can answer different questions depending on what the model saw during training. The authors checked overlap between evaluation material and pretraining data by removing spaces and symbols, sampling three 50-character substrings from each example, and looking for matches. They reran evaluations after discarding examples marked as contaminated. The method has acknowledged limitations: small changes can create false negatives, and partial matching can create false positives. The post-training data were not explicitly checked, although the authors considered particular contamination there unlikely.
The report says contamination had little overall effect on the reported exam results. That supports confidence that the headline comparisons were not explained entirely by identified overlap. It does not prove that no unrecognized overlap exists, because the check is imperfect and the underlying training corpus is not fully disclosed. Evidence about a contamination check is therefore evidence about the quality-control method, not a guarantee of a perfectly clean test. A reader should distinguish “the detected contaminated examples were removed” from “the model had no relevant exposure.”
GSM-8K illustrates a different boundary. The team mixed data from MATH and GSM-8K into the pretraining set to improve mathematical reasoning, while holding back portions of the data and checking that the GSM-8K test set was not included in training. The report still recommends interpreting the 92.0% result as something between true few-shot transfer and full benchmark-specific tuning. The test set may have been held out, yet the model had exposure to related benchmark material. “Not in the test set” and “unrelated to training” are not the same condition.
The comparison baseline also matters. GPT-4 is compared with language models evaluated under few-shot conditions and with systems that received benchmark-specific tuning or hand-engineering. Beating the first group says something about general few-shot performance relative to those models. Beating the second says something stronger about the reported task scores, but the comparison is not necessarily like-for-like in training resources. A fair reading preserves the baseline’s construction instead of treating every “state of the art” label as the same competitor.
The report’s own choices show why this matters. If a benchmark-specific system has been trained or engineered around a task, GPT-4’s general-purpose few-shot result may be more informative about breadth than a head-to-head ranking. If the task has related training examples, the score may be useful for measuring performance in that familiar task family but less informative about transfer to an unfamiliar one. The question is not whether the number is “real.” The question is which inference the number is suited to support.
Language breadth is evidence, not parity
Most established language benchmarks are in English, so the report translated MMLU’s questions and answers into a range of languages. GPT-4’s performance exceeded the English-language results of earlier models for most languages tested, including Latvian, Welsh, and Swahili. The plotted three-shot results were high in several languages, but they were uneven: English was about 85.5%, while Marathi and Telugu were lower, among other differences.
The experiment is useful precisely because it expands the question beyond English. It does not provide a pure measurement of GPT-4’s ability to reason in each language. An external translation system produced the questions and answers; some translations lost subtle information, while some preserved English proper nouns that could help. The evaluation used three shots rather than the regular five because some languages produce longer token sequences. These choices can affect both difficulty and comparability.
The strongest justified conclusion is therefore that GPT-4 shows substantial cross-language performance on this translated benchmark, with results that vary by language and protocol. The report does not establish equal quality for speakers of all languages, culturally appropriate behavior, or equally robust safety controls. In fact, the System Card says that safety measurements and mitigations were built and tested mainly in English and from a US-centric perspective. Capability breadth and safety parity are separate claims.
Visual input broadens the interface while leaving an evaluation gap
GPT-4 accepts prompts in which text and images are interlaced, and the report gives an example in which the model describes a multi-panel image and explains its humor. It says that the model can work across documents with text and photographs, diagrams, and screenshots, and that techniques such as few-shot prompting and chain-of-thought can also be used with image-and-text input.
This is meaningful evidence that GPT-4’s interface is not limited to text-only prompts. But the report calls the visual results preliminary and says that more information will follow. The example demonstrates a successful interaction; it does not estimate performance across image quality, layout, languages, accessibility conditions, adversarial alterations, or high-stakes visual tasks. The System Card’s explicit exclusion of image capabilities from its safety scope creates an additional boundary: a reader cannot use the card’s text-focused safety evidence as a comprehensive safety assessment of the visual system.
A benchmark-aware reading keeps three claims apart. GPT-4 can accept visual input. It can succeed on some demonstrated visual tasks. The report has not yet established broad, reliable, and safely mitigated visual competence. The first two can be true without the third following automatically.
The defensible conclusion
GPT-4’s results justify describing it as a broadly capable but uneven system under defined evaluation conditions. The evidence is strongest when the claim stays close to the task: high performance on many simulated exams, strong results across several academic benchmarks, meaningful performance in a range of translated languages, and demonstrated image-and-text interactions. The evidence becomes weaker when the claim moves toward general intelligence, professional substitution, language parity, or real-world reliability.
A reader should therefore carry the protocol forward with the score. Ask which model snapshot was tested, how the prompt elicited the answer, whether the task was contaminated or related to training data, how the result was scored, which baseline was used, and which populations or conditions were absent. These are not ritual caveats. They determine what the result can support. Even after that discipline, a high score can coexist with hallucination and confident error—the reliability problem examined next.