A Calibrated Judgment About GPT-4

The central question for GPT-4 is not whether the system is capable or incapable. It is what kind of capability the evidence supports, how reliably that capability can be used, which risks appear when it enters a larger system, and whether the available controls fit the intended context. This final judgment brings those questions together. It treats the Technical Report and System Card as a conditional evidence record: strong enough to establish broad and uneven capability, candid enough to show serious limitations and risk surfaces, and incomplete enough that deployment decisions must remain bounded, monitored, and open to revision.

What the report establishes about capability

The positive conclusion is substantial. GPT-4 demonstrated strong performance across a wide range of formal tasks. In the report’s simulated Uniform Bar Examination, it scored 298 out of 400, around the 90th percentile of test takers. It reached approximately the 99th percentile on GRE Verbal and performed strongly on many AP, science, reading, and mathematics examinations. On academic benchmarks, the reported results include 86.4% on MMLU, 67.0% on HumanEval, and 92.0% on GSM-8K under specified prompting conditions. GPT-4 also showed meaningful performance on translated MMLU questions and demonstrated image-and-text interactions.

These results justify describing GPT-4 as broadly capable under many defined evaluation conditions. They show that a single large multimodal model can perform well across tasks that previously required separate demonstrations of academic knowledge, language understanding, coding, and visual input. The scaling work adds a further positive finding: some outcomes, including final loss and HumanEval performance in most analyzed difficulty buckets, could be predicted from much smaller runs before the final training process was complete. That predictability can improve the timing of evaluation and preparation.

The claim becomes weaker as it moves away from the tested task. GPT-4’s GRE Writing result was around the 54th percentile, its Codeforces rating was below the 5th percentile, and hard Leetcode performance remained limited. Its visual evidence was preliminary and narrow. Its multilingual results were mediated by translation quality and a three-shot protocol, and the report did not establish equal performance or safety in every language. The report’s comparisons also involve simulated exams, particular model snapshots, prompting procedures, scoring choices, and contamination checks.

The defensible capability statement is therefore precise: GPT-4 displays broad, uneven competence on many formalized tasks, and some of its performance trends can be forecast under the tested scaling setup. That statement is stronger than a product description and narrower than a claim of general intelligence, uniform human equivalence, or professional substitution. A bar-exam percentile is evidence about a bar-exam simulation. A HumanEval score is evidence about the benchmark’s coding problems. A visual example is evidence about a demonstrated interaction. The task boundary travels with the result.

The report’s disclosure boundary matters too. It does not provide model size, full architecture, training compute, dataset construction, or enough information to reproduce the training system independently. The behavioral evidence is still valuable, but readers should not infer an undisclosed mechanism from the observed score or treat the developer’s characterization as an independent audit. What can be concluded is what the model did under the recorded conditions, not every reason why it did it or every situation in which it might behave similarly.

What the report establishes about reliability and safety

The positive capability conclusion must be paired with a negative one: GPT-4 is not fully reliable. It can hallucinate in both closed-domain and open-domain settings, make reasoning errors, accept false premises, fail to double-check likely mistakes, and produce vulnerable code. Its knowledge of events is generally limited by a pretraining cutoff in September 2021, and it does not learn from experience during use. A more coherent answer can make these errors harder to detect, particularly for users who lack the expertise or time to verify them.

There is meaningful progress. The technical report reports a 19-percentage-point improvement over the latest GPT-3.5 on internal adversarial factuality evaluations. The System Card reports gains in avoiding open- and closed-domain hallucinations for its launch comparison. Post-training produced a large improvement over GPT-3.5 on TruthfulQA, while the base GPT-4 model was only slightly better on that task.