GPT-4: What the Evidence Can—and Cannot—Tell Us
GPT-4 is the subject of this series, but the central question is not simply whether it is “powerful.” The more useful question is what the evidence in the GPT-4 Technical Report and its System Card actually supports: which abilities were demonstrated, under what conditions, with what limitations, and with what safeguards around deployment. That distinction matters because GPT-4 can perform impressively on formal tests while still producing convincing errors and creating risks that a test score cannot measure. This series treats capability, reliability, safety, and deployment context as connected parts of one system.
The report describes GPT-4 as a large-scale, multimodal model that accepts image and text inputs and produces text. It was pretrained to predict the next token in a document and then fine-tuned with Reinforcement Learning from Human Feedback, or RLHF. That description tells us something important about the evidence: the report is primarily a characterization of observed behavior, not a complete disclosure of how the model was built. OpenAI withholds the model size, detailed architecture, hardware, training compute, dataset construction, and similar information. We can therefore examine the results and the stated evaluation methods, but we cannot reconstruct the full training system or treat the report as an independent audit.
One subject, several behaviorally different versions
“GPT-4” does not name one behavior that remains constant across every test. The technical report evaluates a post-trained model on exams and language benchmarks. The System Card focuses especially on GPT-4-early, an instruction-following version with minimal safety mitigation, and GPT-4-launch, a later version shaped for greater helpfulness and harmlessness. The report also distinguishes the pretrained base model from the post-RLHF model, and different exams were run on different snapshots. These distinctions are not administrative details. They determine whether a result describes latent capability, instruction-following behavior, refusal behavior, or the operation of a larger deployment system.
The System Card uses GPT-4-early in many risk examples because that version helps expose what the model can produce before later controls reduce the probability of harmful responses. GPT-4-launch is generally safer on the tested dimensions, but the card does not claim that it is safe in every context. A result from an early model cannot be read as a prevalence estimate for the launch model; a safer refusal cannot be read as proof that the underlying capability has vanished; and an exam score from one snapshot cannot automatically be assigned to every later or earlier version.
This is the first interpretive rule: before asking whether GPT-4 can do something, identify which version, stage, and surrounding system generated the behavior. The report gives us a behavioral map with explicit boundaries. It does not give us a fully inspectable machine.
What the headline results show—and what they leave out
The capability evidence is substantial. In a simulated Uniform Bar Examination, GPT-4 scored 298 out of 400, which the report places around the 90th percentile of test takers. GPT-3.5 was placed around the 10th percentile. GPT-4 also scored around the 99th percentile on GRE Verbal and performed strongly on several science, history, mathematics, and professional examinations. On academic benchmarks, the report lists 86.4% on MMLU, 67.0% on HumanEval, and 92.0% on GSM-8K under specified few-shot or chain-of-thought procedures.
Those numbers support a meaningful conclusion: under a range of formalized tasks, GPT-4 shows unusually broad competence and often improves substantially over earlier language models. They do not support the conclusion that GPT-4 is uniformly expert, generally reliable, or equivalent to a human professional in open-ended work. The same table reports a GRE Writing result around the 54th percentile, a Codeforces rating below the 5th percentile, and only limited success on hard Leetcode problems. A system can excel at recalling and organizing knowledge in an exam format while struggling with sustained implementation, judgment, or tasks where small errors compound.
The score is also inseparable from its protocol. The exams were simulations using public past exams or published practice material. The researchers iterated on methodology using non-holdout material and used held-out exams for final scoring where possible. Multiple-choice questions received few-shot explanations; free responses were sampled once and graded according to exam-specific rubrics or by qualified contractors. Different snapshots and sampling methods were used in different evaluations. Contamination checks removed some examples, but the report acknowledges that substring matching can produce false positives and false negatives. GSM-8K is especially qualified: related math benchmark data entered training, so the result lies between true few-shot transfer and full benchmark-specific tuning.
The same discipline applies to breadth claims. GPT-4 performed strongly on translated versions of MMLU in many languages, including Latvian, Welsh, and Swahili, but the questions and answers were translated by an external system. Translation imperfections, preserved English proper nouns, and a three-shot protocol can all affect the result. GPT-4 also accepts interlaced text and images and can answer questions about documents, diagrams, photographs, and screenshots. The report calls the visual evidence preliminary and narrow, while the System Card explicitly leaves image safety out of scope. Modality and language breadth are real findings, but they are not evidence of uniform performance or uniform safety.
The evidence chain can be summarized as follows:
The diagram is not a claim that every result passes through one literal software pipeline. It is a reading model. A capability observation becomes meaningful only when its version and protocol are known; it becomes practically useful only after reliability and context are considered; and a deployment judgment requires more than either capability or safety evidence in isolation.
Capability does not cancel unreliability
The report is unusually direct about this tension. GPT-4 remains vulnerable to hallucination: it can produce content that is nonsensical or untruthful relative to a source, or confidently state false information about the world. It can make simple reasoning errors, accept obviously false premises, fail to double-check its work, and introduce security vulnerabilities into code. Its knowledge of events is generally limited by a pretraining cutoff in September 2021, and it does not learn from experience during use.
There is evidence of improvement. On internal adversarial factuality evaluations, GPT-4 scored 19 percentage points higher than the latest GPT-3.5. The System Card reports 19- and 29-point gains on open- and closed-domain hallucination evaluations for GPT-4-launch relative to its comparison model. On TruthfulQA, post-training produced a large improvement over GPT-3.5, although the base GPT-4 model was only slightly better on that task. These results establish progress on selected measures. They do not establish a universal error rate.
Calibration adds another complication. In the reported MMLU subset, the pretrained model’s confidence was relatively well matched to its accuracy, while post-training worsened that calibration measure. A model can become more helpful, more aligned with user intent, and more likely to refuse disallowed requests without becoming a better judge of when its own answer is correct. The authoritative tone that makes a response useful can also make a mistake harder to detect. Users with little subject expertise may be least able to supply the missing check.
This is why overreliance is not a side issue. If a system is often right, users reasonably learn to trust it. If it is also fluent and detailed, they may stop checking it precisely when the remaining errors are most consequential. The relevant question is not whether GPT-4 is “smart” or “unreliable” in the abstract. It is whether a particular use has enough grounding, human review, domain expertise, and tolerance for error to make the observed reliability acceptable.
The System Card asks a larger question
The System Card extends the inquiry from “What can the model answer?” to “What can this model enable in a real environment?” More than 50 external experts tested areas including alignment, cybersecurity, biorisk, international security, fairness, disinformation, law, education, healthcare, and economics. The process combined qualitative red teaming with quantitative evaluations using targeted prompts, classifiers, and human analysis. It was iterative rather than exhaustive, and the authors acknowledge that the selection of experts—many connected to English-speaking Western professional settings—shaped which risks were explored and how they were interpreted.
The reported risks range from harmful content and social bias to targeted disinformation, privacy linkage, cybersecurity assistance, dual-use information, and possible future agentic behavior. The mechanism is often an interaction between capability and context. More coherent language can make persuasion more credible. Combining several ordinary pieces of public information can make it easier to identify a person. A model that is weak at building a novel exploit may still help draft targeted social engineering or summarize information gathered during an intrusion. Connecting a model to search, external tools, or human operators can create a system with capabilities that a model-only benchmark never measured.
The mitigations follow the same systems logic. Data filtering, supervised fine-tuning, reward modeling, RLHF, rule-based reward models, classifiers, usage policies, monitoring, human review, and product documentation address different parts of the risk surface. The reported reductions in disallowed and toxic outputs are meaningful improvements on selected tests, but the System Card also records brittleness, over-refusal, biased moderation, jailbreaks, language gaps, and the possibility of false assurance. A refusal is a behavior change, not proof that the capability is gone; a classifier is a control with its own error modes; and a policy is not the same thing as technical prevention.
The practical judgment supported by this series is therefore conditional. GPT-4’s report provides strong evidence of broad, uneven capability and evidence of meaningful safety progress. It also provides equally important evidence that benchmark success, fluent answers, and safer default behavior do not settle questions of reliability or deployment readiness. To read GPT-4 well, we must follow the evidence from model stage to test protocol, from result to failure boundary, and from model behavior to the people, tools, incentives, and safeguards that surround it. The next step is to understand how the model’s lifecycle and scaling work make some of those observations predictable—and leave others open.