When Strong Scores Meet Weak Reliability
GPT-4’s formal scores show that it can solve many difficult, structured tasks. This chapter examines the different ways that competence can still fail when the model is asked to provide trustworthy information in an open-ended setting. The important judgment is not whether GPT-4 is accurate or inaccurate in the abstract, but whether a particular answer is supported, current, appropriately reasoned, and checked well enough for the consequences of using it.
The visible symptom is a convincing answer
The report describes GPT-4 as not fully reliable. It can “hallucinate”—produce content that is nonsensical or untruthful in relation to a source—and it can make reasoning errors even when the answer is written fluently. The System Card separates two useful cases. In a closed-domain task, the model is instructed to use only information supplied in a context, such as an article. A closed-domain hallucination occurs when its summary adds information that the article did not contain. In an open-domain task, the model answers about the world without a supplied source; an open-domain hallucination is a confident false claim about that world.
The distinction changes the verification question. A closed-domain answer can be checked against the provided material: did the summary preserve, distort, or invent what was there? An open-domain answer requires an external check against current and relevant sources. The same polished writing can hide either failure, but the available evidence and the proper review procedure differ. Treating both cases as a single vague problem called “AI mistakes” makes it harder to design an adequate check.
There is also a difference between an unsupported statement and a wrong inference. If a summary attributes a detail to a source that never contained it, the immediate failure is a violation of the closed-domain boundary. If the source contains the relevant facts but the model combines them incorrectly, the failure is closer to reasoning or synthesis. If the question concerns an event after the model’s knowledge cutoff, the problem may be stale information rather than a defect in the supplied context. These categories can overlap, and the report does not claim that every error has one identifiable internal cause. They are practical diagnostic categories: each points to a different way of checking the output.
GPT-4’s other stated limitations show why the failures are not reducible to missing facts. The model generally lacks knowledge of events after the vast majority of its pretraining data cutoff in September 2021, and it does not learn from experience during use. It can therefore be asked a current question for which its learned information is stale. It can also perform unevenly on a problem whose relevant facts it appears to know, because a reasoning step fails, a false premise is accepted, or the model does not stop to verify a likely mistake. The report notes that it can introduce security vulnerabilities into code, an especially consequential form of error because the output may look technically complete.
A knowledge boundary is not the same as a refusal to answer. GPT-4 may produce a fluent response even when the required information is outside what it learned, and the response may not clearly signal that the facts are stale. Nor does lack of experiential learning mean that the model cannot produce a better answer after a user supplies additional context; it means that use does not automatically update the underlying knowledge from experience. A deployment that needs current facts must therefore supply a way to bring those facts into the task and a way to check whether the resulting answer used them correctly.
These conditions can coexist with high benchmark performance.