The Confidence Gap Begins with the System
AI literacy is tested at the boundary between a learner and an AI system: what happens when a tool produces an answer that feels more certain than the evidence warrants? This chapter explains how probabilistic generation, fluent language, limited transparency, and human-like conversation can make confidence an unreliable guide to competence. The practical goal is to help readers separate a plausible output from a trustworthy one and identify what they need to understand about a system before relying on it.
The problem does not begin with careless users. It begins with a mismatch between how AI systems work and how their outputs are experienced. A system can return a polished explanation in seconds. The user experiences clarity, speed, and conversational cooperation. Those experiences are real, but they are not the same as proof that the system understood the question or that the answer is correct.
A system can sound certain without understanding the task
The OECD/European Union framework describes AI systems as using algorithms and statistical inferences to process data, detect patterns, and generate outputs such as predictions, content, or recommendations. Generative AI can produce text, images, audio, and other forms of content that appear original. The framework’s crucial qualification is that generative systems lack authentic human understanding and intent. They can produce a useful result without possessing the kind of subject understanding a learner brings to a lesson, an argument, or a lived situation.
That distinction is easier to grasp when the process is reduced to its essential sequence. A person presents a goal, question, or set of constraints. The system processes the input through the patterns and design choices that shape it. It then generates an output that fits those patterns and the requested form. The answer may be accurate, partly accurate, or fabricated. The generation process does not announce which category the user has received simply by sounding fluent.
This is why plausibility and truth must be treated as separate properties. A response can be plausible because its wording follows familiar patterns, its explanation has a neat structure, or its details resemble things the learner has seen before. Those features may help the answer communicate. They do not establish that its claims are grounded in reliable evidence. The same smooth style can carry a correct explanation and a serious error.
The issue is not limited to dramatic “hallucinations.” A system may omit a condition from a calculation, flatten an important distinction, recommend something that does not fit the user’s circumstances, or combine individually familiar facts into a misleading conclusion. A learner who knows the subject may notice the problem quickly. A learner who does not may experience the answer’s structure as evidence of understanding. Confidence then reflects the presentation of the answer more than the learner’s ability to evaluate it.
The framework also notes that outputs can vary with the input and with how a system is designed to select or prioritise features and parameters. This matters for learning because a successful first interaction can create an overgeneralised expectation: “The tool handled that, so it will handle this.” But a system that performs well on a narrow, familiar task may be unreliable when the task changes, when context is missing, or when the consequences of an error are higher. Familiarity with one pattern of use is not the same as understanding the system’s range.
Limited transparency makes this harder. Some aspects of an AI system may be explained well enough for a user to understand its purpose and constraints; other aspects may be difficult or impossible to interpret. The user may see the input and the answer without seeing the internal route between them. That does not mean every output is untrustworthy. It means the user cannot treat the system’s apparent confidence as a substitute for an independent basis of judgement.
The diagram shows the measurement problem. Confidence enters after the system’s presentation of an answer, before the learner has necessarily checked it. A learner may be confident because the result is correct, but they may also be confident because the route to the result is effortless and the language is polished. Without observing what they do at the verification branch, we cannot tell those cases apart.
A confidence survey may therefore be measuring the wrong stage of the interaction. It can reveal whether a learner feels ready to use a system. It cannot reveal whether they know when a system is appropriate, whether they can identify a missing assumption, or whether they would change course after an error. Those are not invisible philosophical qualities. They are behaviours that can be elicited and examined.
Fluency, warmth, and opacity can lower scrutiny
Language adds a second layer to the problem. AI systems do not need to claim that they are human in order to be experienced as conversational partners. A polite reply, a reassuring explanation, or an apology after a mistake can make the exchange feel cooperative. The user may begin to respond according to social expectations: explaining more, expressing gratitude, or trying to coax the system toward a better answer. Conversation can be useful for clarifying a goal, but it can also change the user’s standard of evidence.
The naturalistic study of student–ChatGPT interaction describes this pattern as Anthropomorphic Sensemaking. In the researchers’ corpus, students often used polite language and sometimes addressed the system in a casual, relational register. Such language was especially visible around high-stakes requests and after failures. The researchers interpret these exchanges as treating the system as a social interlocutor, drawing on the idea that warmth and social presence can mediate trust in generative AI.
The important point is not that politeness is irrational or that every friendly exchange produces misplaced trust. The study did not experimentally estimate how much warmth changes scrutiny, and its observations do not establish a universal effect. The more limited mechanism is enough to matter: when an answer feels like it comes from a cooperative partner, the user may evaluate the relationship as well as the content. A confident-sounding interlocutor can invite acceptance before the learner has asked what evidence supports the claim.
The paper calls this risk the Affective Trust Trap. The same warmth that reduces the anxiety of asking for help may also reduce the discomfort that normally signals a need to check. In an educational setting, that discomfort can be productive. It can prompt a learner to retrieve what they know, compare an explanation with a worked example, or notice that an answer does not fit the problem. If the interaction removes every sense of friction, the learner may receive an answer without receiving a reason to examine it.
This creates a particularly subtle form of overconfidence. The learner may not believe that the system is infallible. They may simply find the conversation easy enough that checking feels unnecessary. They may be willing to correct the tool when an error is obvious while accepting an unfamiliar claim because nothing in the exchange feels alarming. Confidence in the interaction can then coexist with uncertainty about the subject, the system, and the result.
A literate user needs to keep two kinds of audit separate. The first is an output audit: Is this particular answer accurate, relevant, complete, fair, and suitable for the task? The second is a system audit: What kind of AI is this, what is it designed to do, what information does it rely on, what limits affect this use, and what risks follow from the way the interaction is handled? An output audit is local. A system audit helps determine how much confidence the local check should require and what should not be shared in the first place.
The distinction appeared clearly in the student study’s account of boundary awareness. Students scrutinised what ChatGPT said, but the corpus contained no questions from them about the system’s identity, version, browsing ability, memory, or data retention. The system’s knowledge cutoff and operational limits entered conversations mainly when ChatGPT disclosed them. Two attempts to use a circulated jailbreak prompt treated the system’s boundaries as obstacles to get around rather than as properties to understand.
This is a revealing observation, but it must be kept within its denominator. It describes the messages in a corpus of 36 undergraduates at one research university; it does not prove that those students had no system knowledge outside their chats, and it does not describe all learners. Its value is diagnostic. It shows how a person can be alert to the content of an answer while remaining incurious about the conditions under which the answer was produced.
That gap matters in practical use. Suppose a learner catches a factual error in a draft. They have performed a valuable output check. But if they do not know what information the system is receiving, how its capabilities differ across tasks, or which limitations are relevant, they may still use it in a way that exposes sensitive material or assigns it a role it cannot safely perform. Catching one wrong sentence does not demonstrate a general understanding of the system.
The reverse is also possible. A learner may understand that AI output is probabilistic and may know that a system has limits, yet still fail to check a particular answer. Knowledge creates a better starting model; it does not guarantee good action. This is why the framework connects technical understanding to critical thinking, problem solving, communication, and self and social awareness. A person needs both a concept of the system and a habit of applying that concept to the task at hand.
The most useful system questions are therefore concrete rather than mystical:
- What kind of AI system is being used, and what task is it designed to support?
- Which parts of this answer are claims that require independent checking?
- Could the output vary if the prompt, data, or system changed?
- What limitations, privacy concerns, bias risks, or human consequences matter in this context?
- What evidence would make it sensible to accept, revise, reject, or abandon the result?
A learner who can answer these questions is not necessarily always right. They are showing the conditions for calibrated judgement. A learner who cannot answer them may still be highly confident, especially if the system has made the task feel simple.
The confidence gap is therefore not repaired by telling users to be permanently suspicious. Nor do the sources justify claiming that warmth, fluency, or confidence always mislead. The stronger conclusion is narrower: these features can make subjective assurance a poor stand-alone proxy for competence, while the available evidence does not quantify a universal relationship between confidence and literacy. The appropriate response is to observe the learner at the points where judgement is required.
That shifts the question from “Do you feel confident using AI?” to “Can you explain what this system is doing, decide whether it belongs in this task, and justify what you do with its answer?” A full account of AI literacy must turn those questions into visible capabilities. It must show what knowledge, skills, and attitudes look like when a learner has to evaluate, direct, or challenge a system rather than merely enjoy a fluent interaction.