What Students Do When AI Answers
The empirical subject here is naturalistic student–AI practice: what learners actually do in their conversations with a generative system, rather than what they say they can do. To understand what confidence can hide, this chapter examines a year of student–ChatGPT interaction logs, separates five recurring ways of relating to an answer, and shows why efficient use should not be mistaken for a general level of AI literacy.
The distinction matters because a confidence survey and an interaction record answer different questions. A survey can tell us how capable a person believes they are. A log can show whether they ask for a direct answer, add constraints, challenge a failure, test their own reasoning, check a claim, or leave when the result cannot be trusted. Neither source gives a complete picture of learning, but the second offers evidence about enacted judgement that the first cannot provide on its own.
A log captures practice, not a feeling
Ammari and colleagues analysed complete ChatGPT histories from 36 undergraduates at a research university in the northeastern United States across one academic year. The corpus contained 1,631 chats and 10,536 messages; the mean session duration was 13.2 minutes. Participants were adults and active ChatGPT users who voluntarily supplied their histories for an anonymised research protocol. The study therefore observes people who were already using the system, not a random sample of all students or a controlled group assigned to a learning intervention.
The researchers used a sequential mixed-methods design. They first manually coded a sample of 1,882 messages, developing 42 detailed subcategories and grouping them into five broader categories covering information seeking, content generation, language use, student–ChatGPT interaction, and response behaviour. They then used GPT-4o to scale the human-developed coding across the larger corpus and checked the scaled labels against human judgements. The reported Cohen’s kappa values for the five main categories ranged from 0.75 to 0.91, indicating substantial to near-perfect agreement for those validation samples.
These methods make the study useful for a specific purpose. The messages preserve moments of action and response: what a student asked, how they reacted when the answer failed, and whether they tried to verify or revise. The data can reveal patterns that a person may not remember or report accurately. It cannot reveal everything that matters. The researchers did not measure participants’ self-confidence, compare their chats with a validated AI-literacy test, or demonstrate durable learning gains. An unrecorded check against a textbook, a conversation with a teacher, or a private decision to distrust an answer will not necessarily appear in the corpus.
The sample also sets a firm boundary around interpretation. It comes from one university, includes no demographic or degree-program information beyond what could be inferred from the exports, and may have a modest STEM skew. It focuses on ChatGPT rather than the wider ecology in which students move between a chatbot, conventional search, course materials, peers, and other tools. The study can illuminate how competence may be enacted in one setting. It cannot establish how common each pattern is among learners generally, whether one pattern causes better learning, or whether confidence predicts any of them.
That distinction between method and conclusion is important. The researchers argue that AI literacy develops through everyday practice and that the genres show how competence is enacted in situated interaction. This is a theoretically informed interpretation of observed behaviour, not a direct measurement of learning.