The User Makes the Difference: Augmentation, Automation, and Durable Learning
The relationship between generative AI and student learning becomes clearer when access and use are separated. This chapter examines a randomized college experiment that allowed some students to use AI while learning an unfamiliar topic, then observed what remained when the tool was removed. The evidence supports a practical judgment: access can help, but the mode of use determines whether the tool adds power to the learner’s effort or substitutes for it.
That distinction is easy to miss because “AI use” sounds like one treatment. It is not. A student might ask a chatbot to explain a difficult concept, compare two interpretations, or point out a gap in an argument. Another student might ask it to draft the argument and submit a lightly edited version. Both have used the same kind of system. They have not undertaken the same learning activity.
What the experiment actually tested
The study by Contractor and Reyes took place at Middlebury College. In the first of two in-person sessions, 211 undergraduates were randomly assigned to an AI-allowed or AI-forbidden condition. Each student learned about one unfamiliar topic—blockchain technology, carbon capture systems, or CRISPR gene editing—and wrote an analytical essay. The learning phase lasted up to 35 minutes, and students could divide that time among reading, searching, drafting, editing, and other activities.
Random assignment is important, but it identifies a narrower question than many headlines imply. It estimates the effect of being allowed to use AI during the learning phase, whether or not a student actually used it. It does not force a student to ask for explanations rather than drafts, and it does not tell us what would happen if every course used AI continuously for years. Actual use remains a choice shaped by prior habits, confidence, incentives, task difficulty, and the surrounding teaching design.
The study made that choice visible. Students in the allowed condition had access to a logged-in ChatGPT account as well as ordinary information resources. Students in the forbidden condition had access to traditional resources but were told not to use generative AI. Proctors observed the sessions; the researchers also used account activity, self-reports, and platform traces to monitor compliance and usage. Assignment to the allowed condition increased observed AI use by 67.3 percentage points from a near-zero control baseline. By self-report, any generative-AI use rose by 59.9 percentage points. Among users, 88% said the tool was somewhat or very helpful.
The first outcome was an immediate test taken without external resources. Students in the control group improved substantially from their baseline, as one would expect after studying an unfamiliar subject. AI access added another 6.7 percentage points to the share of questions answered correctly. Standardised against the control group’s variation, the effect was 0.27 standard deviations. That is a meaningful improvement, but it is an improvement in a defined learning session, not a general measure of AI’s value in education.
The second session is what turns the result into evidence about learning rather than only assistance. It took place about one week later. All students completed a ten-question test and wrote another essay without AI or other external resources. The AI-allowed group remained 5.1 percentage points ahead on the test—about 76% of the immediate gain. The effect had partly decayed, but it had not vanished. In this setting, AI access was associated with knowledge that persisted after removal of the tool.
The writing results also show why a single output score would be inadequate. During the first session, when AI was available, treated students’ essays were only slightly and imprecisely better overall. Some of the visible text could be attributed directly to the tool. In the second session, when students wrote without AI, style and clarity and relevance to the prompt improved significantly. The overall quality estimate was positive but imprecise. The experiment therefore detected both immediate assistance and later unaided gains, but different measures captured them.
This is not a loophole in the paradox. It is the point. AI can improve learning in one context without making the negative mathematics result described earlier disappear. The studies differ in age group, subject, duration, task, tool configuration, and what happens after practice. The college experiment gave students a new topic, freedom to choose how to use an off-the-shelf system, and an unaided follow-up. Its result shows a possibility, not a universal guarantee.
The sequence matters because the final assessment is not another AI-assisted task. It asks what the student can do after the temporary support has gone. It also reveals the limit of the design: a one-week follow-up cannot establish long-term skill development, and the lab’s fixed time window may differ from the choices students make in ordinary coursework.
Two ways to use the same access
The researchers classified the conversations of AI users into two broad modes. Augmentation meant that AI worked with the student: the student remained engaged in the task while using the system to explain or support the work. Automation meant that AI did the work for the student, especially by generating a response that could stand in for the student’s own production. The categories are not moral identities. They describe where the study observed the work being performed.
Of the treated AI users, 49% were classified as pure augmentation users and 32% as pure automation users. Another 8% used the system in a mixed way, and 11% were classified as other because their interactions were off-topic. The classification was based partly on an LLM reading the conversation logs, so it should not be treated as a perfectly objective or randomly assigned trait. Still, three kinds of evidence support the distinction.
First, the students asked for different things. Augmentation users turned to AI more often for explanations. Automation users relied more on drafting and editing. That difference is more informative than the number of prompts alone. A long conversation can involve little learning if the system is repeatedly asked to produce text. A short exchange can be useful if it clarifies a misconception that the student then tests independently.
Second, the groups allocated their time differently. Compared with augmentation users, automation users spent about 17% less time on research activities such as reading and searching, and about 8% less time overall. Across the AI-allowed group, total learning time did not significantly change relative to the control group, but its composition did: writing activities took about 5.3 percentage points less of the learning phase, while reading and searching took about 4.4 percentage points more. The relevant question is therefore not simply whether AI saves time. It is what replaces the saved time and what the student does with the time that remains.
Third, the submitted text reflected the different roles. Pangram flagged 53.6% of automation users’ text as AI-generated, compared with 20.9% for augmentation users. This is not a measure of learning by itself, and detection tools can have limitations. In the study, however, it is consistent with the behavioural evidence: automation users were more likely to have the system produce the visible answer, while augmentation users were more likely to use it as part of their own inquiry.
The outcome pattern is the clearest evidence of the difference. In the first session, automation users had a large essay-quality advantage—0.54 standard deviations—while augmentation users’ essay-quality effect was small, at 0.05 standard deviations. This is exactly what we would expect if automation improves the product by supplying more of the writing. The same group did not show a corresponding advantage in learning tests. Augmentation users had the larger test-score effect, 0.44 standard deviations compared with 0.33 for automation users, a pattern consistent with AI helping them understand rather than simply write.
In the second session, the output advantage of automation disappeared. Automation users’ overall essay-quality effect was close to zero, at 0.02 standard deviations. Their test-score effect also attenuated and was imprecisely estimated. Augmentation users retained a positive, though imprecisely estimated, effect on unaided essay quality of 0.22 standard deviations and showed a larger test-score effect of 0.29 standard deviations. The statistical uncertainty around subgroup estimates matters; these numbers do not prove that every explanation-seeking student learned more than every drafting student. They do show a coherent contrast between borrowed performance and more persistent capability.
This is the operational meaning of augmentation. It is not “using AI responsibly” as a vague virtue. It is a pattern in which the learner continues to read, decide, explain, and revise while the system increases the reach or speed of those activities. Automation is not defined by whether AI appears anywhere in the process. It is defined by whether the system takes over the activity whose performance is meant to become the learner’s skill.
The benefits are real, and so are the qualifications
The experiment does not support a simple story in which AI is either a crutch or a tutor. Students with access reported enjoying the learning task more: enjoyment increased by 0.66 points on a ten-point scale, a 13% increase over the control mean. The tool may have made it easier to explore an unfamiliar subject, reduced the frustration of getting started, or shifted time toward reading and searching. Those are legitimate educational benefits when they lead to more understanding rather than less effort.
But enjoyment is not the same as mastery. A comfortable activity can be productive, unproductive, or mixed. The study also found more rule violations on the test among students with AI access. On the combined measure of proctor-observed and self-reported violations, the increase was 12.6 percentage points from a control mean of 4.9%. The authors’ rough calculation suggested that cheating could account for at most a modest share of the test-score gain, under a strong assumption about the score difference between cheaters and non-cheaters. That estimate should not be read as proving that integrity concerns are negligible. It shows that the positive learning result cannot be reduced entirely to rule breaking, while still documenting a real cost of open access in the experimental setting.
The distribution of benefits also complicates the average result. Gains appeared mainly in the middle of the score distribution. They were smallest among students in the lowest GPA or SAT quartile and larger among students with stronger prior academic records. The study therefore suggests that unrestricted access may widen some pre-existing differences even while raising the average score. A tool that helps a well-prepared student ask sharper questions may not help a less-prepared student who cannot yet recognise a misleading answer.
The setting limits what can be inferred about time. Because all students had a fixed learning phase, access changed how the same block of time was used. In ordinary academic life, students may use AI to finish sooner and then spend the saved time elsewhere. Whether total learning rises will depend on what that time becomes: more reading, practice, discussion, rest, or no learning at all. The experiment provides evidence about learning per unit of time, not about the total amount of learning produced by a changed schedule.
These qualifications suggest a conditional use judgment rather than a universal winner. If the learning goal is to build conceptual understanding, explanation-oriented use is the safer starting point: ask for a clarification, compare it with the assigned material, state the idea in one’s own words, and test it on a new case. If the goal is to produce a polished document after the reasoning has already been done, limited automation of surface editing may be reasonable. If the task is specifically to practise argument construction, problem solving, or synthesis, asking AI to perform that work at the outset risks turning the assessment into a product-production exercise.
A practical diagnostic is to inspect the learner’s next unaided move. After using AI, can the student explain the idea without looking at the response? Can they identify why the answer is correct, what evidence supports it, and where uncertainty remains? Can they solve a nearby problem or defend a different conclusion? If the answer is no, the tool may have reduced the discomfort of learning without building the capability.
The same test applies to teachers and institutions. A rule that says “AI is allowed” says nothing about whether the design encourages augmentation or automation. A better design makes the desired human work explicit. It may require an initial attempt, ask for a brief explanation of how an answer was checked, or include a later assessment without the tool. It may allow AI for exploration but not for the final reasoning, or allow drafting only after the student has supplied the claim and evidence. The exact boundary should follow the capability being developed.
The experiment’s most important lesson is thus not that students are naturally good or bad at using AI. It is that use mode is connected to incentives and task structure. When a high-quality submission is the main reward, automation is attractive. When the learner must understand, defend, and transfer the material, augmentation becomes more valuable. The next design question is how feedback itself can make that distinction easier to enact—whether an AI system should hand over a list of corrections or require the learner to judge the work before receiving its critique.