What Happens After the Mistake
The most revealing moment for an AI tutor is not when a student answers correctly. It is the interval after a wrong answer, when the student can guess again, skip the problem, read a solution passively, ask for help, or work through the misconception. This chapter uses the NUMI experiment to examine that moment directly. It shows why embedded guidance can improve the quality of recovery while slowing the pace of practice, and why that mechanism is stronger evidence than a platform completion metric but still not the same as proof of durable learning.
A tutor built around the error
NUMI was designed as both a middle-school mathematics platform and an experimental environment. In a single class period during March and early April 2026, students worked through an assignment intended to last about 50 minutes. The following week, they took a delayed assessment lasting roughly 15 to 20 minutes. The study included students in grades 6 through 8 across 20 Hamilton County schools and nearly 100 teachers; 6,997 students were recorded as logging into the Week-1 activity, and 6,327 students who later took the assessment entered the delayed-test analysis.
The design held the basic learning environment constant while varying three dimensions at student login. Students were assigned to one of two grade-specific topic bundles, to AI support or CAL-only support, and to mastery or non-mastery progression. Both AI and CAL-only students received instructional videos, practice questions, immediate feedback, and worked solutions. The AI condition added a tutor beside the exercise: students could request help getting started, receive a guided walkthrough after a wrong answer, or select a step of the worked solution for a concept-focused explanation.
That comparison is important. The study did not ask whether any computer support is better than no computer support. It asked whether a guard-railed language-model tutor added value over a similar computer-assisted learning platform that already provided explanations. The tutor was designed not to reveal the final answer. Before an attempt, it introduced a first-step approach and asked a short reasoning question. After a mistake, it addressed a likely misconception and led the student through the solution one step at a time, using checks that could be yes-or-no, multiple-choice, or short answer. When the solution was visible, the student could request an explanation of a selected step rather than merely rereading it.
The surrounding progression rule created the condition in which these supports might matter. Students in the mastery group had to watch at least one minute of the relevant video and answer three questions in a row correctly before moving to the next part of the assignment. A mistake reset the streak. Students in the non-mastery group could move more freely after attempting questions. The rule did two things at once. It increased the cost of moving on after an error, which could create a reason to seek help. But it also consumed time, so a student who spent longer repairing Exercise 1 might reach fewer questions in Exercise 2 or never reach the exit ticket.
The causal logic is therefore conditional.