The Test for AI Tutoring: Can It Turn Access into Learning?

This series has examined AI tutoring as a school learning environment rather than as a model displayed in isolation. The final question is whether the evidence supports the title’s central claim: is engagement, rather than access, the bottleneck that determines whether an AI tutor contributes to learning? The strongest answer is yes—with an important qualification. Access remains the entry condition, but the available experiments repeatedly show that educational value depends on what schools and students do after access has been supplied, and on whether the resulting activity survives beyond the platform.

The chain, not the feature

The most useful synthesis is a chain of transitions. A school provides an account, device, or scheduled block. A student must then start, return, and spend time on the intended work. That work must be productive rather than merely logged or completed. When confusion occurs, the student must engage with support in a way that improves the next act of reasoning. Finally, the student must be able to perform later without relying on the same prompts. “AI tutoring” can be present at the first transition and absent at the fourth.

The literacy trials make the first break visible. Students had scheduled time and access to an AI literacy platform, but only 60.7% in one district and 53.3% in the other ever used it in the control condition. Average weekly use was 2.18 and 5.23 minutes, and students used the system in only about four to five weeks even though the interventions lasted 14 to 31 weeks. The problem was not that the platform had no instructional content. It was that the intended learning process often never got enough opportunity to begin, or did not recur.

Human support moved part of that chain. Tutors focused on motivation, participation, troubleshooting, group norms, and reflection rather than delivering the reading instruction. Their presence raised usage by about 1 minute per week in one district and 4.4 minutes in the other, and raised stories completed by 0.20 and 0.92 per week. Relative to very low control means, those were 71% and 80% increases. In absolute terms, however, the accumulated gains were about 22 minutes and 98 minutes over the interventions. End-of-year reading achievement did not improve.

That combination is not a failure of the evidence. It is the evidence. Human attention can change whether students participate while still failing to create a sufficient academic dose. A large relative change from a small baseline is an implementation result, not automatically a learning result. It also suggests that “engagement” must be defined in relation to the outcome a school wants. If the immediate goal is to get students to return, minutes and active weeks matter. If the goal is reading achievement, a school needs enough content completed and a test of later performance.

The Khanmigo study shows what happens when a school solves more of the access-to-practice transition. Across 18 Tennessee middle schools, the program placed individualized Khan Academy practice and a guard-railed AI tutor inside existing daily remedial mathematics blocks. Assignment generated about 30.8 additional minutes of weekly platform practice and a pooled achievement effect of about 0.04 standard deviations per term on the full study-population scale. The annual estimates were larger in the second year as implementation matured. Here, a protected routine produced substantial practice and measurable mathematics gains.

Yet practice did not become dialogue at the same rate. Ninety-six percent of activated students in the Year-2 message-level sample contacted Khanmigo at least once, but the median student messaged in only 17% of exercise sessions containing a mistake. Only 14.5% of classified messages contained a mathematical question or step of reasoning. The program solved one behavioral bottleneck—students doing individualized work inside a remedial block—more effectively than another: students choosing to articulate confusion to a conversational tutor.

NUMI examines the next transition more directly. It placed guard-railed AI support beside practice and used mastery progression to make mistakes consequential.