When Access Becomes Practice—but Not Dialogue
This chapter follows the Khanmigo school program from a scheduled remedial block to the student’s actual work and eventual test result. The important distinction is between practice dosage—how much structured work students complete—and tutoring dosage—how often they engage the AI in a substantive exchange. Keeping those channels separate shows both what the two-year experiment demonstrates and why it cannot be read as a pure chatbot trial.
What the school-scale program actually changed
The study took place in Hamilton County Schools in Tennessee, where students below grade level in mathematics attended Response to Intervention, or RTI, blocks during the school day. These were existing 25- to 40-minute sessions, usually in small groups, not an after-hours experiment added on top of the school’s schedule. Across 18 middle schools, one or two grades in each school were assigned to use Khan Academy with Khanmigo during those blocks. The randomization created 53 grade-within-school clusters and followed 6,902 student-term observations over the 2024–25 and 2025–26 school years.
The treatment was called KWiK in the study, but its meaning is more important than its label. Students in treated grades received individualized Khan Academy practice, with pathways placed from their mathematics screening scores, and access to Khanmigo, an AI tutor configured to coach rather than give final answers. The tutor could ask guiding questions and provide hints while students worked. Control grades continued ordinary RTI mathematics. That counterfactual varied from school to school: some teachers used no computer-assisted learning, some used it exclusively, and many combined software with teacher-led work, worksheets, or other adaptive platforms.
This detail determines the causal claim. Assignment to KWiK compared a coordinated remedial program with the mixture of remediation schools were already delivering. It did not compare “AI” with “no AI” while holding everything else constant. The treatment changed the content pathway, the consistency of the practice environment, the amount of platform work, and access to Khanmigo. A positive result is therefore evidence about the bundle. It may include value from the AI tutor, but the experiment does not isolate that component from the surrounding structure.
The program nevertheless changed the access margin in a way that matters. Students were not asked to discover an optional chatbot at home. The software was placed inside a mandatory school routine, supervised by teachers or intervention staff, and tied to a block that students already attended. Attendance in the RTI sessions was essentially the same across treatment and control—about 94%—so assignment changed what students did within the block rather than whether they showed up. In practice, however, a scheduled block was still not the same as uninterrupted platform time. Cancellations, testing, competing services, and ordinary school interruptions reduced the delivered dose.
The result was a substantial increase in structured practice. Treated students averaged about 30.2 minutes of weekly Khan Academy learning time, compared with almost none in the control grades, for an intent-to-treat increase of about 30.8 minutes. They worked roughly 4.3 skills per week and brought about 2.9 skills to proficient mastery. Among students still actively scheduled in RTI, practice was higher—about 38 minutes per week overall and about 45 minutes in the second year as implementation matured. These figures show a school system solving a real problem: it made individualized practice occur with much greater regularity than a voluntary resource normally does.
The practice increase came with a measurable achievement result. The pooled assignment effect was about 1.26 national percentile ranks per term, equivalent to roughly 0.04 standard deviations when scores were standardized against the full population of students in the study schools. The annual stacked estimate was about 0.062 standard deviations, and the Year-2 estimate was about 0.084. Scaling the Year-2 intent-to-treat effect by the share of assigned students still participating implied about 0.14 standard deviations for active participants under mature implementation. These are different expressions of the same experimental contrast, not separate gains to add together.
The pattern also tracked delivery. The first-year annual estimate was small and statistically indistinguishable from zero, while the second-year estimate was larger as practice increased. The difference between years was not itself estimated precisely enough to establish a definitive year effect, but the alignment is consistent with a simple interpretation: the program generated learning largely by producing more structured, individualized practice. The paper compares the return per hour with independent estimates of Khan Academy practice without Khanmigo and finds it within the same general range. That comparison does not prove that the AI added nothing. It does mean that the achievement data do not require a strong conversational mechanism to explain the program-level result.
The cost comparison makes the same point from another angle. Because KWiK used remedial blocks that the district was already staffing, the reported marginal cost was approximately $15 per student per year for the license. That is a meaningful value proposition for a district choosing how to use an existing block. It is not a claim that a $15 software license substitutes for a trained human tutor, nor a complete estimate of the personnel, training, implementation, and opportunity costs required to reproduce the study’s conditions.
The sequence makes the bundle visible. A school assigns time; staff make the routine usable; the platform creates individualized practice; the student may or may not invoke the tutor; and achievement is measured after the combined experience. The optional branch is important. The program can succeed at moving students from attendance to practice even when relatively few students take the next step into dialogue.
Why practice gains did not become tutoring dialogue
The strongest evidence on the conversational channel comes from the study’s Year-2 message-level records. Researchers matched Khanmigo messages to exercise logs for 563 activated treated students across 16 schools, covering 25,286 platform-active student-days and 100,017 exercise sessions. This was the year with the strongest implementation and the largest achievement estimate. The sample is also conditional: it includes students who were activated, had a successful data match, and recorded exercise activity. The authors caution that the observed engagement may overstate what would happen after the program became routine for a broader population.
At the extensive margin, contact was widespread. Ninety-six percent of students in this matched sample sent Khanmigo at least one message. At the intensive margin, the picture changed sharply. The median student messaged on 33% of the days they practiced, in 14% of exercise sessions, and in 17% of sessions that contained a mistake. The typical exercise session therefore passed without a message, including the very sessions in which a student had encountered an error that a tutor might help diagnose. Average chats were about 1.7 per week in the pooled student-term measure, rising to about 2.7 among actively scheduled students in Year 2. A mean that includes many zero-chat weeks can conceal that the typical student rarely opened a conversation.
The form of the messages matters as well. Among classified student messages, 39.4% were bare answers to Khanmigo’s own questions, 24.2% were clicks on suggested help prompts, 12.6% were low-effort messages such as “idk” or requests for the answer, 14.5% contained a mathematical question or a step of reasoning, and 9.3% were off-task. These categories should not be turned into a moral judgment about students. A short answer to a tutor’s question can be a legitimate part of a Socratic exchange, especially when the tutor is designed to solicit exactly that reply. The narrower and better-supported conclusion is that student-initiated mathematical articulation—asking a question in their own words or explaining a step—was uncommon.
This is the difference between a platform that contains a tutor and a tutoring interaction that actually occurs. Khanmigo was configured to withhold answers and to guide students through questions. That design may be pedagogically sound, but it makes help-seeking a choice with a cost. The student must recognize confusion, decide that help is worthwhile, open or select the relevant support, and respond to the tutor rather than simply submit another answer. Mandatory practice time removes the barrier of finding a session. It does not automatically remove reluctance, uncertainty about what to ask, or the temptation to move on.
The result is not a contradiction. The same environment can generate 30 additional minutes of practice while producing little dialogue. Practice is built into the assigned routine; conversation is activated at the student’s discretion. A student can complete adaptive exercises, receive feedback, and make progress through the pathway without asking Khanmigo to interpret a misconception. This is why a school dashboard should report both minutes of platform work and the share of mistakes or exercise sessions that invoke the tutor. One number measures the opportunity to practice; the other measures whether the conversational capability was actually used.
The achievement evidence reinforces that separation. Within the treated arm, skills brought to proficiency were positively associated with later achievement: each additional skill mastered per week was associated with about 0.015 population standard deviations. Weekly chats had a small positive association on their own, but it fell to approximately zero after controlling for skill mastery. These are descriptive within-treated relationships, not randomized channel effects. Students who master more skills or initiate more chats may differ in preparation, persistence, or motivation. Still, the pattern is consistent with the fact that the largest and most reliably shifted activity was structured practice, while conversation was relatively infrequent.
The researchers also rescale the experimental result by different engagement measures, including weekly minutes, skills mastered, and chats. Those calculations can look like separate returns—for example, an effect per weekly chat—but they are alternative ways of expressing one reduced-form assignment effect through different first stages. With a single instrument, they cannot tell us whether a conversation caused the gain, whether conversation is a marker of productive practice, or whether both reflect another feature of the program. The evidence supports a cautious interpretation: the KWiK program produced learning, and its measured achievement signal is most readily explained by the structured practice it reliably created. The marginal contribution of a rarely used conversational tutor remains unresolved rather than disproved.
That boundary is especially important when comparing this trial with a product demonstration. A model may answer a difficult mathematical question correctly in a test environment. A school may then expose thousands of students to the same feature. The first fact establishes capability; the second establishes availability; neither establishes that students will invoke the capability at the moment it matters. The Khanmigo data show that even an embedded, guard-railed tutor, placed beside practice in a supervised block, was seldom asked to conduct substantive mathematical dialogue by the typical student.
The finding should not be reduced to “students do not want help.” The trial does not measure every reason for non-use, and a message classification cannot reveal a student’s internal state. It does show that the design left a meaningful part of tutoring dependent on student initiative. That is an important implementation choice. If a system is intended to support reasoning after errors, then the relevant test is not whether students can open a chat at least once. It is whether the environment causes help to appear, or makes it sufficiently easy and worthwhile, when students are confused—and whether that support improves later independent performance without consuming too much of the practice window.
The two-year experiment therefore resolves one problem and exposes another. It solved much of the access-to-practice transition by putting individualized work inside a real remedial routine. It did not solve the practice-to-dialogue transition. The positive math result is credible as a program-level effect, but its mechanism is chiefly structured practice rather than a separately measured conversational benefit. To understand when the AI itself can add value, the next question must move closer to the moment of need: what happens when a student makes a mistake and the tutor is built directly into the recovery process?