The First Bottleneck: Getting Students to Use the Tool
This chapter examines the first operational bottleneck in AI tutoring: whether students use an available platform often enough for it to become instruction. The two elementary literacy trials at the center of the evidence are useful because they separate access from take-up and take-up from dosage. They also show how to interpret a positive engagement estimate without mistaking it for proof of better reading.
Ever using a platform is not the same as using it
The most common headline about educational technology is an access headline: every student received an account, a device, or scheduled time. That is a necessary condition, but it hides at least two different questions. Did a student ever open the system? And, once they did, did they return often enough to accumulate meaningful practice? The first is the extensive margin of use. The second is the intensive margin—frequency, minutes, weeks, and completed work among the students who have an opportunity to use it.
The distinction is visible in two randomized trials reported by Robinson, Gormley, Ribeiro, and Loeb. Students in both districts were expected to receive at least two 30-minute sessions per week with an AI literacy platform. The interventions lasted between 14 and 31 weeks. Yet among control-group students, only 60.7% in District A and 53.3% in District B ever used the platform. Average use was just 2.18 minutes per week in District A and 5.23 minutes in District B. Students who used the platform did more on the weeks when they showed up—13.2 and 25.8 minutes per week, respectively—but most weeks contained no use at all. On average, students used the system in only four to five weeks across the intervention.
Those numbers describe a large gap between scheduled capacity and realized learning opportunity. A 30-minute session on the calendar is not a 30-minute learning dose. Even “ever used” is a weak achievement if a student logs in once and disappears. The low average arose from two kinds of loss: many students never started, and many of those who started did not return consistently. Treating the two losses as one would make it harder to choose a remedy. A login problem, a timetable problem, reluctance to ask for help, and low persistence may all appear as “low engagement,” but they require different interventions.
That distinction also changes how the trial should be read causally. The control group was not a no-technology group; it was the counterfactual for students who had the same expected access but worked independently. Treatment estimates on ever-use were not statistically distinguishable from zero, while minutes and stories rose. The evidence therefore points to human support operating mostly after or around initial take-up—helping some students return, stay, and complete content—rather than reliably creating the first login. This is not proof of a single psychological mechanism: tutors simultaneously supplied accountability, troubleshooting, and social structure. It is, however, more informative than saying simply that tutors “boosted adoption.” Adoption has stages, and the intervention moved some later stages more clearly than the first.
The same logic applies to interaction with the tutor itself. In the two-year Khanmigo experiment, 96% of activated Year-2 students messaged the AI at least once. That sounds like successful adoption until the intensive margin is examined: the median student messaged Khanmigo on only 33% of the days they practiced, 14% of exercise sessions, and 17% of sessions in which they made a mistake. First contact was nearly universal; recurring, situated help-seeking was not. A school that reports only the first number would conclude that the tutor had been widely used. A school that wants to know whether tutoring was present at moments of confusion needs the latter measures.
This is why engagement should not be reduced to screen time either. Time can indicate opportunity, but not what happened during that time. A student may read, attempt, explain, revise, or reason through a mistake. They may also guess, click through prompts, or remain logged in while doing little. The literacy trials measured both usage, defined as minutes on the platform, and engagement, defined as stories completed. The Khanmigo study added message-level measures because platform practice and conversational tutoring were not the same behavior. The measurement question should follow the learning claim: if the claim is about individualized practice, measure practice; if it is about tutoring dialogue, measure help-seeking and the content of the exchange; if it is about learning, test later performance.
The state model is not a claim that every student moves through these stages in one fixed order. It is a way to keep the transitions visible. Human support can help a student move from first use to repeated use, but it does not guarantee that the resulting activity is sufficient or productive. Nor does a failed transition prove that the platform itself is incapable of teaching. It shows that the intended learning process has not yet been established.
What human support changes—and what it cannot
The literacy trials tested a particularly revealing form of human involvement. The tutors were not substitute reading teachers. In District A, program staff worked with students in five after-school sites; in District B, middle-school students tutored younger students during the school day. In both settings, students were randomly assigned either to work independently on the AI platform or to work with a small-group tutor. Tutors supported motivation, participation, technology troubleshooting, group norms, and reflection. The platform remained responsible for individualized literacy instruction and feedback.
The treatment sessions were deliberately not devoted entirely to platform minutes. A typical session included about 10 minutes for a check-in, 15 minutes for platform work, and 2–5 minutes for reflection. Control students were expected to spend the full 30 minutes on the platform. On a narrow accounting view, human support should have reduced usage by taking time away from the software. Instead, it increased both measured outcomes. In District A, assignment to a human tutor raised platform use by about 1.0 minute per week and stories completed by 0.20 per week. In District B, the increases were about 4.4 minutes and 0.92 stories per week. Relative to the control means, the engagement gains were 71% and 80%.
The mechanism is not mysterious, but it is easy to understate. A tutor can make the session a social and organizational event rather than an optional task competing with every other demand on a student’s attention. A check-in can establish that participation is expected. A group norm can make it less embarrassing to struggle in public. Troubleshooting can prevent a technical obstacle from becoming a reason to quit. Reflection can reconnect an otherwise isolated exercise to a sense of progress. None of these actions is direct reading instruction, yet each can change whether the instructional platform is actually used.
The results also demonstrate why relative and absolute effects must be reported together. A 71% increase sounds large because the control mean in District A was only 0.28 stories per week. The corresponding increase was 0.20 stories per week—roughly 4.4 additional stories over the intervention—not a transformation of the weekly learning dose. Similarly, a 4.4-minute increase in District B was statistically meaningful and educationally relevant as a participation change, but it left students well short of the provider’s stated 30-minute weekly benchmark for measurable reading gains. Across the full intervention, the added use amounted to approximately 22 minutes in District A and 98 minutes in District B. Both are less than two additional hours on average.
The achievement result follows from that dosage. The researchers found no evidence that assignment to a human tutor improved end-of-year reading achievement in either district. The estimated effects on standardized spring ELA scores were negative in magnitude but not statistically significant. The appropriate interpretation is not that human support harmed reading, nor that relationship-building is irrelevant. It is that the observed increase in platform engagement was too small or too inconsistently delivered to generate a detectable achievement gain in these implementations. The study demonstrates the difference between evidence of increased activity and evidence of improved learning.
The two districts also warn against treating “human support” as a uniform treatment. The locations differed in age groups, setting, tutor type, treatment capacity, and control-to-treatment ratio. District A assigned 84 students to treatment and 90 to control; District B assigned 36 to treatment and 145 to control because tutoring capacity was limited at the participating schools. Site-level effects varied as well: some sites showed little or no increase in platform minutes, while others showed substantial gains. That variation is not merely statistical clutter. It suggests that tutor preparation, attendance, group management, local scheduling, and the credibility of the routine can determine whether the same broad idea becomes a functioning intervention.
A positive average can therefore conceal an implementation question: which conditions made support effective in one site and weak in another? The source does not provide enough evidence to identify one decisive feature. It does establish that the intervention was not a simple switch labeled “human present.” The content of the tutor role, the organization of the session, the students’ prior participation, and the local capacity to deliver support were part of what students experienced.
There is an equity reason to examine the intensive margin closely. In the control groups, students who used the AI tutor were more likely to be higher achieving and less likely to receive special-education services. More than 80% of the overall sample was economically disadvantaged. The authors note that individual accommodation decisions may have influenced who used the platform, so this pattern is descriptive rather than a clean estimate of the effect of student characteristics. Even with that qualification, it is a warning against inferring equal opportunity from equal availability. If students who already navigate school successfully are also the students most likely to persist with an unfamiliar digital tool, an opt-in system can distribute its practice toward those least in need of additional access.
That possibility changes what a school should ask of its dashboard. “How many licenses were assigned?” is an administrative question. “How many students used the platform?” is an extensive-margin question. “How many used it every week, and for how long?” measures dosage. “Who is missing at each stage?” makes the equity problem visible. And “what support moved a student across the gap?” turns the dashboard into a tool for learning about implementation rather than a report of adoption.
The trade-off is real. Human support adds personnel, training, coordination, and scheduling requirements. A fully AI-led model is more scalable in labor terms, but the literacy evidence suggests that it may struggle to generate consistent use. A human-led model can invest more heavily in relationships and accountability, but it cannot be assumed to produce proportionate learning gains if the underlying platform dose remains low. The source describes these options as a spectrum of relational intensity, not as a ranking in which one end always wins. An AI-assisted human tutor might even use software to reduce logistical work and reserve more human attention for the relationship.
The first bottleneck, then, is not solved by declaring support “available.” The literacy trials show that human attention can improve participation and content engagement even while reducing the minutes nominally reserved for the platform. They also show the limit of that improvement: a large relative effect from a very low baseline can remain a small absolute dose, and a small dose can leave achievement unchanged. The next test is what happens after a school succeeds at creating regular platform practice. Does more practice also produce meaningful dialogue with the AI tutor, or does the conversational bottleneck remain?