The Prompt Before the Number
This chapter follows the Interactive Learning Analytics Dashboard (ILAD) tested in a university programming course. Its focus is not the novelty of adding a chatbot to a chart, but the sequence created when learners judge their own mastery before seeing the dashboard’s estimate. That sequence lets us examine what learners did with telling and eliciting agents, what changed in their judgements over five weeks, and how far those observations can support a claim about learning.
A dashboard arranged as a test of interaction order
The ILAD was embedded in an introductory algorithms and data-structures course at a European university. Thirty students volunteered, were paid for participating, and used the system for five weeks in the context of their ordinary course work. The researchers chose an authentic setting rather than a laboratory exercise: students were already submitting assignments and working through a demanding programme, and the dashboard was placed as the landing page of a heavily used learning platform. That choice made the case more realistic, but it also meant that the study observed a small, self-selected group in one course rather than a representative student population.
The researchers randomly assigned participants to three versions of the dashboard. One had no pedagogical agent. A second had a telling agent that supplied information about the learner’s data. A third had an eliciting agent that primarily asked questions about the data. The dashboard and its Judgement of Learning feature were available across the conditions; the agent was the principal experimental difference. This matters because “the AI dashboard” is not a single intervention here. The case compares different ways of placing cognitive work around the same kind of learner evidence.
The underlying display looked familiar. A line graph showed exercise scores and submission lateness. Competencies organised topics and skills, with indicators for completed or viewed exercises and lecture material. The system also estimated three related quantities for each competency: progress, confidence, and mastery. Progress represented advancement through the associated material and tasks. Confidence drew on heuristics such as improvement over time, task difficulty, and solution speed. Mastery combined progress with that confidence factor.
Those definitions are important because the rating was a model of learning, not a direct reading of understanding. A learner could have completed most of the material while still lacking confidence on difficult tasks, or perform well on a recent exercise without having covered everything in a competency. The system’s estimate made those dimensions visible, but it did not turn them into an independent gold standard. The case asks whether learners became better aligned with the estimate, not whether the estimate itself perfectly measured mastery.
The distinctive choice was the JoL gate. For a competency that met a prompting condition—when the learner’s progress reached more than 80% of the average progress or the recommended completion date was near—the interface asked the student to rate their mastery on a five-step scale. The system initially hid its own mastery and progress rating. After the learner submitted a JoL, the system revealed its estimates and displayed recommendations or related material. The general score and lateness graph remained available while the learner judged, as did the option to chat with the agent.
This ordering changes the role of a number. If the system reveals “mastery: four stars” first, a learner may use that value as the starting point for their judgement. If the learner must first form an estimate, the system value becomes feedback on an existing belief. The comparison can expose under-confidence, over-confidence, or an assumption about what counts as knowing. It can also expose a defect in the system’s model. The point is not to make the learner lose a guessing game; it is to make the judgement visible enough to examine.
The gate did not remove all context before the learner answered. The score and lateness graph remained on the page, and the learner could still chat with the agent. What was withheld was the system’s competency-level rating—the very value most likely to anchor the self-assessment. This is a more precise intervention than simply hiding data. It preserves evidence that a learner might reasonably use while delaying the system’s conclusion until the learner has committed to an interpretation.
The trigger for the prompt was also tied to the course’s rhythm. A learner was asked to judge a competency when their progress had passed more than 80% of the average progress or when the recommended completion date was close. The first condition makes the prompt responsive to accumulated activity; the second makes it timely in relation to the course schedule. Such a rule can bring a judgement close to a real study decision, but it is not neutral. A learner who is behind, working at a different pace, or using material outside the recorded course may encounter the prompt at an awkward moment. The study did not compare alternative thresholds, so the value lies in the interaction principle rather than in this particular trigger as a universal setting.
The three displayed quantities also create different possible comparisons. Progress tells the learner how much associated material or task activity has been completed. Confidence modifies the picture using signals about improvement, difficulty, and speed. Mastery combines the two. A learner might therefore see why a high completion rate does not necessarily produce a high mastery estimate, or why a difficult but successful task can affect confidence differently from merely viewing a lecture. Explaining these relationships matters more than showing the labels. If the system presents a composite rating without making its ingredients intelligible, the learner has little basis for deciding whether to trust or question it.
The study’s setting helped the researchers observe these choices in ordinary use. The dashboard was not an external research tab that students had to remember to open; it was the landing page of a mature learning platform used across universities. Participants received minimal instructions—use it at least once a week for any three weeks of the five-week period—so their conversations and judgements could develop in a relatively natural course context. That ecological value comes with a trade-off: usage was not controlled tightly enough to isolate a single exposure or guarantee that every participant engaged with the same evidence in the same way.
The diagram also shows what the gate does not do. It does not hide every piece of evidence, force a single correct response, or guarantee that a learner will study differently. It creates a pause in which a self-assessment can meet a system assessment. Whether that pause supports reflection depends on what the learner does with the discrepancy and whether later activity shows any consequence.
What learners did with the comparison
The researchers examined the case in two connected ways. They looked at the content of learner-agent messages to see how the two agent strategies shaped interaction, and they tracked JoL ratings over time to see whether learners’ judgements became more consistent or more aligned with the system’s estimates. These are process and intermediate measures. They are useful for examining the proposed mechanism, but they are not substitutes for exam performance, observed study scheduling, or long-term retention.
The message analysis began with a random sample of more than 300 interactions. The researchers coded messages into categories including concrete exercise questions, requests for external resources, repeated clarification, course organisation, planning, and reflection. About two-thirds could be assigned clearly; a second researcher coded a random 15% sample of categorised messages, producing a Cohen’s kappa of 0.8. This is a reasonable check that the categories were not entirely idiosyncratic, although it does not tell us whether a message classified as “reflection” represented deep or effective reflection.
The pattern differed by agent strategy. Students with the eliciting agent wrote more messages containing reflection about their study activity than students with the telling agent. The paper reports a two-proportion z-test of 2.47 with p = 0.013 for that difference. Telling-agent users more often sought clarification or asked organisational questions. A learner asking, “What is the difference between competencies and mastery?” needs information to interpret the dashboard. A learner asking about the relationship between time invested and scores is doing a different kind of work: they are testing a hypothesis about their own learning process.
That contrast should not be turned into a ranking. Clarification can be a necessary step before reflection, and a learner with low visualisation literacy may need a direct explanation before an eliciting question becomes fair. The telling strategy may also be useful when the system must define a measure or correct a misunderstanding quickly. What the case suggests is narrower: when two learners have access to similar data, the wording of the agent can orient them toward the system as an information source or toward their own activity as something to investigate.
The JoL data show a second pattern. Participants submitted 482 judgements during the intervention. The variance among sets of three successive JoLs decreased over time, which means that learners’ ratings became more consistent. The distribution also changed across competencies: early judgements often under-rated mastery, middle judgements more often over-rated it, and later ratings were more frequently aligned with the system’s assessment. Consistency and alignment are related but not identical. A learner can become consistently overconfident, so neither measure alone proves accurate understanding.
For accuracy, the researchers correlated learner JoLs with the system’s progress, confidence, and composite mastery ratings. In the eliciting condition, based on 104 JoLs, the reported correlations were r = 0.287 for confidence, r = 0.326 for progress, and r = 0.408 for mastery. The mastery correlation increased across three time intervals: from r = 0.332 in the first interval, with 32 JoLs and p = 0.064, to r = 0.378 in the second, with 19 JoLs and p = 0.111, and r = 0.482 in the third, with 53 JoLs and p < 0.001.
The other conditions make the interpretation more cautious. The telling condition was initially negatively correlated with system mastery, then showed a non-significant relationship, and finally reached a significant positive correlation of r = 0.571 in the last interval. The no-agent condition moved in the opposite direction, from an initial significant correlation of r = 0.428 to r = 0.110 in the final interval. These trajectories do not support a simple rule that eliciting always works and telling never does. They are patterns within a small case, with unequal numbers of observations in the time bins.
The researchers also asked why students used the JoL feature. Eighty-three percent said they were curious to see how the system rated them. Yet 70% reported that they did not actively pay attention to the learning-analytics graphs while making their judgement. That apparent tension is revealing. The prompt may have created a reason to engage with the dashboard even when learners were not carefully reading every visual element. But curiosity is an entry condition, not proof of learning. The study also relied on self-report when it found that 79% said they changed behaviour after judging their learning, compared with 47% who reported adapting behaviour because of agent interactions.
The strongest conclusion is therefore about a plausible mechanism and a set of hypotheses. Asking learners to commit to a judgement before revealing a system rating can make comparison possible. An eliciting agent can be associated with more reflection in the recorded dialogue, and its condition showed a positive trend in alignment with the system’s estimate. Those observations fit the theory that learner-generated interpretations create more cognitive engagement than passive receipt of information.
They do not establish that the ILAD improved subject-matter knowledge. The participants were paid volunteers, only 30 students took part, and the observation window lasted five weeks. The study did not directly observe changes in study schedules, measure exam performance, test retention or transfer, or determine whether calibration persisted after the incentive ended. The correlations were used to describe trends, not to establish statistically definitive differences between conditions. Even the phrase “more accurate” means closer correlation with an LMS estimate, not necessarily closer access to an independent truth.
That boundary is part of the case’s value. The ILAD demonstrates how a dashboard can be arranged so that a metric arrives as feedback on a learner’s judgement rather than as a command. It also shows why the quality of the loop depends on the model behind the number. To understand the second study’s feedback intervention, we need to ask how a learning system builds its account of a learner from scattered online activity—and how much confidence that account deserves.