From Showing Data to Asking for Meaning
This chapter examines the mechanism that can turn a learner-facing learning analytics dashboard from a readable display into a site of reasoning. The specific question is whether a learner is merely manipulating a chart or is being helped to interpret evidence about their own learning. By the end, the useful test is clear: look for reciprocal cognitive work, relevant evidence, and a defined path from the exchange to a learner decision—not simply a larger number of interface controls or a longer conversation.
Two meanings of interactivity
In ordinary software language, an interactive dashboard is one that responds when a user filters a date range, zooms into a graph, selects a competency, or asks for an explanation. These actions matter. A student who cannot find the relevant week or understand what a line represents may be unable to use the data at all. Narration and personalisation can lower the cognitive load of viewing a visualisation, and natural-language explanations can make a complicated display more accessible.
But access is only the first condition of learning from data. A filter changes what is visible; it does not tell the learner what the change means for their understanding or what to do next. A narration can define “mastery”; it does not require the learner to decide whether the definition fits their own experience. This is the difference between interface interactivity and pedagogical interactivity. The first changes the presentation. The second changes the relationship between the learner and the information.
The distinction can be made concrete with a familiar dashboard signal: a learner’s recent exercise scores have fallen. An interface may allow the learner to isolate those scores, compare them with an earlier period, or request a summary. A pedagogically interactive system goes further by asking the learner to interpret the pattern. Did the tasks become harder? Was the work rushed? Is the learner confusing completion with understanding? What evidence would support one explanation over another? The system need not dictate the answer. Its role is to make the learner perform the comparison that a static chart leaves implicit.
This matters because learning involves integrating new information with what a person already believes and knows. A dashboard’s estimate is new information about the learner, but it competes with the learner’s existing self-assessment. If the interface simply announces “your mastery is low,” the learner can accept the statement, dispute it, or ask what the word means. None of those responses necessarily requires a careful examination of their own study process. If the learner first predicts their mastery and then encounters the system’s estimate, the difference between the two becomes an object of thought.
The first study connects this idea to the ICAP framework, which distinguishes passive, active, constructive, and interactive forms of cognitive engagement. Its relevant point is not that every question is automatically educational. It is that eliciting an explanation, prediction, comparison, or judgement can require the learner to generate and connect ideas, while receiving information can leave more of that work outside the learner. The study applies this tutoring principle to a new setting: not learning a programming concept directly, but learning what one’s own analytics may mean.
The proposed mechanism has three parts. An eliciting prompt is the trigger. The learner’s explanation or judgement is the mediator: it makes an internal interpretation observable to the learner and, sometimes, to the system. Reflection or a better-calibrated estimate is the immediate outcome. A later study decision would be a further outcome. Keeping those steps separate prevents a common overclaim. A prompt can be designed to invite reasoning; a message category can show that a learner wrote about reflection; neither alone proves that the learner adopted a better strategy or learned more subject matter.
A useful design test is to inspect three moments rather than count features. First, what was the learner able to see before the interaction began? If the signal is vague, delayed, or stripped of the course context that gives it meaning, asking for a sophisticated interpretation may only create frustration. Second, what must the learner contribute? Selecting a bar on a chart is an action, but it is not necessarily an explanation or a prediction. The stronger question is whether the learner has to connect the signal to a recent task, a prior belief, or a possible cause. Third, what can happen after the exchange? If the system has no way to support a decision or revisit the evidence, the interaction may be engaging without being useful.
This test also explains why a question is not automatically better than an answer. An open prompt such as “What do you think?” may place a burden on the learner without giving them a meaningful object to inspect. A more useful prompt narrows attention to a relationship the data can actually support: “Your completion is high, but your recent exercise performance is uneven—what would you expect the mastery estimate to show, and why?” The learner is not being asked to guess a hidden truth. They are being asked to make a provisional interpretation that can be compared with evidence. The quality of the loop depends on that fit between the question, the available data, and the decision that might follow.
The same principle applies to disagreement. A mismatch between a learner’s judgement and a system rating can prompt reflection, but it does not identify the cause by itself. The learner may have misunderstood the rating scale; the system may be using a narrow definition of mastery; the recent task may be unusually difficult; or the learner may know something the recorded traces do not contain. A pedagogically interactive dashboard should make the discrepancy inspectable rather than treating the system value as the final authority. The purpose of comparison is to improve judgement, not to replace one unexamined belief with another.
There is a practical sequencing choice here. A telling explanation can come first when the learner lacks the vocabulary to understand a metric. Once the representation is clear, the system can ask the learner to apply it to their own evidence. In the opposite direction, a learner’s initial estimate can come first when the goal is calibration; the system can then explain the difference without turning the comparison into a test the learner is punished for failing. The two moves are compatible because they solve different problems: telling reduces an access barrier, while eliciting creates an opportunity for the learner to generate and examine meaning.
A dashboard can also support different kinds of decisions. A learner may need to decide whether to revisit a concept, seek help, change the timing of practice, or simply continue. The system should not infer that one low indicator dictates one correct response. The first study’s agent was instructed not to prescribe what students should do, reflecting the fact that study behaviour is complex and individual. That restraint is part of the pedagogy. The loop is stronger when the learner can use evidence to choose among plausible actions, rather than receiving an automated command whose rationale they cannot evaluate.
Finally, the interaction must have a stopping rule. A conversation that keeps generating explanations can feel productive while postponing the decision it was meant to support. The relevant endpoint is not “the learner sent another message”; it is a learner-owned interpretation or plan that can later be checked against new activity. This is also where a dashboard’s measurement responsibility begins. It should record enough context to revisit the signal and the learner’s response, while avoiding the assumption that every click or sentence is a transparent measure of cognition. An action loop is complete only provisionally, because the next trace may confirm, complicate, or overturn the original interpretation.
A diagram like this also clarifies what pedagogical interactivity is not. It is not a requirement that every dashboard become a Socratic tutor. It is not a measure of success based on the number of turns in a chat. It is a deliberate placement of cognitive responsibility. A telling response may be the right next step when the learner cannot decode a graph or needs a definition. An eliciting response may be more valuable when the learner understands the display but has not connected it to a decision.
What an agent contributes—and where it stops
The Interactive Learning Analytics Dashboard studied in the source material used an LLM to make this distinction operational. Its agent was powered by GPT-4o and did not receive every item of LMS data in one prompt. Instead, it used predefined tools to retrieve relevant information: exercises, scores and submission times, competency progress and mastery, Judgements of Learning, and lecture content when specialised knowledge was needed. That arrangement has a pedagogical as well as a technical purpose. The exchange can be grounded in the learner’s actual course context without flooding the model with irrelevant material, while the system retains control over which kinds of evidence are available.
The agent had two role versions. The eliciting version primarily asked questions about the learner’s data. The telling version primarily supplied information and answered questions about it. The prompts were otherwise kept alike, so the intended contrast was not two unrelated personalities but two ways of placing work in the interaction. Both versions were instructed not to provide exercise solutions or code and to acknowledge gaps rather than invent an answer. The design therefore treats role, data access, and guardrails as part of the educational intervention.
The difference between the roles is easiest to see in a sequence. Suppose the dashboard shows that a learner has completed most of the material connected with a competency but has uneven performance on its exercises. A telling agent might explain that completion and mastery are different, point to the relevant indicators, and answer a question about how the estimate was calculated. That can remove a genuine obstacle. A learner cannot reflect on evidence that they cannot interpret.
An eliciting agent might instead ask which part of the learner’s activity makes them confident, what the uneven scores suggest, or whether the learner’s estimate changes after looking at the recent tasks. The learner now has to relate system evidence to a mental model. The agent does not know, from the chart alone, whether the learner was distracted, misunderstood a concept, or solved a difficult problem slowly. Asking the learner to consider those possibilities can preserve the learner’s authority over context while still making the evidence harder to ignore.
Neither strategy is sufficient in every situation. Direct information is valuable for lower data literacy, clarification, and orientation. Persistent questioning can become friction if the learner needs a simple definition, is already well calibrated, or is trying to resolve an urgent organisational problem. Conversely, a smooth summary can produce the appearance of support while leaving the learner’s assumptions untouched. A good design must match the interaction to the cognitive problem rather than treating “ask” or “tell” as a universal rule.
The use of an LLM adds another boundary. In the deployment, one student message could generate between one and ten model requests depending on how much tool use and reasoning it required. In a small random sample of 284 responses, 13% were judged faulty—meaning they did not fully match the request or contained false or misleading claims—and 37% were judged very useful in context. Faulty responses were more common early in the five-week period, while later patterns suggested that students learned how to use the agent more productively. That later improvement may reflect learner adaptation, not a guarantee that the model became reliable.
This is why fluency cannot stand in for evidence. An LLM can produce a plausible explanation of a dashboard signal even when the underlying signal is incomplete or the learner’s question falls outside the available data. Retrieval tools reduce one source of error, but they do not establish that the retrieved metric is a valid measure of understanding. Role prompts can make a response more reflective, but they do not ensure that the learner will reflect. Guardrails can block exercise solutions or unsupported requests, but they do not eliminate misleading interpretations.
The agent’s proper contribution is therefore bounded. It can select relevant course evidence, phrase a prompt, explain a visible relationship, ask a learner to examine an assumption, and keep an exchange connected to the learner’s data. It cannot independently certify mastery, supply context that was never recorded, or guarantee that a conversation becomes a changed study practice. Those limits are not defects to hide; they define the responsibilities of the learner, the dashboard, and any human educator around it.
The most important design choice is consequently not whether a dashboard contains an LLM. It is where the system places the next piece of reasoning. If the learner cannot read the evidence, tell. If the learner can read it but has not interpreted it, elicit. If the model lacks the evidence, say so. If the exchange produces no plausible decision or later check, the interaction has stopped at conversation. The next case makes this mechanism concrete by changing not only what the dashboard says, but the order in which the learner and the system are allowed to judge.