What the Evidence Still Cannot Tell Us
A learner-facing dashboard claim should be audited at the same level of detail as the action loop it promises. This chapter examines the gap between what the two studies observed—traces, feedback quality, Judgements of Learning, self-regulated-learning proxies, and short-term self-reports—and what would be needed to show durable changes in study behaviour or learning. The practical outcome is an evidence test: locate the layer supporting a claim, identify the missing transition, and decide what additional observation would justify scaling the intervention.
From a recorded trace to a learning outcome
The first layer is the trace itself. An LMS can record a submission, a score, a timestamp, a viewed resource, or a chat message. That is evidence that an observable event occurred. It is not yet evidence that the learner understood a concept, managed time effectively, or reflected productively. The trace becomes educationally meaningful only through a model that connects it to a question about learning.
The second layer is the proxy. In the online statistics study, selected LMS events were processed into indicators for goal setting, task strategies, time management, help-seeking, and self-evaluation. Those indicators were standardised and combined into profiles. In the programming-course study, exercise performance, completion, lateness, recency, difficulty, and speed contributed to system estimates of progress, confidence, or mastery. These models make hidden patterns easier to discuss, but they remain interpretations of recorded behaviour.
A proxy can be useful without being complete. A pattern of late submissions may suggest a time-management issue, but it may also reflect a course deadline, work schedule, illness, or access problem. Repeated requests for clarification may indicate productive help-seeking or unresolved confusion. High activity may reflect persistence or inefficient struggle. The studies’ authors explicitly note that online logs capture visible actions and do not necessarily reveal internal cognitive or metacognitive processes. Precision in a proxy’s calculation does not remove uncertainty about what the proxy means.
The third layer is the learner’s interpretation. The ILAD measured this indirectly through JoLs and interactions. Learners became more consistent in their ratings over time, and the eliciting condition showed a positive trend in correlation with the system’s mastery estimate. The statistics study measured feedback-related changes in SRL proxy scores. These results matter because a dashboard is meant to help learners understand their own state, not merely classify them. Yet a correlation with a system rating shows alignment with that rating. It does not independently validate the learner’s understanding or the rating itself.
The fourth layer is enacted strategy change. Did the learner change when or how they studied, seek help differently, revisit a concept, or set a more specific goal? The programming study collected interaction logs and self-reports, including reports that some students changed behaviour after judging their learning. It did not directly observe changes in study scheduling or other behavioural outcomes.