What the Evidence Shows—and What It Does Not
The evidence for self-directed learning in health professions education is positive, but it is not a single verdict. This chapter interprets two systematic reviews by separating the size of their observed benefits from the conditions, outcomes, and study designs behind them. The goal is practical: readers should be able to explain what a pooled result means, why high heterogeneity limits a headline claim, and which conclusions about clinical performance or patient care would go beyond the evidence.
The distinction matters because “SDL works” can hide several different propositions. It might mean that learners score higher on a knowledge test, perform a practical task more accurately, regulate their own learning more effectively, or eventually change behavior in practice. Those are not interchangeable outcomes. A review can support one proposition while leaving the others uncertain.
The broad HPE estimate
The larger review examines self-directed interventions across health professions education rather than only one student population. Its evidence base includes medicine, nursing, dentistry, and other health professions. The review identified 6,786 studies during its search process and included 125 studies in the final evidence base. It then used a more restrictive set of studies for the meta-analysis: 48 studies contributing 74 effect sizes. The analysis used a three-level random-effects model because several effect sizes could come from the same study, making them related rather than fully independent observations.
The pooled Cohen’s d was 0.34, with a 95% confidence interval from 0.04 to 0.64. In ordinary language, the average difference favored SDL by a small-to-moderate amount under the conditions represented in those studies. The confidence interval is important. It does not say that the true effect for every intervention lies inside the interval, nor that a future course will produce the average result. It expresses uncertainty around the estimated average in this evidence base. The lower end is close to zero, while the upper end is meaningfully larger, so the estimate supports a qualified benefit rather than a precise promise.
The estimate also comes with substantial heterogeneity: I² was 87%. Heterogeneity asks how much the observed differences among study results exceed what would be expected from sampling variation alone. An I² of 87% does not mean that 87% of learners benefited or that 13% did not. It means that the studies differ considerably in their observed effects, and that the pooled average compresses important variation. The differences may involve the learners, professional field, goal, modality, comparison condition, teacher role, assessment, or implementation quality. The average is therefore useful as an orientation, but it cannot tell an educator which local design will reproduce it.
The outcome distribution makes the boundary clearer. The review reports 98 of the 125 studies as Kirkpatrick Level 2 outcomes, which include knowledge and skills measured through tests or formal assessments. Twenty-seven studies assessed Level 3 behavior, such as clinical performance, and none assessed Level 4 patient or system outcomes. The absence of Level 4 studies is not evidence that SDL has no effect on patients or health systems. It is evidence that this review did not find studies testing those outcomes under its inclusion criteria.
That distinction between knowledge, performance, and patient-level impact is more than a technical classification. A learner can remember a guideline without applying it in a busy setting. They can perform a simulated procedure under familiar conditions without adapting it safely when conditions change. And even a change in observed behavior may not translate directly into a measurable patient or system outcome. Each step adds context, time, and other causes. A review dominated by Level 2 outcomes can establish that SDL is associated with improved measured learning, but it cannot silently carry that conclusion all the way to patient care.
The review also reports a difference depending on how SDL was positioned in a study. When SDL was the experimental intervention, the estimated effect was d = 0.54, with a 95% confidence interval from 0.24 to 0.84. When SDL was used as a comparator condition, the estimate was d = −0.27, with a 95% confidence interval from −0.75 to 0.22. The difference between these groups was statistically significant. This pattern suggests that SDL may work better when deliberately designed as the active intervention than when it is treated as a loosely specified control or background condition.
That is a design-relevant result, but it is not a license to treat group assignment as the mechanism. An intervention introduced intentionally may receive clearer goals, more coherent resources, more attention from educators, or a better match between activity and outcome. Conversely, SDL used as a comparator may be under-described or may be asked to do a different job. The statistical contrast tells us that the role assigned to SDL matters; it does not isolate every feature that made the role matter.
There is also a reporting complication that should remain visible. The article’s abstract says that 48 studies were eligible for meta-analysis. In the detailed results, it says that 49 articles met the criteria for meta-analysis, then reports a three-level analysis of 48 studies and 74 effect sizes. The safest interpretation is to use 48 studies and 74 effect sizes for the analysis actually reported, while treating 49 as a difference in screening terminology or reporting. Quietly choosing one number as if the source were internally uniform would create false precision.
The evidence map shows why a review result must be read as a chain. Included studies produce particular measurements; measurements are combined into an estimate; heterogeneity limits how directly that estimate travels to a new setting; and the outcome level determines what kind of claim is even available. None of these steps makes the result unimportant. They tell us what the result is about.
A narrower comparison and a wider caution
The second review asks a more specific question: how does SDL compare with traditional didactic learning in undergraduate medical education? It screened 2,955 titles and abstracts. Ninety-five articles reached full-text review; 19 articles met the criteria for the systematic review, and 14 met the criteria for the meta-analysis. The systematic review covered 2,098 students, while the quantitative comparison covered 1,792. These denominators are not interchangeable. One describes the broader included evidence; the other describes the subset with data suitable for the pooled exam-score analysis.
Across the 14 meta-analyzed studies, the mean exam-score difference favored SDL by 2.399 points. The 95% confidence interval was 0.121 to 4.678, and the reported p value was less than 0.001. Unlike Cohen’s d in the broader review, this is a difference in the score scale used by the included studies. It should not be compared as if 2.399 points were another standardized effect size. The result indicates that, on the measured exams in these studies, SDL performed better on average; it does not tell us that the same number of points will appear on a different exam or in a different curriculum.
Again, the most important companion statistic is heterogeneity. I² was 98.56%, which is extremely high. Eleven of the 14 studies favored SDL and three favored traditional didactic learning, but a majority of favorable studies does not erase the size and direction of the differences among them. The studies covered different topics, learner years, countries, sample sizes, instructional formats, and outcome measures. Some involved foundational or theoretical learning; others involved practical or clinical tasks. A pooled score difference can summarize those studies while still leaving the reasons for the differences unresolved.
The subgroup estimates make that unresolved question concrete. For theoretical SDL, the mean difference was 2.667, with a 95% confidence interval from 0.009 to 5.325 and I² = 98.76%. For practical SDL, the mean difference was 1.339, with a 95% confidence interval from −1.591 to 4.268 and I² = 91.76%. The practical interval crosses zero, so this review does not establish a reliable average advantage for practical SDL over traditional teaching. That is not evidence that practical SDL fails. It is evidence that the estimate is too uncertain, heterogeneous, or close to no difference to support a confident general advantage.
The underlying cases explain why a single label is insufficient. In the included studies, SDL was used for topics such as neuroscience, ECG interpretation, anatomy, online modules, and procedural training. Some comparisons favored SDL; one laparoscopic skills comparison favored supervised training on a task, and a study of ultrasound-guided vascular access found better performance for face-to-face teaching than for self-guided or no-education conditions. Another comparison of SDL and classroom teaching for Barrett’s esophagus diagnosis found no significant difference. These examples do not form a ranking of methods. They show that the learning goal, the task’s complexity, the opportunity for practice, and the availability of feedback can change the result.
The reviews’ quality assessments reinforce the same caution. In the broad HPE review, the mean MERSQI score was 12.65, with a standard deviation of 1.23 and a range from 10 to 15.5. Twenty-six studies, or 21%, were rated high quality by the review’s threshold, while 99, or 79%, were rated average; none was rated low. Yet 93% of studies came from a single institution, 58% reported no validity measures for their outcome instruments, and the review did not use a dedicated risk-of-bias instrument or exclude studies based on quality score. A quality score is evidence about study features, not a guarantee that the estimate is unbiased.
The undergraduate review reports its own limits: CASP scores ranged from 8 to 14; study designs and sample sizes varied; interventions covered only a limited subset of undergraduate topics; learners came from different years and educational cultures; and publication bias remains possible. The authors also note that SDL interventions vary in format, duration, content, and delivery. These qualifications are not ritual disclaimers appended after the result. They explain why the pooled estimate cannot be converted into a universal instructional rule.
Three conclusions are therefore not supported by these reviews. First, they do not show that every SDL intervention improves learning. High heterogeneity and several unfavorable or null comparisons rule out that interpretation. Second, they do not show that SDL is generally better than structured teaching for clinical skills or behavior. The broad review found smaller estimates for clinical performance and behavior than for test scores, and the undergraduate practical subgroup included no-difference uncertainty. Third, they do not show improved patient or system outcomes, because the broad review found no Level 4 studies and the undergraduate review centered on exam scores.
The evidence does support more limited conclusions. SDL can improve measured knowledge and skills in some health-professions settings. In undergraduate medical education, the pooled exam-score result favors SDL, while its very high heterogeneity makes the average a conditional finding. Deliberately designed SDL appears more promising than an under-specified SDL comparator in the broad review. And the differences among study results make implementation—not just the label or delivery technology—the next question.
That last point returns us to the learning cycle. If studies do not report who set the goal, selected resources, chose the strategy, evaluated performance, or received feedback, we cannot easily tell whether a result belongs to authentic self-direction, to a particular content intervention, or to an unmeasured source of support. The next task is to use the cycle and these outcome boundaries as a checkpoint: inspect an intervention’s learner control separately from the result it produced.