The Statistical Fault Line: When One Trial Becomes Many Outcomes

The central statistical problem in the creatine evidence is not that the 2024 review used an advanced method that readers cannot understand. It is that several measurements may have been counted as though they were independent evidence even when they came from the same participants. The practical task here is to distinguish trials, participants, outcomes, and effect sizes, then see why that distinction can change the apparent precision of the memory result without proving that creatine has no cognitive effect.

The count mismatch is a symptom

A randomized trial is an experiment that assigns people to an intervention or a control condition. Participants are the people whose outcomes are observed. An outcome is a measured result, such as a memory score or the time needed to complete an attention task. An effect size is a summary of the difference between groups on that outcome. These four units can be related, but they are not interchangeable.

One trial can contain many participants. One participant can complete several cognitive tests. One test can be converted into one or more effect sizes, depending on how the analysis represents the result. If a reader sees “24 studies” in one analysis and “16 studies” in the review overall, the first question should not be which number looks more persuasive. It should be what the paper is calling a study or observation at each point in the analysis.

The 2024 review states that 16 studies with 492 participants were included overall. Its memory analysis is described as containing 24 studies and 1,000 participants. The intervention table also shows trials contributing multiple memory subtests. This difference does not, by itself, prove that the 2024 paper’s conclusions are invalid. It does show that the memory result cannot be understood from its headline effect size alone. The denominator and the unit being pooled need to be reconstructed.

The 2026 commentary makes the concern more specific. It points to trials in which the same participants supplied several correlated cognitive outcomes. Examples named in the commentary include studies by Alves, McMorris, and Pires, where a single trial contributed multiple memory subtests. Those subtests may be useful measurements, but they do not represent newly randomized groups of people merely because they are separately scored.

This is the difference between more measurement and more independent evidence. Measuring one group with five tests can tell us more about the range of tasks they completed. It does not automatically give us the same information as conducting five separate randomized trials with different participant groups. The tests share the participants, and therefore their results tend to share features of those participants: their baseline ability, health, motivation, learning during testing, and response to the intervention.