When the Average Effect Disappears
The broad evidence on social-norm messaging asks a narrower question than “can people ever be influenced by norms?” It asks whether randomized health-behavior interventions produce a dependable average effect across the settings and populations studied. Reading that evidence well requires keeping four things separate: what the review included, how large the average was, how widely individual studies varied, and what happens when the analysis accounts for the possibility that positive results are easier to publish.
What the review measured—and what it did not
The review was a preregistered systematic review and meta-analysis of randomized controlled trials involving people aged 16 or older in developed countries. Its target was health behavior: examples included diet, screening, vaccination, alcohol use, prescribing, physical activity, sexual health, and hand hygiene. It included interventions that supplied normative information about a relevant group, whether through social proof, personalized comparison, injunctive approval, or a combination. It did not attempt to summarize every form of social influence. Developing-country populations, children under 16, non-randomized studies, and several specialized college substance-use literatures were outside its stated scope.
That boundary matters. A review of randomized health interventions is valuable for the claim it can test, but its conclusion should not silently expand into a claim about political norms, organizational culture, environmental action everywhere, or any message that happens to mention peers. The review found 110 papers during its searches. Sixty contained enough information for meta-analysis, and multiple studies within papers produced 89 individual studies with 85,759 participants. Those numbers provide a substantial base for an average estimate, but they do not make the included behaviors, countries, delivery channels, or outcome measures interchangeable.
A meta-analysis pools results by converting them to a common effect-size scale. Here the principal scale was Cohen’s d, a standardized difference between intervention and comparison conditions. It is useful because one study may report a change in consumption, another a screening decision, and another a clinical prescription. Standardization permits comparison, but it also removes some of the everyday meaning of the outcomes. A difference of the same standardized size can represent very different consequences depending on whether the target is a one-time intention, a prescription pattern, or a repeated health behavior.
In the conventional random-effects analysis, social-norm messaging had an average effect of d = 0.14, with a 95% confidence interval from 0.09 to 0.19. The result was statistically distinguishable from zero, but its magnitude was small. Sensitivity analyses did not transform it into a large effect: removing influential outliers produced d = 0.11, while excluding studies rated high risk or having some concerns produced d = 0.19. Statistical detectability answers whether the pooled estimate is unlikely to be exactly zero under a particular model. It does not answer whether the change is large enough to matter, replicates in a new setting, or persists after the message is gone.
The average also hides a wide distribution. Four studies reported large positive effects, nine moderate positive effects, and 22 small positive effects. Thirty-six produced effects below d = 0.2, and 18 showed trivial-to-small negative effects. The review estimated a prediction interval from d = −0.43 to 1.19. A prediction interval is not a promise that every future intervention will land at one of those endpoints. It is a warning that the underlying effects across studies are dispersed enough that a new application could perform much worse—or much better—than the pooled average.
This is the first important distinction: a small mean is not evidence that every intervention produces a small change. It is a summary of unlike studies under a model that assumes the true effects vary around a distribution. If the practical decision is whether to place a message in one clinic, university, or public facility, the average may be less informative than the reasons the results differ. The review’s heterogeneity estimate was moderate, and the authors caution that pooling substantially different behaviors, populations, and delivery methods limits what the pooled value can explain.
The confidence interval around the mean and the prediction interval answer different questions. The confidence interval describes uncertainty about the estimated average under the analysis. The prediction interval incorporates the fact that a new intervention may come from a different part of the underlying distribution. A narrow confidence interval around a small mean can therefore coexist with a wide prediction interval. There is no contradiction in saying that the average was estimated fairly precisely while also saying that the next implementation is difficult to predict.
That distinction changes how a decision maker should use the number. If a message costs almost nothing and causes no harm, a small genuine effect might still be worth pursuing at population scale. But that argument depends on the effect being real, on delivery reaching the intended audience, and on the behavior being important enough to justify attention. A small published estimate that disappears under a bias-aware analysis cannot carry the same decision weight. The relevant question is not “is 0.14 positive?” but “how much confidence should be placed in a small shift across this kind of behavior, for this kind of audience, under these conditions?”
A pooled result can also obscure asymmetric consequences. If a norm message helps some people but causes others to move toward a less desirable average, the mean may conceal that trade-off. If a message changes intentions more readily than actual behavior, combining both outcomes may make the intervention look more behavior-changing than it is. The review’s broad evidence is useful precisely because it reveals these limits: the aggregate estimate is a starting point for judgment, not a portable performance guarantee.
Why the positive signal is not the final answer
The more serious challenge concerns how the literature came to contain the results it contains. Studies with positive findings may be more likely to be written up, accepted, highlighted, or selected for analysis than studies that find no effect. If so, a collection of published estimates can look mildly positive even when the set of studies that could have been conducted is closer to null. This is publication bias—not necessarily deliberate misconduct, but a selection process in which the visibility of a result depends partly on its direction or apparent newsworthiness.
The review found a pattern consistent with that concern. Positive effect sizes were overrepresented among studies with low precision, meaning studies whose estimates were more uncertain. The authors report funnel-plot asymmetry and a significant Egger test, but they also warn that visual inspection and Egger tests are often underpowered and are not conclusive on their own. The important move was not to treat one diagnostic as proof. It was to apply robust Bayesian meta-analysis that models the possibility of selective publication and averages across complementary assumptions about the bias.
Before the adjustment, a Bayesian model-averaged analysis reproduced the small positive estimate: d = 0.14, with a 95% credible interval from 0.09 to 0.19. After robust adjustment for publication bias, the estimate fell to d = 0.01, with a 95% credible interval from 0.00 to 0.09. The review reports strong evidence for the null in that adjusted model and very strong evidence for publication bias. The difference between 0.14 and 0.01 is not a cosmetic change in presentation. It changes the practical interpretation from “there is a small average benefit in the observed literature” to “the observed small benefit does not remain a reliable average once a plausible selection process is modeled.”
That conclusion still needs careful wording. The adjustment does not show that publication bias explains every positive result. It does not show that every norm appeal is ineffective, nor that a well-designed intervention cannot work in a particular setting. Bayesian estimates also depend on the models and priors used, although the review reports sensitivity analyses across different priors. The defensible conclusion is narrower and stronger than a slogan: the broad evidence is compatible with no dependable average health-behavior effect after accounting for publication bias.
The review then examined whether a predictable subgroup could rescue the intervention claim. It considered the type of norm message, the delivery format, the target behavior, and the target population. In frequentist analyses, omnibus tests indicated variation across categories. But an omnibus result only says that some variation exists somewhere in the set; it does not identify a category that reliably beats the others. When individual levels were compared with the grand mean in Bayesian and bias-adjusted analyses, none showed a meaningful departure from the overall effect. A small positive deviation among university students in the conventional model disappeared after publication-bias adjustment. The adjusted evidence did not establish a dependable advantage for a particular message type, channel, health domain, or population.
This is why a checklist such as “use descriptive norms,” “deliver digitally,” or “target students” cannot be extracted from the review as a general recipe. The absence of a reliable category-level advantage is not evidence that wording, channel, or audience never matters. It means that the available studies do not support a stable rule at that level of classification. The categories were also correlated. Because a multiple meta-regression would have suffered from intolerable multicollinearity, the authors avoided presenting an unstable model that pretended to disentangle effects it could not cleanly estimate. Heterogeneity remained substantial within the moderator analyses, suggesting that the important conditions may lie below or across these broad labels.
There is a difference between a moderator that is too broad to explain variation and a condition that has no influence. “Delivery mode” can group together a carefully timed reminder, a poster passed in a corridor, a personalized clinical letter, and a message shown inside an app. Their common channel may tell us little about whether the person was paying attention, whether the action was available, or whether feedback arrived at the right moment. Likewise, “population” can combine people who already perform the behavior with people at higher risk, or people who see the reference group as meaningful with people who do not. When a broad category contains these contrasts, a null category-level comparison can coexist with important case-level differences.
The same caution applies to the review’s behavior domains. Diet, vaccination, prescribing, screening, and exercise differ in effort, timing, repetition, institutional support, and the consequences of noncompliance. A norm message may be able to influence a low-friction choice more readily than a behavior that requires access, planning, or sustained resources. The available moderator analysis did not isolate those mechanisms. It therefore tells practitioners what not to assume—there is no demonstrated universal recipe—while leaving the more useful explanatory work to better specified studies.
The diagram captures a change in evidentiary status, not a sequence in which one study literally turns into another. The conventional estimate is a legitimate description of the unadjusted data. The bias-aware estimate is a more cautious judgment about how much confidence that description deserves. Heterogeneity remains on the path throughout: even if the average is near zero, some local effects may be positive, some null, and some adverse. A near-null average is therefore not the end of inquiry. It is a reason to ask what conditions distinguish cases without assuming that the first visible category will supply the answer.
There is also a measurement issue behind the review’s caution. The included studies combined objective behaviors with intentions and used follow-up periods ranging from immediate measurement to two years. Many clustered between one and six months, but duration was not uniform. A message that changes a stated intention immediately is not equivalent to one that changes a prescription pattern for months. A one-time choice can be meaningful for a public-health program, but it cannot be called maintenance without evidence that the behavior continues. Pooling different endpoints can be useful for a broad orientation, yet it makes the average less specific than the label “behavior change” suggests.
The review further notes that only 10 of the 89 studies elicited perceived norms before intervention, and only 17 included manipulation checks verifying awareness or understanding of the norm. That limitation does not invalidate the randomized outcome comparisons. It limits the explanation of them. A null outcome might mean the intervention did not alter behavior after changing perception, or that the message never changed perception in the first place. A positive outcome might reflect the norm, but it might also reflect salience, humor, feedback, or another feature bundled into the intervention. The broad review can tell us that the overall outcome claim is fragile; it is less able to tell us which psychological link failed in each study.
The resulting judgment is a disciplined middle position. The review does not justify treating social-norm messaging as a reliable general-purpose button. Nor does it justify declaring every positive field result illusory. It says that a small published average is vulnerable to selection, that individual effects are heterogeneous, and that the tested broad categories do not supply a trustworthy shortcut for choosing an intervention. A local success remains a question worth studying precisely because it may reveal a condition hidden by the average. The next step is to inspect such conditions in settings where behavior is observed directly, while keeping the review’s warning in view: a real effect in one place is evidence about that place first, and about a universal button only much later—if at all.