What “Effective” Means Depends on the Question
For influenza public health and social measures, the first task is not to decide whether a reported percentage is impressive. It is to identify what that percentage estimates, who was compared with whom, and which decision the result can inform. This chapter gives readers a practical way to take apart an effectiveness claim so that they can distinguish effect magnitude from evidence certainty and from the broader judgment of whether a measure should be used.
Start with the question, not the percentage
An effectiveness claim has a structure, even when a news headline hides it. At minimum, it names or implies a measure, a target population, a comparator, an outcome, a time horizon and an objective. Change any one of those elements and the meaning of the result may change.
Take the measure first. “Masks” could refer to symptomatic people wearing a medical mask for source control, everyone wearing a mask in crowded indoor spaces, or a different type of face covering used with different instructions. “School measures” could mean spacing desks, excluding ill children, separating classes, staggering breaks or closing schools. These are related policies, but they do not expose people to the same intervention. A claim about one cannot automatically be assigned to all the others.
The target population matters just as much. A community intervention can be directed at infected people, household contacts, schoolchildren, workers, travellers or the general public. It can be implemented in homes, schools, workplaces, transport hubs or across a whole country. The population determines which contacts are changed and which opportunities for transmission remain. Evidence from a household does not automatically describe a national policy, and evidence from a university residence does not automatically describe a crowded multigenerational household.
The objective supplies the next distinction. WHO describes PHSM as capable of preventing or delaying introduction, suppressing community transmission, or mitigating the impact of an epidemic or pandemic. A policy intended to delay importation may be judged by time gained before local seeding. A policy intended to suppress spread may be judged by infections or the effective reproduction number. A policy intended to mitigate impact may be judged by hospital demand, deaths or the height of a peak. The word “effective” does not tell us which of these purposes is being evaluated.
That is why endpoints must be read literally. Transmission is not the same as infection. Total infections are not the same as hospitalizations, and hospitalizations are not the same as deaths. A measure can reduce the height of an epidemic peak without changing the eventual cumulative number of infections very much. It can delay a peak long enough to protect hospital capacity or buy time for preparedness. Those outcomes can be valuable even if they do not produce a large reduction in the final case count. Conversely, a small delay may be inadequate if the decision requires sustained suppression.
A useful question is: what would count as success if the measure worked? If success means fewer infections, the analysis needs an infection outcome over a defined period. If it means avoiding a sudden surge in hospital demand, the timing and shape of the epidemic curve matter. If it means buying time before a vaccine or antiviral becomes available, the value of delay depends on what authorities can accomplish during that interval. These are not competing definitions that can be averaged into a single score. They are different policy objectives.
The comparator is equally important. WHO’s effectiveness reviews generally compared an intervention with a period of influenza or another respiratory-virus circulation in which no intervention was implemented. Other studies may compare a measure with usual practice or with a different package. “No intervention” is not always a literal absence of behavior change; people may alter their behavior in response to perceived risk even without a formal policy. “Usual practice” may already include hygiene, isolation or ventilation. A comparison with another package asks a different question from a comparison with doing nothing.
Time horizon changes the result as well. A measure introduced before a community is seeded may delay introduction. The same measure introduced after sustained local transmission may have little effect on importation but could still affect contacts or the peak. A short intervention may reduce growth temporarily and then be followed by renewed transmission when it is relaxed. A prolonged intervention may produce a different epidemiological effect because adherence and social conditions change over time. A result covering one week cannot answer a question about an entire pandemic wave without additional assumptions.
The source of the result should remain visible. A randomized controlled trial assigns people or settings to different approaches and can make groups more comparable, although community PHSM are difficult to randomize cleanly and some comparisons are ethically or practically constrained. An observational study examines what happened under real conditions, which can reveal implementation and population behavior but can also be affected by changes that occurred alongside the measure. A simulation explores a set of assumed conditions and can test scenarios that are difficult to observe directly, but its conclusion is conditional on those assumptions. These designs contribute different kinds of knowledge; they should not be treated as interchangeable containers for the same fact.
Here is the basic interpretive sequence:
The sequence is not bureaucratic. It prevents a result from being detached from the question that generated it. If a study reports a change in transmission, it should not be quoted as though it directly measured deaths. If it models a delay under early border control, it should not be used as evidence that the same policy lowers the final caseload after local seeding. The more a claim travels from its original question, the more carefully its boundaries must travel with it.
How evidence becomes a recommendation
Once the estimate has been identified, the next task is to judge how much confidence it deserves. WHO uses the GRADE approach for the question of whether a PHSM can reduce influenza transmission in community settings. In broad terms, randomized trials generally begin with greater certainty than observational studies, and observational studies generally begin with greater certainty than simulations. But study design is only the starting point. Certainty can be lowered by risk of bias, inconsistent findings, indirect evidence, imprecise estimates or publication bias.
These terms describe different problems. Risk of bias concerns systematic ways a study may depart from the effect it is trying to estimate. Inconsistency means studies do not point to a sufficiently similar result. Indirectness arises when the population, intervention, comparator or outcome differs from the question of interest. Imprecision means the data do not narrow the plausible range enough. Publication bias means the available literature may not represent all of the studies that were conducted. A result can therefore be statistically clear in one dataset and still provide limited confidence about a different population or policy.
Point estimates and intervals must also be kept separate. The WHO influenza review of community mask wearing included 12 randomized trials with more than 20,000 participants. Its pooled point estimate suggested a 12% relative reduction in laboratory-confirmed influenza among mask wearers, expressed as a risk ratio of 0.88. The 95% confidence interval ran from 0.61 to 1.27, and the review said the evidence was insufficient to exclude chance as an explanation for the apparent reduction.
Three cautions follow from that result. First, the point estimate is not a guarantee that every mask policy reduces influenza by 12%. It summarizes the data under the included trial conditions. Second, a relative reduction is not the same as a reduction of 12 percentage points. To calculate an absolute number of infections prevented, one would need the relevant baseline risk and the population to which the estimate is being applied. Third, the interval is part of the result. It is compatible with a meaningful reduction, a small reduction and little or no reduction in the studied comparison. The correct interpretation is a modest point estimate with substantial uncertainty, not a universal mask number and not proof that masks do nothing.
This distinction between estimate and certainty also explains why a large apparent effect does not automatically deserve high confidence. A simulation may project a large reduction under stringent adherence, but the projection may be highly sensitive to assumptions about timing or behavior. An observational study may find a large association while other policies are changing at the same time. Conversely, a modest effect from a well-conducted comparison may be more reliable for the population and outcome actually studied. Magnitude tells us how large the estimated difference is; certainty tells us how much the evidence supports treating that estimate as close to the truth for the question at hand.
WHO’s recommendation categories add another layer. The guideline reports both the certainty of evidence and the strength of recommendation. A strong recommendation reflects confidence that desirable effects outweigh undesirable effects in the decision context. A conditional recommendation reflects greater uncertainty or a balance that may change with values, resources or local conditions. Neither label is a disguised effect-size ranking. A strong recommendation can accompany a low-certainty estimate when the likely benefits, low harms and feasibility support action. A conditional recommendation can concern a measure with plausible benefit that is worthwhile only for particular severity levels or settings. A recommendation against a measure is also not necessarily a claim that its biological effect is exactly zero; it can mean that the likely benefit is too uncertain or too small to justify the burden in the circumstances considered.
The WHO table makes this visible. Hand hygiene receives a strong recommendation with moderate-certainty evidence. Mask wearing receives a conditional recommendation with moderate-certainty evidence, aimed especially at symptomatic or positive people around others and at crowded indoor settings during high- or extraordinary-severity influenza. Increased ventilation receives a strong recommendation even though its evidence is low certainty, in part because the likely harms are limited and the mechanism is plausible. School measures and closures receive a conditional recommendation with very low-certainty evidence, with sustainable alternatives preferred to full closure and coordinated proactive closure reserved for high or extraordinary severity. Travel restrictions receive a conditional recommendation against with very low-certainty evidence because any benefit may be limited to early delay while the economic and social costs can be substantial.
Those differences do not form a contradiction or a ladder. They show the evidence-to-decision process at work. Certainty matters, but so do the balance of benefits and harms, resource implications, equity and human rights, acceptability and feasibility. A measure may be hard to study but inexpensive, low-risk and easy to implement. Another may have a plausible effect but impose severe disruption or unequal burdens. The recommendation asks what should be done in a decision context; the estimate asks what effect was observed or projected under specified conditions.
This is also how to read disagreements between studies. Some apparent conflicts are genuine uncertainty: studies may estimate different effects and the interval may be wide. Others are differences in question. One study may measure laboratory-confirmed infection while another measures influenza-like illness. One may evaluate source control by symptomatic people while another evaluates community-wide use. One may look at a package that includes isolation and hand hygiene, while another tries to isolate a single measure. The settings may differ in baseline transmission, age structure, housing or access to support. A careful synthesis preserves these distinctions instead of choosing the most favorable number.
New evidence would matter when it reduces one of those uncertainties. Better randomized or quasi-experimental studies could clarify effects under defined conditions. More complete reporting of adherence could show whether a modest intention-to-treat result hides a larger effect among people who used a measure correctly. Studies that separate bundled interventions could improve attribution. Local surveillance could test whether a finding transfers to the current population and epidemic phase. Economic evaluations from low-resource settings could change judgments about affordability and burden. None of these developments promises one final number; they make particular conditional claims more precise.
A reader can therefore audit an influenza PHSM result with a short set of questions: What measure was actually implemented, and for whom? What outcome and objective were measured? Compared with what, and over what period? Which design produced the estimate, and what assumptions or sources of bias remain? How wide is the uncertainty, and does the studied population resemble the one under consideration? Finally, what benefits, harms, costs and rights are included in the recommendation that follows? The answers establish what the evidence supports—and just as importantly, what it does not establish.
Once those boundaries are clear, evidence can be connected to mechanism. The next problem is to understand why measures aimed at different points in transmission generate different kinds of results, and why a package can matter even when no single component has a cleanly separable effect.