When the Numbers Mislead: Causality, Measurement, and Fairness

A social and emotional score can be statistically useful and still be a poor basis for judging an individual. This chapter examines what the supplied studies can support, what remains uncertain, and why a signal that predicts an outcome in a population can become unfair when used as a gatekeeping rule. The practical goal is not to distrust every association. It is to know when a result describes a pattern, when it suggests a mechanism, and when it is too indirect to guide a high-stakes decision.

An adjusted association is not a causal result

The OECD analysis uses a broad adult survey and adjusts its estimates for characteristics such as education, literacy, age, family circumstances, and other demographic factors. Those controls improve comparability. They make the comparison less likely to confuse a social and emotional domain with differences the model explicitly includes. But adjustment changes a question; it does not answer every question.

Suppose higher emotional stability is associated with employment after literacy and education are held constant. The result says that the association remains in the adjusted comparison. It does not prove that increasing emotional stability would cause someone to obtain a job. Unmeasured health, social support, local opportunity, discrimination, prior work history, and access to credentials may still matter. A control can reduce one source of confounding while leaving others untouched. It can also remove part of the pathway the reader wants to understand. If education helps develop a behavior that employers reward, controlling for education may reduce the relationship even though the behavior is part of how education affects work.

The strongest warning is reverse causality. Employment can change the very characteristics being measured. Work supplies routine, social contact, and opportunities to practice communication or persistence. Unemployment can bring financial pressure, isolation, disrupted routines, and stress. Those experiences may change how someone behaves and how they answer a questionnaire. A cross-sectional survey that measures employment and a social and emotional domain at one point cannot reliably separate “the trait helped produce employment” from “employment helped produce the measured trait,” or from a feedback loop in which both influence each other.

This does not make the OECD result useless. It means the result is best treated as evidence of a conditional relationship and a prompt for mechanisms, not as an intervention estimate. Longitudinal data, credible natural experiments, repeated measures, or designs that establish temporal order could change how confidently a causal interpretation is made. Until such evidence is available, a careful sentence says “is associated with” or “is consistent with,” not “causes.”

The World Bank wage synthesis has a related boundary. It searched a much larger literature, then narrowed 1,896 coefficients to 613 estimates from 16 papers for the wage analysis. The paper includes both ordinary least squares and instrumental-variable estimates, but the presence of an instrument or a reported coefficient does not make the pooled result automatically causal. The studies differ in samples, measures, specifications, and populations. The random-effects method allows the underlying effects to vary; it improves the summary of heterogeneous evidence, not the identification of a single universal effect.

A meta-analysis answers a different question from an experiment. It can indicate whether a pattern recurs across studies and give a weighted estimate of its average size.