Checkpoint: What Counts as a Successful Intervention?

The evidence so far concerns one subject: digital communication between a Public Employment Service and unemployed job seekers. The useful judgment is not whether the Swedish emails were “good” or “bad,” but where their effect appeared, where it stopped, and which explanations the data can support. Once reporting, applications, job finding, and job quality are kept as separate stages, the Swedish result is no longer a contradiction. It is a precise warning about confusing an institutional response with a labor-market result.

The chain is causal, but not continuous

The Swedish messages had a clear opportunity to matter. About 72% of messages in the observed cohorts were opened, and the first email reached the supportive and controlling groups at roughly comparable rates. That makes it difficult to explain the later pattern by saying that no one saw the intervention. But an opportunity for influence is not influence itself. The open rate does not tell us whether recipients read closely, accepted the premise, or changed how they searched.

Both treatments then increased the submission of monthly activity reports. The controlling emails produced the larger response: reporting rose by 1.13 percentage points over the first six months relative to the control group, compared with 0.76 percentage points for supportive emails. The control-group reporting share was 45.6%. The strongest response occurred for the first report and became smaller later. This is a real treatment effect on a behavior that mattered to benefit eligibility and to the agency’s ability to observe activity.

The next step was not automatic. The controlling treatment did not increase the primary measure of applications during months 2–7 or the primary measure of leaving unemployment within seven months. Its effects on related search and job-finding outcomes were also small and statistically insignificant. Reporting moved; applications and employment did not. A person could therefore complete the requirement without changing the search behavior that made a job more likely.

The supportive treatment went further in the wrong direction on some outcomes. It reduced applications by 0.094 per month over months 2–7, about 1.1% relative to the control mean of 8.286 applications. Job finding within seven months was not significantly different from control, while job finding within four and twelve months was slightly lower, with the twelve-month decline statistically significant. The study found no systematic improvement in job-search quality or job quality. The result is modest rather than catastrophic, but it is consistently misaligned with the intervention’s purpose in the measures that matter most.