The Hidden Difference Between Compliance and Effective Search
For digital interventions aimed at unemployed job seekers, the crucial analytical task is to locate the effect in the chain from a message to a job. This chapter focuses on the difference between an agency’s observable records and a person’s effective search: whether a message is noticed, whether a report is submitted, where the person looks, and whether the search produces work. That distinction supports a practical judgment about evidence. A first-stage response is useful only if it is connected to the outcome the intervention is meant to improve.
The distinction is easy to lose because agencies naturally count what their systems record. An email platform can report an open. A benefits system can report a submitted activity form. A job board can report a click or an application. These measures are convenient, frequent, and often available for nearly everyone in a trial. Employment, by contrast, may take months to appear and can be affected by the type of job found, the number of hours worked, and conditions in the surrounding labor market. A serious evaluation needs the convenient measures, but it cannot stop there.
The employment funnel is not one outcome
The first stage is exposure. Did the intended recipient receive the message, open it, or click one of its links? Exposure matters because a message cannot affect someone who never encounters it. But exposure is an opportunity for a mechanism, not evidence that the mechanism operated. An open does not show careful reading, agreement, comprehension, or action. It also cannot tell us which sentence mattered, if any.
The Swedish experiment makes this point unusually clear. Across two cohorts, 72% of 15,436 messages were opened. The first email was opened at nearly the same rate in the supportive and controlling groups—about 75% in each—so the two message schemes had a substantial chance to reach recipients. Later behavior differed somewhat: the controlling version generated higher subsequent opens and conditional clicks. That difference confirms that the content could influence engagement, but it does not yet tell us whether people searched more effectively.
The next stage is administrative compliance. A recipient may submit the activity report required to remain eligible for unemployment benefits. This is a real behavior, not a meaningless proxy. Reporting can keep a person in contact with the employment service, create information for a caseworker, and be part of a genuine search process. Yet a report can also be completed because a deadline was made salient, while the underlying search remains unchanged. The meaning of the response depends on what the report captures and what it leaves out.
In the Swedish trial, both motivational email schemes increased reporting. Over the first six months, the controlling messages raised the share of submitted reports by 1.13 percentage points relative to the control group; supportive messages raised it by 0.76 percentage points. The control-group mean was 45.6% over that interval. The effects were larger for the first report and diminished later, a pattern consistent with a deadline reminder having its strongest effect when the message is first encountered. This is evidence of a behavioral response at the reporting margin. It is not evidence of a corresponding rise in applications or employment.
That gap becomes visible at the search stage. Job search is not simply an amount of activity. It includes the number of applications, the occupations considered, the channels used, the quality of applications, and the match between a person’s capabilities and the opportunities available. More activity in a poor market may be less valuable than a smaller amount of activity directed toward a better market. Conversely, a message that changes the number of recorded activities without changing the search direction may improve the agency’s compliance statistics while leaving the job seeker’s prospects largely where they were.
The downstream outcome is job finding, but even that label needs interpretation. A binary employment indicator tells us whether someone worked at all during a period. Hours worked show whether the change is only a small part-time foothold or a more substantial employment gain. Earnings add information about the economic value of the work. Job duration, satisfaction, perceived fit, or the occupation entered can reveal whether a faster exit from unemployment came at a cost. The Dutch experiment illustrates why these distinctions matter: its reported gains were concentrated on the extensive margin, meaning that more people worked, rather than already-employed recipients working substantially more.
This layered view prevents a common error: treating every positive movement as evidence for the same theory. A message may be good at making deadlines salient but poor at changing search strategy. It may increase the number of people who inspect alternatives but not the number who apply. It may increase employment while changing occupational destinations, which would support an information mechanism rather than a general increase in effort. Each result answers a different question.
The arrows should not be read as a guarantee that each stage causes the next. They describe the path an intervention must travel if an agency wants a change in messaging to become a change in employment. Institutional rules can directly affect reporting and search. Labor-market tightness can alter the return to an application even when individual effort is unchanged. Spillovers can change opportunities for people who did not receive the message. These conditions are part of the causal environment, not noise to be removed from the story.
How evidence locates the effect
Randomized assignment is the foundation for the causal comparisons in both studies. When eligible people are assigned by a rule that is unrelated to their pre-existing prospects, later average differences between treatment and control groups can be attributed to assignment to the intervention, within the population and setting studied. The Swedish experiment used birthday-based assignment across supportive, controlling, and control conditions. The Dutch experiment also randomized job seekers among information, information-plus-video, and control conditions.
That design answers a specific question: what was the average effect of being assigned to the treatment package rather than to the study’s control condition? It does not answer every nearby question. It does not prove that the people who opened the email were the ones responsible for the effect. It does not isolate the effect of a single phrase from the sender, timing, links, and other components of the package. It does not prove that the same intervention would have the same effect in another country, under another benefits regime, or among people with different labor-market choices.
The Swedish control group received no study emails, but they did receive ordinary communications from the Public Employment Service. The treatment emails also contained practical job-search advice and links, even though the supportive and controlling versions were designed to be as similar as possible apart from their motivational framing. The causal estimate is therefore best understood as the effect of assignment to a complete study-email intervention relative to no study email, not as a laboratory estimate of wording in isolation. This distinction matters especially when interpreting the proposed psychological mechanisms.
Administrative data are valuable because they cover large populations and record consequential outcomes over time. In Sweden, register data allowed the researchers to follow reporting, applications, job finding, and unemployment duration for a very large sample. In the Netherlands, administrative records captured employment, hours, earnings, and benefit receipt for the full experimental population. These data reduce the problem of relying only on what a respondent remembers or chooses to report. They also have their own blind spots. A reporting system may record the activity without revealing its quality; an employment record may show earnings without showing the occupation or the reason a person changed search direction.
Surveys can fill in those missing meanings. They can ask about autonomous and controlled motivation, beliefs about job prospects, willingness to explore alternatives, search activities, and the occupation of a new job. But survey evidence in both experiments is less precise than the main administrative evidence because far fewer people respond. The Swedish motivation measures did not show statistically significant changes, even though the messages changed reporting and some labor-market outcomes. That pattern may reflect limited power or measurement difficulty; it should not be turned into proof that the messages had no psychological effect. At the same time, the absence of a measured motivation change means that the psychological account remains less directly established than the treatment effects themselves.
The Dutch occupation results show a different use of survey evidence. Administrative data did not contain the occupation of a new job, so outflow surveys were used to compare the treatment and control groups’ destinations. The survey response rate was somewhat higher among treated people, creating a potential selection concern. The researchers addressed this with checks, including an extreme-case calculation, but the occupation findings still deserve more qualification than the employment effects measured in administrative data. The evidence supports broader occupational movement, not a claim that every treated person entered one of the listed recommendations.
The Dutch video component also illustrates why treatment take-up must be interpreted carefully. The first information message was opened by 64% of recipients; across the initial and reminder messages, 78.8% opened at least one, and 30.7% of those who opened clicked at least one occupational link. The motivational video was rarely accessed: only 0.5% clicked its link in the first message, and a later video-only email raised the click rate to 7.5% without making overall exposure high. Because the two treatment arms received the same occupational information and video exposure was low, the authors combine them for analysis. The employment effect should therefore be read primarily as evidence about the shared information intervention, not as evidence that a motivational video produced the gain.
Finally, individual-level randomization does not make the labor market a collection of isolated units. If one job seeker shifts from an overcrowded occupation into a tighter one, that can change the chances of other job seekers who remain behind. It can also create more competition in the destination market. The Dutch paper addresses this with a calibrated search-and-matching model. That model is useful for asking how the observed behavioral change might affect treated people, controls, and people outside the experimental group. But it is not the same as directly observing an equilibrium response under full rollout. Its conclusions depend on assumptions about matching, switching costs, and market groups.
The result is a hierarchy of questions rather than a single scorecard. Exposure tells us whether the intervention had a chance to act. Compliance tells us whether it moved a behavior the institution could observe. Search and occupational outcomes tell us whether the person changed the route toward work. Employment, hours, earnings, and job quality tell us whether that route produced a consequential result. Spillover analysis asks whether an individual gain survives contact with the wider market. A digital intervention deserves confidence only to the extent that its evidence reaches the stage that matters for the decision at hand.
That is why the Swedish reporting response is important but insufficient. It shows that the emails were not inert. The controlling messages did not, however, increase the primary measures of applications or job finding. The first-stage result and the downstream result are not contradictory once they are placed at different points in the funnel. The next question is what kind of motivational relationship produced that pattern—and whether the problem was motivation at all.