Why the Right Prompt Can Still Fail

A JITAI’s hardest component is not the rule that chooses an intervention; it is the changing world the rule is trying to observe. This chapter examines the interfaces between real-life context, measurement, decision, and human response so that a reader can tell whether a disappointing result reflects a bad signal, a poor option, excessive burden, low receptivity, or an outcome that was never actually changed. That distinction matters because a JITAI can follow its protocol perfectly and still fail to provide just-right support.

A tailoring variable is a window, not the whole system

A JITAI is built for a setting in which behavior does not unfold like a laboratory equation. In a simple, linear picture, the same starting condition and the same intervention should produce roughly the same result. If an anti-smoking message reduces tobacco use in one carefully controlled setting, the picture suggests that repeating the message with a similar group should produce a similar effect. The everyday environment is less obedient. People move through changing places, moods, demands, relationships, and routines. The intervention itself changes what people notice and do, which then changes the conditions facing the next decision.

This is the intuition behind a complexity-science approach to JITAIs. One form of complexity concerns nonlinear trajectories: small differences in initial conditions can lead behavior down different paths, and the path may not be predictable from a single input. Another concerns aggregate complexity: many interacting factors combine to produce an outcome that none of them determines alone. A person’s decision to smoke, walk, eat, or complete a task may reflect internal states, environmental cues, social pressures, previous feedback, and immediate practical constraints. These influences can reinforce or counteract one another over time.

The point is not that every JITAI must model every influence. That would be impossible and would make real-time intervention unusable. The practical move is narrower: identify signals that mark moments when a particular intervention might matter. For someone trying to stop smoking, a location associated with past use may indicate elevated risk. A sensor pattern that resembles a smoking gesture may signal that a lapse is beginning. These signals can help a rule decide when to offer a warning or coping strategy, but they are not the behavior’s complete cause. A location does not force a lapse, and a gesture classifier does not reveal everything about craving, social pressure, or the person’s ability to respond.

That distinction between a signal and a cause is easy to lose when a system produces data continuously. A phone may know where a person is, how much they move, or when they open an app. It may infer a state from those observations. But the inferred state is still a representation selected for a purpose. It is useful if it improves the decision it feeds; it is not valuable merely because it is granular.

A tailoring variable is therefore best understood as a window onto a changing system. It lets the JITAI focus on a potentially important moment without claiming to see the whole situation. The narrower the window, the more important it becomes to ask what it leaves out. A location can identify risk exposure but not receptivity. A self-report can describe a feeling but may arrive after the most useful decision point. A movement signal can be timely but ambiguous. The design has to connect the signal to an intervention option while preserving the uncertainty between observation and interpretation.

The intervention also participates in the system it measures. Suppose a prompt changes a person’s behavior at one decision point. The next state is no longer the state that would have occurred without the prompt. A later absence of risk behavior may reflect successful support, a change in context, a delayed consequence, or an error in measurement. A repeated rule is not observing a passive subject; it is operating inside an evolving feedback loop. This is one reason a result from one context may not reproduce automatically in another, even when the same message and threshold are used.

flowchart LR A["Changing internal and contextual state"] --> B["Passive or active assessment"] B --> C["Tailoring variable as a partial proxy"] C --> D["JITAI decision rule"] D --> E["Intervention option"] D --> F["No delivery"] E --> G{"Can the person use it now?"} G -->|Yes| H["Physical, cognitive, or emotional engagement"] G -->|No| I["Low receptivity, burden, or possible harm"] H --> J["Adherence or retention"] I --> K["Missing or ineffective response"] F --> J J --> L["Proximal and distal outcome"] L -.-> A B -.-> M["Noise, missing data, or classifier error"] M -.-> D

The system has several interfaces, and each interface can alter the meaning of the one after it. Context becomes an assessment; an assessment becomes a tailoring variable; the variable is interpreted by a rule; the rule selects an option; the person decides, consciously or practically, whether and how to engage; and the resulting response is recorded as evidence about an outcome. A failure at the final outcome does not tell us automatically which interface failed. Nor does successful delivery prove that the earlier interfaces were valid.

This is why complexity should not be used as a grand explanation for every disappointing result. It tells us to expect interacting conditions and to avoid assuming that a fixed recipe will behave identically everywhere. It does not show that an adaptive intervention will outperform a fixed one, and it does not excuse an unspecified mechanism. A complexity-aware JITAI still needs a concrete tailoring variable, a stated decision rule, a plausible option, and a meaningful outcome. Complexity makes those requirements more important, not less.

Where the chain breaks

The first major choice concerns how the system observes the state it will use. A JITAI may rely on passive assessment, in which devices or sensors collect information without asking the person to answer a question at each moment. Location data, movement, heart rate, or device-use patterns can be collected in the background and used to infer a situation. Passive data can support rapid decisions with less repeated interruption, but it often measures a proxy. A phone can detect that someone is near a place; it may not know whether that person is entering, passing by, or accompanied by someone who changes the situation. A sensor can record motion; it may not know what the motion means.

An active assessment asks the person to report a thought, feeling, symptom, or behavior. This can provide information that a sensor cannot observe directly, particularly subjective states. Someone may report a craving, an emotion, or a perceived obstacle that would otherwise be invisible. But active assessment has a cost. Repeated questions consume time and attention, invite subjective interpretation, and can become tiring when the same person is asked to answer similar items many times a day.

The choice is not a contest in which passive measurement always wins. It is a question of what the rule needs to know and what type of error is tolerable. If the relevant state is a private feeling, active reporting may be the most direct available measure despite its burden. If the state is a physical event that can be sensed with reasonable accuracy, passive assessment may be preferable. Combining the two may provide a richer picture, but it also increases the number of interfaces at which missing data or disagreement can occur. A self-report may say “not ready,” while a device may show the person is in the usual place for a behavior. The rule must specify how such signals are reconciled, or its apparent precision will conceal an unresolved judgment.

The available evidence illustrates the trade-off without settling it universally. In the complexity-science meta-analysis, interventions using passive assessments alone had a median retention rate of 90%, compared with 85% for interventions using both passive and active assessments; the difference was statistically significant in that coded sample. But the result is an association, not proof that passivity caused people to remain engaged. The included evidence was limited, and no intervention in that review used active assessment alone. A lower reporting burden may help retention, while a missing subjective state may weaken the decision. Evidence that one group completed more follow-up measures is not evidence that its JITAI changed behavior more effectively.

The next interface is the difference between the tailoring variable and the state the intervention claims to address. A tailoring variable may be a useful marker of opportunity or vulnerability, but it can be incomplete, delayed, or wrong. If a location is used as a proxy for relapse risk, the intervention may be timely whenever the person approaches that location. Yet the same location may be a workplace, a route home, or a place where the person is with supportive friends. The variable identifies a condition that may matter; it does not settle the meaning of the condition in every instance.

Classifier error creates another failure surface. Sensors collect raw data, and algorithms may translate those data into categories such as stress episodes, smoking gestures, or physical activity. Every translation introduces a possibility of false positives and false negatives. A false positive can generate an unnecessary prompt, adding burden or making the system seem intrusive. A false negative can withhold an option when it might have helped. The relevant question is not whether the classifier is “smart,” but how its errors affect the intervention’s decision and the person’s experience.

Even an accurate signal may feed the wrong option. A state of vulnerability can tell the system that a problem is likely soon, but not whether a warning message, a skill exercise, or feedback is the appropriate response. The option has its own demands and its own mechanism. A prompt that requires reading, reflection, and action is not interchangeable with a brief reminder. If the option does not address the process that is likely to change the proximal outcome, better sensing only delivers a mismatched intervention more precisely.

This is where receptivity becomes a system interface rather than a vague personal characteristic. Receptivity is the set of conditions in which a person is likely to engage effectively with a specific option. A person may be driving, in a meeting, preoccupied with an urgent task, or surrounded by other people. Stress may increase risk while also making it harder to attend to and process a digital message. A prompt that recommends a short walk may be useful in one setting and impossible in another. The question is not whether the person is generally receptive; it is whether they can use this option at this moment.

The distinction has a practical implication: the most important moment is not always the most usable moment. A rule that responds only to vulnerability can send support precisely when cognitive or practical capacity is constrained. That does not mean a JITAI should ignore difficult states. It may need a simpler option, a different modality, or a rule that waits for a safer opportunity. Those are design questions to test, not reasons to label the person unmotivated.

Burden is another interface where local decisions become system-level effects. Each prompt may ask for only a few seconds, but repeated requests can accumulate. Participants may invest physical energy in opening a message, cognitive energy in understanding it, and emotional energy in responding to it. The intervention may also ask them to complete a nondigital task whose completion is difficult to observe. If the system treats every detected risk as a reason to intervene, it can create interruption, fatigue, or resistance to persuasion. Repeated stimulation may reduce response through habituation, even when the content remains relevant.

The reverse error is possible. A rule designed to minimize burden may withhold support so often that it misses the opportunities it was meant to use. Resource efficiency means achieving more benefit with fewer resources, not simply reducing the number of messages. A system that sends fewer prompts but does not change its intended proximal outcome has demonstrated restraint, not necessarily efficiency. The design must make the trade-off visible: what expected benefit is lost when delivery is withheld, and what burden or harm is avoided when it is not delivered?

The final interface concerns what is counted as evidence that the system worked. Engagement is the investment of physical, cognitive, and emotional energy in a stimulus or task. Opening a prompt may demonstrate physical engagement, but it does not prove that the person read or understood it. Completing a short exercise may show participation, but not that the exercise changed the mechanism it targeted. Adherence concerns following the intended intervention or study requirements. Retention commonly refers to remaining in the intervention or completing later assessments. An outcome is the target change, such as a behavior, symptom, or aspect of functioning. These measures can relate to one another, but they are not substitutes.

A person can remain enrolled while ignoring prompts. They can open every message without changing the target behavior. They can change a behavior for reasons unrelated to the intervention. They can stop responding to assessments while continuing to use the intervention—or because the intervention is not helping. If a study measures retention more reliably than the health outcome, the easy measure can quietly become the implied result.

The meta-analytic evidence reinforces this caution. In the complexity review, retention did not moderate the observed objective health effects, and intervention duration did not correlate significantly with retention. The authors also note that incentives may have encouraged people to continue participating, so high retention may not have come from the intervention’s engagement design. In the mental-health evidence base, deviations from intended interventions, dropout, and missing outcome data were important sources of risk. These findings do not show that engagement is irrelevant. They show that engagement must be connected to the outcome and interpreted with attention to why data are missing.

Missingness is not always neutral. People who stop answering may differ from those who remain, particularly if the intervention is burdensome, ineffective, or poorly matched to their circumstances. If a study analyzes only participants who completed the protocol, it may make the intervention look more effective than it would be for everyone offered it. The safest conclusion is not “dropouts did not benefit” or “dropouts would have failed.” It is that the missing outcomes limit what can be known unless the missingness is addressed and reported.

A similar discipline applies to apparent success. Suppose retention is high, the app logs frequent interaction, and the target behavior improves. That combination is encouraging, but it does not identify which part of the JITAI mattered. The rule may have selected the right option, an incentive may have driven behavior, a concurrent treatment may have supplied the mechanism, or participants may have improved for reasons outside the intervention. Conversely, if the target outcome does not change, the failure may lie in the signal, threshold, option, receptivity, measurement, or follow-up—not necessarily in the idea of adaptation.

A system teardown is useful because it prevents the word “adaptive” from compressing all these interfaces into one black box. The context is changing. The assessment samples it. The tailoring variable represents it imperfectly. The rule owns the delivery decision. The option carries a mechanism and a burden. The person’s receptivity determines whether engagement can occur. Retention and adherence describe participation in different ways, while the outcome tells us whether the intended change appeared. Feedback from the outcome then alters the conditions for future decisions.

The practical conclusion is deliberately modest. Complexity does not make JITAIs unknowable, and passive sensing does not make them objective by default. A well-specified intervention can still miss because its proxy is weak, its option is mismatched, or the person cannot use it when the rule fires. Before asking whether a JITAI “works,” we need to ask where the chain is expected to work and where it might break. That fit test is the bridge between a delivery rule and a defensible claim about benefit.

References