From Evidence to a Safety Boundary

For patient-facing health AI, a safety boundary is not a disclaimer or a general promise to be careful. It is the set of conditions under which a system may continue with information, must ask for more context, or should route the person toward human care. This chapter turns the evidence about unsafe advice, triage errors, output variability, and reliance into a use-case-specific way to evaluate that boundary. The aim is not to prescribe a universal clinical workflow, but to show what a defensible system must make visible before its answer is allowed to stand.

The boundary should change the interaction

A system that merely adds a warning to an answer has not necessarily become safer. The boundary must alter what the system does when the information is insufficient or the consequence of error is high. It should be possible to tell, from the interaction, which task the system is performing, what it knows, what remains uncertain, and what happens next if it cannot support a low-urgency recommendation.

The first condition is task clarity. A product that explains health information is not automatically a triage tool. A product that offers care-seeking advice should state whether it is distinguishing emergency care, non-emergency care, and self-care, because those are different decisions with different error costs. The benchmark of ChatGPT models demonstrates why this matters: a model can be useful for some general questions while remaining unreliable when asked whether a person can safely stay home. Without a named task, a favorable answer-quality result can be misread as evidence for a stronger clinical role.

The second condition is context adequacy. The system should notice when a missing fact could change the recommendation, rather than silently filling in the gap. The relevant fact may concern age, pregnancy, medications, chronic disease, the identity of an ingested object, duration, severity, or an associated warning sign. The exact questions will depend on the use case; the supplied studies do not define a universal list. The general requirement is stronger and more useful than a fixed checklist: the interaction must expose the condition on which the advice depends.

This is a trigger for clarification, not a demand for an exhaustive medical interview. If the missing detail is immaterial to a narrow educational explanation, the system can answer within that limit. If the recommendation would change once the detail is known, the system should ask for it, qualify the advice, or route the user rather than present one branch as settled. That is the difference between acknowledging uncertainty and managing it.

The third condition is an urgency screen before low-urgency guidance. The care-seeking benchmark found that models were generally better at identifying cases requiring medical care than at recognizing when self-care was appropriate. That pattern supports a cautious ordering: a system should not lead with reassuring self-care information while an urgent possibility remains unresolved. At the same time, the benchmark also shows why “send everyone to care” is not an adequate boundary. High sensitivity combined with low specificity can make a tool conservative but unhelpful.

The boundary therefore has to protect against two different pathways. In the first, the model gives a lower urgency than the situation warrants, and the user delays needed care. In the second, the model assigns a higher urgency than necessary, increasing anxiety, avoidable visits, or pressure on services. The consequences are not symmetrical, but both are relevant to a system claiming to support decisions.