Why Patient-Facing AI Needs an Escalation Boundary
Patient-facing large language models—public chatbots that answer people’s health questions—are the subject here. The important question is not simply whether they can produce medically fluent text, but whether they can help a person make a safe next decision when the question leaves out facts that matter. This series develops a way to judge that boundary: when should a system answer, when should it seek more context, and when should it direct someone toward human or emergency care?
That distinction matters because health questions are not usually examination questions. A patient may type, “How do I get rid of my headache?” or “What should I do if my child has a fever?” The words are short, the desired action is unclear, and the details that change the risk may be absent. A system can respond smoothly while still failing at the more important task: recognizing that it does not yet know enough to recommend a safe course.
A patient question is not a medical exam question
Large language models have often been assessed on advanced medical questions, including examination-style prompts with technical vocabulary, a carefully selected history, and one intended answer. Those tests can tell us something about whether a model has encoded medical knowledge or can reason over a supplied case. They do not reproduce the conditions of a patient asking for advice.
In a patient-facing interaction, the model may not know the person’s age, pregnancy status, medications, chronic conditions, symptom duration, severity, or associated warning signs. It may not see how ill the person looks, hear changes in their voice, consult a medical record, or draw on a relationship built through earlier visits. The patient may also describe the situation inaccurately or use a self-diagnosis as if it were established. Safe clinical reasoning begins partly by finding out which of these facts are relevant. A chatbot that treats the initial wording as a complete case is solving a different, easier problem.
Care-seeking advice makes the difference especially clear. The output is not merely an explanation of a disease. It is a recommendation about urgency: emergency care, a non-emergency visit, or self-care. That recommendation sits at a boundary between states with different consequences. A medically knowledgeable paragraph can still be unsafe if it leaves the user in the wrong place on that boundary.
The distinction also explains why impressive performance on a medical benchmark should not be translated directly into the claim that a system can replace a clinician. Knowing an answer when the relevant information has already been supplied is not the same capability as deciding which information is missing, asking for it, recognizing danger, and communicating uncertainty to a person who may act immediately.
Fluent answers can fail in clinically different ways
A physician-led red-team study makes the risk concrete. It collected 222 advice-seeking primary-care questions across internal medicine, pediatrics, and women’s health, then evaluated 888 responses from four publicly available chatbots. The questions were presented as a patient might present them, without extra instructions. Sixteen physicians judged the responses while blinded to chatbot identity.
The results were not a story of one bad model and three perfect ones. Problematic responses ranged from 21.6% for the best-performing chatbot in that comparison to 43.2% for the worst; the other two were between those figures. “Problematic” was broader than “dangerous”: it included unsafe advice, false or otherwise problematic content, missing important information, missing history-taking, and poor writing. Unsafe responses alone still ranged from about 5% to about 13% across the tested systems. The highest-rated model was therefore not a safe-by-default model. It was a better performer within a study in which a meaningful minority of answers still fell below the expected quality of a physician’s written response.
The categories matter because they point to different remedies. A confusing paragraph is not the same failure as a dangerous instruction. An omitted warning is not the same as a fabricated fact. Missing history-taking is especially revealing: the answer may contain individually sensible advice, yet the model never asks the question that would determine whether that advice applies.
Consider a child who has swallowed what a caregiver calls a penny. A response that accepts the label may offer routine observation. A safer interaction would first establish whether the object really was a penny, whether more than one object was swallowed, and whether it could have been a button battery or magnet. The key failure is not only a wrong sentence. It is the untested assumption that the initial description contains all the information needed for action.
Other examples show why a reassuring tone cannot substitute for a boundary. A recommendation about pregnancy back pain can be unsafe if it accepts “sciatica” without considering another cause. Advice about increasing potassium can be harmful for some people with kidney disease or particular medications. Guidance for an infant cannot assume that advice suitable for an older child is safe. In each case, the model’s fluency can make the missing condition less visible to the person reading it.
The red-team study’s examples illustrate potential worst-case outcomes, not predictions that those outcomes will usually occur. That qualification is important. The evidence does not show that every problematic response causes injury. It does show that a patient can receive actionable advice whose safety depends on context the system did not establish. In a high-stakes setting, that possibility is itself a reason to test the interaction rather than judge the prose alone.
The useful boundary is a decision interface, not a refusal wall
An escalation boundary is a limit on what the system should do without further evidence. It does not mean refusing every health question. It means that the interaction changes mode when the answer depends on information that is missing, when warning signs cannot be safely excluded, or when the system cannot support a low-urgency recommendation with enough confidence.
In practical terms, the boundary has several linked functions. First, the system must notice when the question is underspecified and ask for, or clearly surface, the missing context that could change the recommendation. Second, it must screen for urgent danger before offering low-urgency guidance. Third, it must distinguish an explanation from a care-seeking recommendation: “here are possible reasons” is not the same as “it is safe to stay home.” Finally, it must leave the user with a legible route to human care rather than allowing a confident paragraph to close the decision prematurely.
The evidence from a separate benchmark of 22 ChatGPT model versions reinforces why these functions should be evaluated separately. Across 45 validated case vignettes, each model was tested ten times on whether the case called for emergency care, non-emergency care, or self-care. The best overall model solved 74% of the repeated assessments, and newer releases did not show a general improvement simply by being newer. Models were generally better at identifying cases requiring care than at recognizing when self-care was appropriate. That conservatism may reduce some missed emergencies, but if a system recommends medical care for nearly every minor case, it is not providing useful self-triage.
A boundary therefore has two sides. It must make dangerous reassurance difficult, but it must also avoid turning safety into universal referral. The right target is calibrated action: urgent cases should be routed promptly; cases needing ordinary clinical attention should not be disguised as self-care; and low-urgency advice should be offered only when the context and evidence justify it. The supplied studies do not establish a universal threshold or a finished product design. They do establish why those questions cannot be skipped.
This is a mental model, not a diagnostic protocol. It shows why “answer” is only one possible outcome of a patient-facing interaction. Clarification, escalation, and a carefully bounded response are capabilities too. They protect the user from a model’s tendency to make an incomplete prompt look complete.
What this changes about usefulness
The promise of patient-facing AI remains real. A system that explains unfamiliar terms, helps a person formulate questions, or recognizes a possible emergency could extend access to health information. But benefit and harm travel through the same channel: a person receives language and decides what to do with it. The safety question is therefore not only whether the output is accurate in the abstract. It is whether the recommendation is appropriate for this context, stable enough to rely on, and presented in a way that supports a sound decision.
That last step brings the user into the safety model. Research on trust in AI-generated health advice distinguishes trust in a specific piece of advice from a general belief that the technology is trustworthy. A person may accept a response because it feels reassuring, or reject a useful one because it feels uncertain. Measuring that relationship matters because a technically correct answer that is over-trusted can create risk, while a cautious answer that is ignored may not provide its intended benefit. The goal is not to make people trust the chatbot more. It is to make reliance proportionate to the quality and urgency of the situation.
This is why “better answers” is too narrow a standard. The system also needs to know when its answer is conditional, when its uncertainty is consequential, and when another form of help should take over. The central test is not whether a chatbot can always say something plausible. It is whether it can recognize when plausibility is not enough. The next question, then, is how missing history and silent assumptions turn an ordinary patient prompt into a dangerous recommendation.