Overview
Patient-facing health AI can sound medically fluent before it knows enough to support a safe next decision. This series traces the safety problem from missing context and triage error through unstable outputs and user reliance, then turns the evidence into a use-case-specific escalation boundary. It argues that accuracy alone is insufficient: responsible systems should inform, clarify, or route to human care according to the task, urgency, uncertainty, and evidence available. The studies support conditional assistance, not autonomous triage or a universal safety threshold.
Learning outcomes
- A fluent medical answer is not safe by default when missing context could change the next action.
- Triage evaluation must separate emergency, non-emergency, and self-care decisions and show both undertriage and overtriage.
- Repeated outputs and pooling rules can change apparent safety; best-case accuracy does not prove safe deployment.
- Trust should support appropriate reliance, while current evidence supports conditional, use-case-specific assistance rather than autonomous triage.
Keywords
Preview every chapter · Unlock full text, audio, and video from chapter 4
PlansFAQ
Why is medical fluency not enough for patient-facing AI?
Medical fluency is not enough because patient-facing AI may lack decision-changing context while presenting conditional advice as safe.
What should a patient-facing AI escalation boundary do?
It should identify missing context, screen for urgent danger, limit claims, and provide a clear route to human care; it should not merely add a disclaimer or refer everyone.
Can ChatGPT safely decide whether someone needs care?
The cited benchmark does not support relying on tested ChatGPT versions alone: best performance was 74% on 450 repeated trials, self-care was weak, and only two emergency cases were included.
How should trust be evaluated in AI-generated health advice?
Measure trust and distrust toward the specific advice instance and compare them with reliance and care-seeking or outcome measures; the scales do not prove causation or improved health outcomes.