When a Fluent Draft Is Clinically Unsafe
This chapter examines the kinds of errors that can remain in an AI-drafted patient-portal message even after a practicing clinician reviews it. The focus is not on whether a reply sounds professional; it is on whether the response recognizes the patient’s actual problem, uses current information, and makes the right next action visible. By the end, readers should be able to distinguish a wrong statement from a missing safeguard and interpret evidence about missed errors without treating a small simulation as a universal rate.
Fluency is especially misleading in asynchronous care because the patient cannot immediately ask what the response meant. A draft can be grammatically clean, organized, and reassuring while quietly narrowing the situation to a benign explanation. The danger is not that every generated message is wrong. It is that a message can be mostly right in a way that makes the part that is absent difficult to notice.
Four errors, four different kinds of failure
A primary-care simulation made this problem concrete. Twenty practicing primary care physicians used a simulated electronic health-record portal to respond to 18 patient messages. Four of the AI-generated drafts contained errors. The study was designed to test whether clinicians would identify and correct those errors before sending, not to estimate how often all real-world AI drafts are unsafe. That distinction matters, but the cases still show why “accuracy” is too narrow a safety criterion.
The first error was an objective typo: the draft carried forward a patient’s misspelling and treated it as the name of a medication. The defect is small in clinical content but important as a category. A system can introduce an incorrect identifier even when it is trying to be helpful, and the error may be missed precisely because the sentence looks ordinary. Not every typo causes harm, but a wrong medication name can create confusion about what the patient is taking or asking to adjust. Surface correctness is therefore not binary. Review must ask whether the named object is the right one, not just whether the sentence is fluent.
The second error was outdated advice about the timing of COVID-19 vaccination after a recent infection. The draft gave a plausible general recommendation, while current guidance allowed a different choice about delaying a booster. This is a temporal failure: the response can be coherent and medically familiar yet no longer align with the guidance that should govern the answer. It shows why a clinician cannot assess a draft only against memory or general plausibility. Some claims are sensitive to date, policy, or changing standards, and their correctness depends on the reference point used for review.
The other two errors were potentially harmful omissions. In one simulated message, a patient described ankle swelling and red spots. The draft mentioned the possibility of a blood clot and suggested making an appointment, but did not communicate the urgency of seeking evaluation in a way the study’s physician reviewers considered sufficient. In the other, a parent described a child who was vomiting, drinking a lot of water, and having urinary problems. The draft suggested a stomach virus or infection and routine consultation if vomiting continued, but did not address possible diabetic ketoacidosis, a condition requiring urgent intervention.
These examples should not be read as instructions for diagnosing an individual. They are study cases showing how a response can fail at the level of action. The blood-clot draft did not need to be wholly nonsensical to be unsafe; it needed to understate what the patient should do. The draft about the child did not need to invent every detail; it needed to omit a clinically important possibility and the urgency attached to it. In both cases, the most consequential defect was not an incorrect sentence but the absence of a sentence that would change the patient’s next step.
The distinction generalizes to the broader error vocabulary constructed in a retrieval-guardrail preprint. Its taxonomy includes incomplete responses, missed escalation of care, missed safety-net instructions, omitted differential diagnoses, workflow violations, and problems in communication quality and readability. These categories are not interchangeable. A long or awkward message may frustrate a patient; a missed escalation may delay care. A checker that counts every deviation from a preferred style as equivalent to a missing safety instruction will create noise and obscure the failures that deserve immediate attention.
The cases also show why “the model hallucinated” is an incomplete diagnosis of the problem. Hallucination usually names a false or fabricated detail. The simulation included a typo and outdated advice, but its most important cases involved triage, omission, and urgency. A draft can be factually anchored to the words it received and still fail to reason about what the words imply. It can answer the explicit question while ignoring the latent question: Is this dangerous? Do I need help now? What should I watch for? What would make the situation worse?
A clinically adequate review therefore compares the draft with the patient’s need, not just with the source text. It asks whether every material part of the question was answered, whether the response uses current guidance where timing matters, whether an alternative explanation changes the risk, and whether the patient is given an appropriate next action and safety net. Those checks are more demanding than grammar review because they require the reviewer to reopen the situation that the draft appears to have already interpreted.
The two paths in the diagram are not a measured causal model. They are a way to connect the study’s observed misses with the review conditions that could allow an omission to survive. The central distinction is between accepting the draft’s framing and reopening the patient’s problem. A reviewer who only asks “Does this answer sound reasonable?” may follow the upper path. A reviewer who asks “What would make this answer unsafe or incomplete?” creates a chance to follow the lower one.
Why a human can still miss what is in front of them
The simulation provides unusually concrete evidence about the gap between formal review and effective review. Each of the four erroneous drafts was insufficiently addressed by 13 to 15 of the 20 physicians. Seven to nine physicians submitted each erroneous draft entirely unedited. On average, participants missed 2.67 of the four erroneous drafts. Only one participant sufficiently addressed all four, and all but one sent at least one simulated response that still contained an error.
The study defined a “miss” conservatively: both of two practicing primary-care physicians who independently evaluated the edits had to judge that the response was insufficiently edited to address the error. That makes the result more informative than a disagreement over style, but it does not turn it into a clinical prevalence estimate. The physicians worked remotely in a simulated environment, did not receive full patient records, and were not tested on the same messages without an AI draft. The set of 18 messages was selected from real portal messages but did not represent the full variety of outpatient communication. The findings show a review vulnerability under those conditions; they do not tell us the probability that any particular clinician or deployed system will fail in the same way.
Why might the vulnerability exist? The study authors propose four plausible human-factors mechanisms. The first is functional fixedness: once a familiar answer appears, the reviewer may remain fixed on that type of solution rather than reconsidering the problem from another angle. A draft that interprets a child’s symptoms as an ordinary stomach illness can make the alternative possibility harder to retrieve, even when the original message contains clues that deserve a different response.
The second is confirmation bias. A clinician may compare the draft with an expectation already formed from the message and notice the details that fit, while failing to search actively for disconfirming information. This does not require carelessness or bad faith. A response that resembles the clinician’s anticipated wording can feel like confirmation that the initial assessment was sound. The danger is greatest when the draft is not absurd; its plausibility gives the reviewer less reason to reopen the assessment.
The third is automation complacency, a decline in vigilance when an automated system usually appears to work. If most drafts contain no obvious problem, the reviewer may develop a monitoring pattern suited to routine acceptance rather than active error search. The fourth is automation bias, the tendency to rely too heavily on an automated recommendation, which the authors note can be intensified by workload. These mechanisms can reinforce one another: a busy clinician receives a generally useful draft, recognizes a familiar pattern, and spends attention on whether the wording is ready rather than on whether the clinical frame is complete.
The physicians’ own perceptions are consistent with that possibility, although they do not prove it. Nineteen of the 20 participants said the drafts were helpful. Sixteen agreed that the drafts reduced the cognitive workload required to respond. Eighteen agreed that they trusted the tool’s performance; 14 agreed that the drafts were accurate; and 15 agreed that they were safe to use. The positive survey results do not establish that confidence caused the misses, and the study did not manipulate confidence or workload. They do show that favorable impressions are compatible with missed errors. A review process has to be designed for that coexistence rather than assuming that a clinician will become more skeptical simply because the system is known to be fallible.
This is also why blaming individual reviewers is a weak explanation. The task itself may encourage the wrong cognitive posture. A blank-page response requires the clinician to interpret the question and construct a plan. A proposed response offers an interpretation first and asks the clinician to evaluate it. If the interface gives equal visual weight to a harmless stylistic imperfection and a missing escalation instruction, it does not help the reviewer allocate attention according to risk. If the workflow rewards rapid closure, the easiest form of review may become a quick scan for obvious errors.
A meaningful review must instead test the draft at several levels. At the factual level, the reviewer checks names, dates, recommendations, and patient-specific details. At the clinical-reasoning level, they ask whether the response considered a materially different possibility and whether it recognizes the level of urgency suggested by the message. At the communication level, they ask whether the patient can tell what to do, what to watch for, and when to seek additional help. At the workflow level, they ask whether the portal is the right channel and whether the response should be escalated to a call, visit, or another responsible team member. These levels are related, but success at one does not imply success at the others.
That standard should remain proportional rather than mechanical. Not every refill requires an elaborate differential assessment, and an exhaustive check of every low-stakes message could consume the time the technology was meant to recover. But a routine-looking draft should not lower the review threshold when the patient’s words signal uncertainty, deterioration, or a need for urgent action. The evidence does not provide a tested formula for assigning review intensity. It does support the principle that review should be sensitive to stakes and to the kinds of errors most likely to change patient action.
The broader lesson is that clinical safety is partly about absence. A reviewer can verify that the sentences present are accurate and still fail to notice that an instruction, alternative explanation, or escalation path is missing. Automated tools may help surface such gaps, but their labels will also have limitations. Human-factors design, local workflow knowledge, and an explicit release responsibility remain necessary because the question is not simply whether the draft contains an error. It is whether the draft has done enough for this patient, in this situation, through this channel.
A clinician send action still matters: it is the point at which the organization permits the message to leave the care team. But it is the last gate, not the whole inspection. The evidence now supports a more demanding question for a safe workflow: does it make the important absence visible before the message reaches the patient? That question leads to a conditional test of whether efficiency is actually improving communication rather than merely accelerating release.