Triage Is a Different Task from Knowing Medicine
For patient-facing ChatGPT models, care-seeking advice is a distinct safety problem: the system is not only explaining what a symptom might mean, but helping a person decide how urgently to seek care. This chapter examines what a large comparison of model versions actually measured, what its accuracy and error patterns show, and where the evidence stops. The judgment to carry forward is that a cautious recommendation is not automatically a useful one; a safe boundary has to manage both missed danger and unnecessary escalation.
Care-seeking advice is a three-way decision
A model can answer a health question in many forms. It can define a term, summarize possible causes, suggest questions for a clinician, or generate a differential diagnosis. Care-seeking advice asks a narrower and more consequential question: given this situation, should the person seek emergency care, arrange non-emergency care, or manage the issue with self-care and watchful waiting?
Those choices are not interchangeable. “Emergency care” means immediate routing under the benchmark’s definitions. “Non-emergency care” means trying to see a doctor without an immediate emergency-room need. “Self-care” means allowing the issue to improve on its own and reviewing the situation again after a few days. The model may use medical knowledge to reach a recommendation, but the evaluation is about the recommendation’s urgency, not about whether the model can write a convincing explanation of a disease.
This distinction corrects a common shortcut in discussions of medical AI. A model can perform well on a question whose relevant history, examination findings, and answer choices are already supplied. That result does not tell us whether it can recognize that a person needs urgent care, distinguish a routine problem from an emergency, or justify telling someone that professional care is not immediately needed. Triage is a decision at the edge of action. The cost of an error depends on which direction the model moves the person.
The benchmark by Kopka, He, and Feufel was designed to measure that decision directly. It included 45 validated case vignettes based on real cases in which people sought guidance about whether and where to seek care. The set contained two emergency cases, 30 non-emergency cases, and 13 self-care cases. Two licensed physicians independently assigned the appropriate urgency level and discussed disagreements until reaching a consensus. That physician-derived answer served as the study’s gold standard.
The researchers tested 22 ChatGPT model versions. Each model received every vignette ten times using a standardized one-shot prompt, yielding 9,900 assessments in total. The prompt defined the three urgency categories and asked for a triage classification. The repeated trials served two purposes: they made model comparisons more systematic, and they exposed whether the same model gave the same recommendation when the input did not change.
That design has an important advantage. It holds the wording of the case and the classification task steady while comparing models and releases. It also avoids claiming that one carefully designed prompt represents every way a person might ask for help. In everyday use, people may describe symptoms in their own language, omit different facts, ask follow-up questions, or respond to what the model says. The study’s result is therefore evidence about a controlled care-seeking task, not a complete simulation of a patient conversation.
The gold standard has a similar scope. Two physicians reaching consensus is a defensible reference for a vignette study, but it is still an approximation of clinical ground truth. A real assessment may require examination, laboratory tests, imaging, and follow-up. The researchers explicitly note this limitation. The vignettes were designed to be representative of online care-seeking situations, but they cannot contain every fact that would be available in a clinical encounter.
The small number of emergency cases is especially important. There were only two. A model’s emergency performance can appear very high when it identifies those cases, but two examples are not enough to estimate emergency recognition with the same confidence as performance over 30 non-emergency or 13 self-care cases. The evidence supports cautious interest in emergency detection, not a broad claim that a model reliably recognizes every medical emergency.
The diagram separates the model’s recommendation from the gold-standard comparison. “Overtriage” means the model assigns a higher urgency than the reference; “undertriage” means it assigns a lower one. These are not equivalent mistakes, and the study’s usefulness lies partly in showing their direction rather than reporting one undifferentiated error count.
There is also a time boundary around the comparison. The authors included OpenAI models identified through the company’s API documentation and sorted them by release date, with additional collection rounds for newer releases through January 2026. The results therefore describe the model snapshots and settings tested in those rounds. They are not a permanent ranking of every future version, nor do they mean that a newer model’s behavior is identical across consumer interfaces.
Accuracy, caution, and the self-care gap
The headline result is easy to misread. The best overall performance came from o1-mini, which classified 74% of its 450 model–case trials correctly. The lowest came from gpt-4.1-nano at 44%. The authors found no general improvement in average accuracy as releases became newer. The average across the model family remained around 70%, with meaningful differences by model and urgency level.
A 74% result does not mean that 74% of patients would be safely managed. It means that, under this prompt and this set of 45 vignettes repeated ten times, the model’s three-way label matched the physician-derived reference in 331 of 450 assessments. It combines emergency, non-emergency, and self-care cases, and it says nothing by itself about the consequences of the 119 mismatches. A score becomes decision-relevant only when we ask which cases were missed, which were unnecessarily escalated, and whether a typical user would receive the same answer consistently.
The model-generation pattern complicates the simple story that progress is linear. Newer reasoning models were more willing to recommend self-care in some cases, and self-care accuracy improved over time, particularly with chain-of-thought models and o4-mini. But that improvement did not make self-care advice reliable enough for standalone use. The study’s conclusion is not that reasoning models have no value. It is that their gains are task-specific and incomplete. “Newer” is not a safety category.
The direction of error explains why. All tested models tended to assign a higher urgency than the gold standard. In the binary version of the task—whether any medical care was required or self-care was appropriate—sensitivity ranged from 85% to 100%, while specificity ranged from 0% to 49% across models. Sensitivity here reflects how often a model identified cases for which medical care was required. Specificity reflects how often it correctly recognized cases where self-care was the appropriate category.
High sensitivity can be reassuring because missing a genuine emergency may be more harmful than recommending an unnecessary visit. But high sensitivity with very low specificity can mean that the system’s effective rule is “seek medical care” for almost everyone. That rule may avoid some undertriage while failing the user who needs help deciding whether a minor problem can be watched at home. It can also increase anxiety and avoidable use of care, with consequences for people who depend on already-constrained services. The study discusses these system-level implications, but it did not measure actual utilization or costs in the experiment.
The self-care cases exposed the weakness most clearly. Seventy percent of all errors—2,573 of 3,696—occurred in self-care cases. No self-care case was classified correctly by every model in every trial. Ten of the 13 self-care cases were solved correctly at least once by some model, but three were never solved by any model in the tested set. This pattern says more than “the models made mistakes.” It suggests that the low-urgency side of the boundary is where the systems have the least dependable performance, even when they have enough information to choose among fixed categories.
Why does that matter so much? For many users, the practical value of a care-seeking tool is not merely confirming that a dramatic emergency deserves attention. People are often uncertain whether a symptom warrants a medical visit or whether it is reasonable to wait. If a system almost always moves them toward care, it may be conservative, but it is not resolving the decision they brought to it. A tool that cannot safely support low-urgency decisions has limited value as a substitute for neither clinical assessment nor ordinary judgment.
The trade-off should not be inverted. The weakness of overtriage does not make undertriage acceptable. A model that tells someone with an emergency to stay home can cause a more immediate and serious harm than a model that recommends an unnecessary appointment. The point is that “more cautious” and “more accurate” are not synonyms. The desired system must make urgent routing difficult to miss while reserving self-care guidance for cases in which the relevant uncertainty has been screened and the recommendation is stable enough to defend.
This is also why the three categories should not be collapsed into a binary score. An emergency recommendation that is one level too high has a different meaning from self-care advice given for an emergency. A non-emergency recommendation for a self-care case can impose cost and anxiety without the same direct clinical danger as undertriage, but repeated over time it can still degrade the usefulness of the tool and burden the care system. A responsible evaluation reports the direction and urgency level of each error rather than celebrating a single aggregate accuracy number.
The benchmark supports a bounded conclusion about usefulness. Current ChatGPT models may help a user think about whether emergency care is necessary, but the evidence does not justify relying on them alone. The emergency estimate is limited by the two-case sample. The overall accuracy is insufficient for standalone guidance. Self-care performance remains weak, and the models differ in ways that make “the ChatGPT answer” an imprecise phrase. These are not reasons to discard all assistance. They are reasons to put assistance inside an explicit escalation boundary.
That boundary should begin with the task being named honestly. If the product answers general health questions, it should not imply that it has established urgency. If it offers triage, it should be evaluated on emergency, non-emergency, and self-care cases separately. If the system’s evidence for a low-urgency recommendation is inadequate, clarification or a route to care is safer than confident reassurance. And if the model changes its recommendation for the same case, the user-facing system cannot hide that instability behind its average score.
The study’s limitations define what new evidence would change the judgment. Larger and more representative vignette sets, especially with more emergency cases, would make urgency estimates more precise. Prompts based on real patient behavior would show how much wording and follow-up questions matter. Prospective observational or clinical studies would be needed to determine how recommendations influence actual care-seeking decisions. Testing after model updates would show whether a safety claim persists. Until then, this benchmark supports use-case-specific caution, not autonomous triage.
Triage evidence thus sharpens the escalation problem. A model can recognize some urgent cases, yet over-refer so often that it offers little low-urgency value. It can improve on one category without improving overall reliability. And its aggregate score can conceal what an individual user experiences in a single response. The next question is whether that response is stable at all when the same case and prompt are repeated, and what happens when a developer tries to combine the competing answers.