When the Same Question Gets Different Answers

The recommendation a patient sees is not necessarily a model’s stable judgment. For patient-facing health AI, the same case and the same prompt can produce a distribution of urgency recommendations, and the way a developer summarizes that distribution can change the apparent safety of the tool. This chapter follows that path—from repeated trials, through model-version differences, to aggregation—so that accuracy, consistency, best-observed performance, and deployment safety are not mistaken for the same thing.

One case has more than one model state

A care-seeking benchmark can be read as though each model has one answer for each case. In practice, the study found something more complicated. The researchers sent each of 45 case vignettes to each model ten times, with the context window cleared before every API call and default decoding settings. The case did not change, but the model’s recommendation sometimes did. A patient who submits a question once sees one draw from that behavior, not the model’s average across all possible trials.

The researchers measured agreement across the ten recommendations with Cohen’s kappa, a statistic that adjusts agreement for what could happen by chance. Across the 22 ChatGPT models, kappa ranged from 0.65 to 0.97. Some models were relatively consistent; others changed their urgency label more often. The range is not a universal measure of randomness in consumer chatbots. It describes variability under this controlled API setup, prompt, and model snapshot. It is nevertheless directly relevant to the patient-facing boundary: an answer that changes when the input does not is a reliability issue even if one of the answers happens to be correct.

The study used three categories of consistency for each case. A model was always correct when every one of its ten recommendations matched the physician-derived urgency. It was sometimes correct when some trials were correct and others were not. It was never correct when none of the ten trials matched the reference. These categories answer different questions. Always-correct cases suggest a stable decision for that case. Sometimes-correct cases reveal a boundary that is sensitive to repeated generation. Never-correct cases show that repetition alone did not rescue the model under the tested conditions.

The pattern across model generations is easy to oversimplify. Older models had more cases that they never solved correctly. For gpt-3.5-turbo, 49% of the 45 cases were never correct across ten trials. For o4-mini-high, that proportion was 8.9%. That looks like progress, but the newer model also had 42.2% of cases classified as sometimes correct, compared with 11.1% for gpt-3.5-turbo. In other words, more cases became reachable, but a larger share of those cases produced mixed answers. Fewer permanent failures can coexist with more instability at the edge.

This distinction matters because “the model can answer this question” is ambiguous. It may mean that the model produced the reference answer once. It may mean that it usually produced the reference answer.