Why Fluent Models Can Still Break in the Middle

This chapter examines a specific danger in language-dependent medical AI: the system can sound certain while its clinical answer is wrong. The focus is the gap between confidence, calibration, and safety, followed by the token-level evidence that multilingual instability can concentrate during the middle of a reasoning trajectory. The practical judgment is deliberately narrow: high confidence should not authorize automatic approval, and an intriguing process signal should not be promoted into a proven causal explanation.

Confidence is not the same as safe triage

When people hear that an AI system is “confident,” they may imagine that the system has recognized a clear pattern and has good reason to trust its answer. In a clinical workflow, that interpretation is unsafe unless confidence has been tested against correctness and consequence. A model can assign a high probability to the next token because that token is linguistically likely, while the overall clinical interpretation is wrong.

The cross-lingual study tested a windowed inverse-perplexity signal as a form of confidence. Perplexity is a measure related to how surprising the model finds the tokens it produces; lower perplexity can be transformed into a higher apparent confidence. The study did not use the first or last parts of the output indiscriminately. It focused on the middle 60% of tokens to reduce the influence of fixed templates and other wording that may be predictable without being clinically meaningful.

That design choice is important, but it did not make the signal a safety filter. Across the tested diagnostic tasks, the reported area under the receiver operating characteristic curve, or AUROC, ranged from 0.41 to 0.66. AUROC asks whether a signal tends to rank correct outputs above incorrect ones. A value near 0.5 is close to chance, while a higher value can indicate better ranking. The range therefore does not support a reliable language-independent threshold for deciding which clinical outputs can be approved automatically.

The study also reported expected calibration error, or ECE, ranging from 0.10 to 0.53. Calibration asks a different question: when a model says it is 80% confident, do outputs in that group turn out to be correct about 80% of the time? A model can rank some correct outputs above incorrect ones while still assigning confidence levels that are systematically too high. AUROC and ECE should therefore be interpreted together, and neither one is the same as a clinical safety endpoint.

The most concerning example combined high apparent confidence with only 32.7% accuracy in a Persian glaucoma condition. That is worse than an output that visibly expresses uncertainty. Visible uncertainty may prompt a clinician to look more closely. High confidence can reduce vigilance and create false reassurance, especially when the output is fluent and fits the expected format. A threshold that says “approve when confidence is high” could therefore select the wrong outputs precisely because they look least in need of review.