From Failure Modes to an Evidence Package

This chapter turns the observed failures of mental-health and patient-facing conversational AI into an assurance program. The focus is not a universal checklist or a single leaderboard score; it is the relationship between an intended clinical behavior, the test that can support that claim, the failure mechanism that appears, and the evidence needed before a system is trusted with people. Readers will be able to assemble a layered evaluation portfolio while keeping clear what synthetic testing, mitigation results, and regulation can—and cannot—prove.

Five questions that a safety case must keep separate

A deployment claim begins with intended use. “The chatbot offers emotional support,” “the chatbot guides a therapy exercise,” and “the chatbot responds to possible suicidal risk” are not interchangeable claims. Each implies different behavior, different harm pathways, and different evidence. A system that is useful for information retrieval may not be ready to perform treatment-like work. A system that responds appropriately to crisis prompts may still lose a therapy protocol across a long session.

The five-axis framework proposed in the supplied research turns these distinctions into a portfolio of tests. The first axis is therapeutic protocol fidelity: does the system follow the recognized phase or task structure? In PE, that means preserving the requirements of the relevant exposure phase. In CBT, it means completing the cognitive-restructuring work rather than substituting a generic safety preamble or stopping after validation. Fidelity is a claim about the intervention's mechanism, not about whether the prose sounds therapeutic.

The second axis is hallucination risk. A hallucination in this setting is not limited to a bizarre invented fact. It can be a fabricated diagnostic criterion, an incorrect treatment step, a false reassurance, or an invented medication interaction. A plausible statement can be more dangerous than an obviously strange one because a user may act on it. The proposed approach is to seed clinical misstatements and verify them against authoritative references, such as diagnostic criteria or clinical practice guidelines. The supplied paper presents this as a benchmark design, not as a completed, validated therapy-dialogue standard.

The third axis is behavioral consistency across turns. A system may take one risk position and later contradict it, validate a trauma narrative and then dismiss it, or lose its therapeutic role as the session lengthens. Consistency is not mere repetition. A system should change its response when the user's state changes, but the change should be justified by the conversation rather than caused by untracked drift. Testing this axis requires standardized multi-turn arcs and measures such as stance stability, risk-assessment stability, and emotional coherence.

The fourth axis is crisis safety. This includes recognizing possible risk, confirming what is happening, guiding the person toward human care when indicated, maintaining a supportive interaction, and respecting AI boundaries. A focused evaluation such as VERA-MH can operationalize these behaviors without claiming to measure treatment efficacy. The crisis axis should not be inferred from protocol fidelity, and fidelity should not be inferred from a crisis score.

The fifth axis is demographic robustness. A system may respond differently to older adults, people with different educational backgrounds, users from different cultures, or people who express distress in a different language.