The Conversation Is Part of the Test

This chapter follows a patient-facing health chatbot beyond its first apparently safe answer. The focus is how to tell a failure of document-grounded accuracy from a failure to follow behavioral instructions, then how persistent conversation can expose both. Readers will gain a way to evaluate red-team results by cause and severity, rather than treating every bad answer as the same kind of hallucination or treating a single-turn pass as evidence of robust safety.

Two kinds of adherence, two kinds of failure

The case study here concerns a chatbot for health-related social needs, or HRSN. Its purpose was to help users find resources such as food pantries, shelters, and financial aid based on a location and other attributes. It is not a therapy system, and its numerical results should not be presented as mental-health treatment outcomes. Its value for mental-health AI lies in the testing logic: a patient-facing chatbot can fail because it misrepresents its approved information source, or because it abandons the rules that govern what it may say and how it should behave.

The study calls the first problem Document Adherence, or DA. A retrieval-augmented generation system, commonly called RAG, retrieves information from a selected document or knowledge base before producing an answer. A DA test asks whether the answer accurately represents that retrieved authority. If a resource document says a pantry is closed on Sundays, the chatbot should not accept a user's claim that it is open twenty-four hours. If the document says nothing about parking or a phone-lending program, the chatbot should not invent those details or treat the user's assumption as a fact.

DA is dynamic because the authoritative material can change from one turn or resource to another. The system must keep track of which document supports the current answer. A model might be faithful to one retrieved resource and then be asked about another. It might also receive a user statement that combines a correct detail with an unsupported premise. The evaluation must compare the output with the document actually available in that exchange, not with a general sense that the answer sounds plausible.

The second problem is Instruction Adherence, or IA. IA concerns the static behavioral and scope rules supplied to the system: remain on topic, engage the user's question, ask useful follow-ups, use only approved knowledge sources, avoid unvetted advice, and respond safely to distress or hostile language. A response can be factually correct in the abstract and still be an IA error if it answers a question outside the chatbot's authorized scope from the model's general memory.

This distinction is especially important for health-related conversations. Suppose a user asks whether fast food is a healthy alternative after the chatbot has found a food pantry. The answer may contain ordinary nutritional advice that is factually reasonable. If the chatbot's rules require it to rely only on the retrieved resource database and redirect questions outside that source, providing the advice is still a safety and scope failure. The problem is not necessarily false nutrition. It is that the system has presented unvetted knowledge as if it were within its approved role.

DA and IA can also fail together. A model might invent details about a resource while ignoring the instruction to correct a user's false premise.