Where Medical AI Agent Workflows Fail: Architecture and Evidence Interfaces
Medical Agentic AI workflows can be organized in different ways, but every arrangement creates interfaces where responsibility and evidence cross from one component to another. This chapter compares those arrangements by what they must do and what they expose to failure, then follows an error through the system to show what a useful audit trail must preserve. The practical result is a way to assess architecture without assuming that a more elaborate design is automatically more reliable.
Architecture is a distribution of responsibility
The recurring architecture families in medical Agentic AI are not mutually exclusive product categories. They are ways of distributing work. A single workflow may use one central controller, several specialized agents, external knowledge, code execution, images, structured records, and clinician oversight at the same time. The useful question is therefore not “Which architecture is best?” but “Who owns each decision, what crosses each interface, and how can the next component check what it receives?”
In a single-agent tool-use design, one language or multimodal model acts as the controller. It interprets the clinical task, plans a sequence, selects a tool, and integrates the returned results. The tools may include an EHR interface, an image-analysis model, a clinical database, a code interpreter, a guideline repository, or a statistical calculator. This concentration can make the control path relatively easy to describe: the controller chooses, the tool performs a bounded operation, and the controller incorporates the result.
The same concentration creates a dependency. The controller must understand the task correctly, choose an appropriate resource, provide valid inputs, and interpret the output within the case. A strong image model cannot repair a decision to invoke it for the wrong modality. A correct calculator cannot compensate for a query that omitted a relevant time period. The architecture is not unsafe because it has one controller; it simply makes controller interpretation and tool selection particularly important places to test.
A multi-agent design distributes work among roles. One agent might focus on image interpretation, another on clinical reasoning, another on medication safety, and another on verifying a draft. The potential benefit is meaningful specialization and the opportunity to expose disagreement. But role labels do not guarantee independent expertise. If the agents share the same blind spot, repeat one another’s unsupported assumption, or communicate conclusions without their underlying evidence, their agreement can look like confirmation while adding little information.
Coordination introduces its own interfaces. Someone must decide which roles are needed, define what each role receives, combine conflicting outputs, and determine when disagreement is resolved or escalated. More communication can increase computational cost and create more paths for error propagation. Research reviewed in the medical-agent literature does not establish that multi-agent systems consistently outperform strong single-model or specialized baselines. Collaboration is valuable when the roles correspond to real task requirements and the interaction contributes independent, checkable evidence—not simply more generated text.
Knowledge-augmented systems make an external source part of the workflow. The agent can retrieve a clinical guideline, drug label, medical paper, institutional protocol, or previous case before reasoning or generating a response. This can improve access to current information and make a claim more traceable than one based only on model parameters. It also shifts part of the reliability problem to the source interface. The system must locate a relevant passage, preserve its meaning, account for its date and authority, and apply it to the patient context for which it was retrieved.
A retrieved source can therefore be genuine but still unsuitable. A recommendation may address a neighboring condition, assume a laboratory value not present in the case, or have been superseded by a newer instruction. The existence of a citation says that a source was attached; it does not say that the source supports the specific claim or action. A knowledge interface needs tests for relevance, temporal validity, coverage, and faithful use, not only tests for whether a link appeared in the response.
Multimodal medical agents coordinate images, text, structured clinical data, and specialized visual tools. This organization fits clinical work because a finding may need to be interpreted alongside a report, laboratory measurements, a prior image, and longitudinal history. But the components do not speak the same representational language. An image model can identify a region, an EHR contains coded events, and a language model produces a textual interpretation. The system must preserve the relationship among them.
That relationship is a cross-modal interface. A report claim should remain consistent with the image evidence and with the output of a localization or segmentation tool. If a visual finding is not grounded in an identifiable region, or if the text silently changes the meaning of a structured result, the final narrative can sound coherent while losing its evidentiary basis. Differences in equipment, acquisition protocols, image quality, and reporting conventions can make the same interface behave differently across institutions.
Code execution cuts across these families. Programmatic analysis can help an agent reason over structured EHR data, calculate temporal relationships, or examine a cohort. It also creates a sequence that should not be compressed into “the code worked.” The program must represent the intended question, execute against the correct data, and yield an output that the agent interprets clinically. Execution errors and data-security concerns are separate from the question of whether the final explanation sounds plausible.
The architecture map is best understood as a set of responsibility boundaries:
The flowchart shows why architecture labels alone cannot answer a safety question. The central controller, specialized components, knowledge sources, and human reviewer may all contribute to one action. What matters is what crosses each interface: a query, a source, a mask, a computed value, a role-specific conclusion, an uncertainty signal, or an authorization. Those objects need enough context and provenance for the next stage to judge them.
A local error can become a system-level story
Consider a multimodal workflow that must answer a question about an image and a patient’s history. The first failure might be a planning error: the agent identifies report generation as the task when the actual need is comparison with a previous study. The next step may then be locally competent but globally irrelevant. A retrieval module can find an authoritative source, and an image tool can produce an accurate segmentation, yet neither result answers the question the workflow should have asked.
A second failure can occur at the tool boundary. The agent may select the right resource but pass an invalid or incomplete input. A segmentation result may be technically well formed but correspond to the wrong region or image series. A code interpreter may execute successfully on a subset of the record while omitting a clinically important time interval. The downstream model sees an output with the appearance of precision. Unless the input and scope are recorded, a reviewer may blame the final language rather than the earlier boundary error.
Interpretation is another distinct point of failure. The tool can return the expected object while the controller reads it incorrectly. This is especially important when a specialized component returns a probability, a mask, a coded value, or a structured parameter rather than an ordinary sentence. The final answer may be fluent because language generation smooths over the distinction between “the tool found this” and “the clinical record establishes this.” Process evaluation must keep those statements separate.
Retrieval failures can propagate in a similar way. Suppose the system retrieves a guideline that is current but applies to a different patient group. If the mismatch is not represented in the state, later steps may treat the recommendation as directly applicable. The citation then strengthens the appearance of confidence while the actual evidence relationship is weak. A trace should show the original question, the source selected, the passage used, the patient attributes considered, and the claim that resulted. It should also show when no adequate source was found or when sources disagreed.
Multi-agent communication adds a social-looking version of the same technical problem. One agent’s unsupported conclusion can become another agent’s input. A verifier may check wording rather than evidence. An arbitration step may resolve disagreement by choosing the majority view without establishing why that view is better. The final system can report consensus even though the underlying errors were correlated. Reliable collaboration therefore needs defined roles, communication protocols, evidence arbitration, disagreement handling, and a route to human review when the conflict remains unresolved.
The important distinction is between local correctness and global validity. A transcription model may correctly transcribe the audio it received while the wrong audio segment was selected. A retrieval component may accurately quote a passage that does not fit the case. A language model may faithfully summarize an intermediate result that was itself misinterpreted. Each component can appear successful under a narrow test while the workflow fails as a whole.
This is why the audit record should follow the chain rather than merely store the final answer. At minimum, a clinically useful record should connect the task decomposition to the selected tools, the inputs supplied, the returned outputs, the sources retrieved, the intermediate state, the uncertainty or abstention decision, the clinician intervention, and the final action. The record need not reproduce an unverifiable internal reasoning narrative. It needs to preserve observable facts and relationships that a reviewer can check.
The design of that record has operational consequences. If tool inputs are not retained, a later reviewer cannot determine whether a failure belonged to selection or execution. If retrieval timestamps and source identities are missing, the reviewer cannot assess temporal validity. If intermediate outputs are overwritten by the final note, the path of error disappears. If clinician overrides are recorded only as edits, rather than as evidence that the system’s proposal was unacceptable or incomplete, the evaluation loses an important measure of workflow burden and system behavior.
Auditability also has to survive change. An agent may behave differently after a model update, a knowledge-base revision, a new tool becomes available, or a dialogue supplies a different intermediate result. In a multi-agent system, communication changes can alter the path even when the final model versions are unchanged. Records should make those dependencies visible enough to compare versions and detect new failure patterns. This is particularly important when a system moves from a controlled benchmark into an institution with different EHR schemas, coding practices, devices, or documentation conventions.
There is a real trade-off between explicit control and system overhead. More stages and records can improve observability, constrain unsupported actions, and make responsibility clearer. They can also increase latency, maintenance work, computational cost, and the number of interfaces that require testing. A system with fewer components may be easier to operate but offer less specialized evidence. A system with many components may cover more kinds of information while making arbitration and failure recovery harder. The decision cannot be made from component count alone; it depends on the clinical objective, the consequences of error, and whether each interface contributes a testable benefit.
The same caution applies to complexity scores or architecture diagrams. The medical-agent review uses a descriptive complexity score based on features such as planning, tools, retrieval, memory, reflection, collaboration, multimodal processing, and workflow integration. It explicitly does not treat that score as a validated measure of capability, system quality, autonomy, or clinical readiness. A more complex workflow may create more opportunities to gather useful evidence, but it may also create more ways for evidence to be lost.
A sound teardown therefore ends with questions about evidence and ownership. What was the system trying to accomplish? Which component selected the next action? What crossed the interface? Could the recipient verify the input and output? Where did uncertainty enter? Who was authorized to proceed, and who could intervene? If those questions cannot be answered from the record, a good final answer may still be only a lucky endpoint.
The next step is to turn these failure surfaces into an evaluation stack. The goal is not to count every interface indiscriminately. It is to choose measures that reveal whether the agent acted appropriately, used supportable evidence, handled uncertainty, and improved the clinical workflow for its intended role.