The Answer Is Not the Whole System: Why Medical AI Agents Need Process-Level Evaluation
Medical Agentic AI is the subject of this series: clinical software that coordinates models, tools, evidence, and human decisions across several steps. The immediate question is not whether such a system can produce a convincing answer. It is whether the path from a clinical goal to that answer was appropriate, traceable, safe, and useful in the workflow where someone may rely on it. Understanding that difference gives readers a practical way to judge claims about medical agents without confusing technical fluency with clinical readiness.
A new kind of clinical system
Much of medical AI has historically been built around a defined mapping. An image goes in and a classification comes out. A risk model estimates a probability. A language model drafts a report. These systems can be complicated, but their intended task is usually narrow enough to describe as an input-output relationship.
A foundation model broadens what a model can represent. It may understand language, interpret an image, connect concepts, or work across modalities. But a foundation model is not automatically an agent. Agentic AI organizes one or more models into a goal-directed workflow. The system may decide what subtask comes next, retrieve information, call a specialized model, maintain intermediate state, inspect a result, revise its plan, or ask a clinician to intervene.
That distinction matters because the system’s behavior is no longer contained in its final generated text. Imagine a clinical imaging workflow. A model might interpret a scan or draft a report. An agentic workflow could also decide that it needs lesion localization, call a segmentation tool, retrieve a relevant prior report or guideline, compare the new finding with longitudinal context, and assemble a result for review. Each step can add value. Each step can also introduce a new way to be wrong.
The word “agent” therefore tells us less than the workflow does. A single tool call does not by itself establish meaningful agency, just as a sophisticated model does not by itself establish multistep execution. The important questions are concrete: What goal is the system pursuing? What actions can it take? What information can it retrieve? What state does it carry forward? What happens when a tool fails or the evidence conflicts? Who can stop, revise, or approve the resulting action?
A recent scoping review illustrates why these questions are timely. Across five sources, it identified a provisional evidence-mapping set of 557 unique studies of medical systems with explicit agentic mechanisms. Those studies covered question answering, medical imaging, electronic health record analysis, drug safety, and clinical-trial research. Yet the evidence base was still dominated by prototypes, public benchmarks, simulated environments, retrospective data, and small expert evaluations. The field has demonstrated substantial technical activity without yet demonstrating a comparable level of prospective clinical effectiveness or generalizability.
This is not an argument against medical agents. It is an argument for describing them accurately. They are emerging clinical workflows whose usefulness depends on how their parts interact and under what boundaries they operate.
Why a convincing answer can still be unsafe
A final answer is an observation at the end of a process. It is not a complete account of that process.
Suppose an imaging agent produces the correct diagnostic label. That result could have been reached after selecting an appropriate detection tool and correctly interpreting its output. But it could also have followed an unsuitable tool choice, confusion between imaging modalities, or a misread segmentation result. The final label may happen to be correct while the path that produced it is unreliable. On another case, the same hidden defect could lead to harm.
The same pattern appears in electronic health records. An agent may generate code to query longitudinal data, execute the code, summarize the returned values, and then recommend a next step. These are different checkpoints. Correct code generation does not guarantee successful execution. Successful execution does not guarantee that the data were interpreted correctly. A fluent summary does not prove that the query addressed the clinical question or that missing information was recognized.
Retrieval creates another layer of risk. A response can contain citations and still be poorly grounded. The retrieved source may be out of date, irrelevant to the patient’s situation, or authoritative in one context but not the one at hand. The system may quote it accurately but draw a conclusion the source does not support. In a clinical workflow, “there is a citation” and “the citation supports this action” are separate claims.
Feedback and iteration do not solve these problems automatically. An agent that notices an intermediate failure may recover by selecting another tool or asking for review. It may also repeat the same mistaken assumption, carry a bad intermediate result into later steps, or turn disagreement among several agents into a superficial consensus. More steps can mean more opportunities for correction, but also more interfaces where errors can propagate.
The execution path is therefore part of the object being evaluated:
The diagram does not describe one mandatory software architecture. It shows why evaluation must follow the transitions that connect context, action, evidence, and decision. A trustworthy system should make those transitions observable enough to inspect. That does not require exposing an unverifiable private reasoning narrative. It does require recording which tools were called, what evidence was retrieved, what intermediate results were produced, where uncertainty entered, and when a clinician changed or overrode the system’s course.
This broader view changes the meaning of accuracy. Accuracy remains essential, but it answers only one class of question: how often did the system’s output match the relevant target under the test conditions? It does not answer whether the agent used the right information, respected its action boundaries, recovered from failure, or reduced rather than increased the work imposed on clinicians.
From benchmark result to clinical evidence
The review organizes a fuller evaluation around six connected dimensions: task performance, process reliability, evidence traceability, safety, clinical utility, and external validation.
Task performance should be measured in a way that fits the intended use. Classification and detection may require measures such as sensitivity, specificity, precision, recall, or F1. Segmentation may need overlap and boundary measures. Generated reports may be compared for lexical or semantic similarity. But similarity to a reference report does not establish that the report is clinically correct, complete, or consistent with the underlying image. The metric must match the consequence of the task.
Process reliability asks what happened on the way to the result. Did the agent complete the task? Did it choose an appropriate tool? Were tool inputs valid? Did execution succeed? Were intermediate outputs consistent? Did it recover from a failed call or escalate when recovery was not justified? These questions are especially important when the system can take multiple actions rather than simply generate a static prediction.
Evidence traceability asks whether the information supporting a claim can be followed back to an appropriate source. It includes authority, relevance, recency, coverage of the clinical question, and fidelity between the source and the claim. Safety adds questions about uncertainty, permissions, fallback behavior, abstention, escalation, and human oversight. Clinical utility asks what the system does to reporting time, revision burden, overrides, latency, trust, and final decisions. External validation asks whether the observed behavior survives new institutions, populations, devices, workflows, and data conventions.
These dimensions should not be collapsed into one master score. A system may perform well on a static task while failing to recover from tool errors. It may retrieve authoritative sources while using them inconsistently. It may reduce drafting time while increasing the number of clinically important corrections. It may work on historical data yet behave differently when placed in a live workflow with interruptions and incomplete information.
The order of evidence matters as well. Static benchmarks provide controlled tests. Interactive agent benchmarks can reveal tool use and state management. Retrospective clinical data add real-world variation without exposing patients to a live system. External evaluation tests transportability. Prospective workflow studies, including silent deployment and controlled clinical use, examine behavior closer to routine practice. A higher stage is more relevant to deployment, but it is not a certificate of safety or effectiveness by itself.
The central judgment is consequently modest but demanding. Before asking whether a medical agent is ready for clinical use, ask what exactly has been shown. A benchmark can establish that a capability is technically possible. A process trace can show how the capability was exercised. A workflow study can show how clinicians and patients are affected. External and prospective evidence can test whether those findings hold beyond the original setting. Each answers a different question.
That distinction is why the next step is to examine the agent’s operating mechanism rather than jump directly to a score. Planning, memory, tool use, retrieval, feedback, and human escalation are not decorative features around an answer. They are the chain through which the answer becomes clinically meaningful—or fails to do so. The practical question is how that chain behaves at each transition, especially when the information is incomplete and the stakes are high.