Beyond Accuracy: Building a Process-Level Evaluation Stack

For a medical Agentic AI workflow, the evaluation problem is to connect the system’s intended clinical role with what it produced, how it acted, what evidence it used, and what happened to the people and workflow around it. This chapter turns that problem into a measurement stack and explains why encounter-specific rubrics can make expert judgment reusable. The goal is not to replace a familiar accuracy metric with another master score, but to choose evidence that answers the right question at each layer.

Match the measure to the clinical role

Accuracy is attractive because it is compact. A single number can make two systems appear easy to compare. But a medical agent does not perform only one operation. It may select a tool, retrieve a source, maintain context, interpret an intermediate result, decide whether to continue, and hand a recommendation or structured action to a clinician. A final-output score can be useful while leaving the rest of that chain unexamined.

The first evaluation question should be about intended use. A system that retrieves information for a clinician has a different burden of proof from one that drafts a note, triages a patient, recommends a treatment, initiates an order, or communicates directly with a patient. The same technical capability can carry different consequences depending on where it sits in the workflow and how much authority it has. Measures should therefore be selected after the task, user, action boundary, and consequence of error are clear.

Task performance remains the first layer. For classification or diagnostic assistance, accuracy, sensitivity, specificity, precision, recall, and F1 may describe whether the output matches an appropriate target. Detection tasks can require measures such as average precision. Image segmentation may need overlap and boundary measures, including Dice, intersection over union, or Hausdorff distance. These measures are not interchangeable; each captures a different relationship between a predicted object and the reference.

Generated clinical text makes the limitation especially visible. BLEU, ROUGE, METEOR, CIDEr, and BERTScore can quantify lexical or semantic similarity to a reference report. A generated report may be similar to a reference and still omit a clinically important finding, invent an unsupported one, misstate a negation, or contradict the underlying image. Labelers and graph-based measures can add signals about clinical entities and relations, but automated comparison still does not establish that a note is clinically correct. Expert assessment of factual accuracy, omissions, and consistency with the source evidence remains necessary for the questions that the text metric cannot answer.

The second layer is process reliability. It asks whether the agent completed the intended task through an appropriate path. Did it identify the right subproblem? Did it select a tool that could answer it? Were the tool inputs valid and within scope? Did execution succeed? Were intermediate results consistent with one another? Did the agent recover from a failed call, or did it continue as if the failure had not occurred? For an EHR query, correct code generation, successful execution, and clinically valid interpretation are separate outcomes.