Monitoring Must Connect Signals to Decisions
A deployed clinical AI system needs more than a dashboard of model scores. It needs a monitoring plan that can tell whether the system is running, whether its outputs remain reliable, and whether those outputs still improve the workflow they were meant to support. This chapter turns the earlier evidence about calibration, telemetry, and delayed labels into an operating design: three complementary lines of sight, each connected to an owner, a review cadence, and a response.
Three lines of sight for one deployed system
The practical framework developed at Stanford Health Care organizes monitoring around system integrity, performance, and impact. These are not three competing definitions of success. They are three views of the same deployed arrangement. A system can be available but inaccurate, accurate but ignored, or useful in principle but unable to deliver its output where action occurs.
System integrity: does the system run end to end?
System integrity follows the path from input retrieval to inference to output delivery. It asks whether the service is available, whether the expected data can be found, whether the model runs at the right time, and whether the result reaches the place where a user or downstream application expects it. Relevant signals include uptime, outages, API or inference latency, data-retrieval failures, missing inputs, warnings, timeouts, and output-generation errors. For a generative system that depends on an external language-model service, request latency, response size, token use, and error codes can add to that view of system health.
Integrity monitoring catches a failure that performance evaluation may not see promptly: no prediction arrives, the prediction arrives after the clinical decision, or it is posted to the wrong destination. A model could still score well on a benchmark while the deployed system is unavailable during the hours when its output matters. Conversely, an uptime report cannot tell us whether the prediction is accurate or clinically useful. Integrity signals establish execution and pipeline health; they do not establish benefit.
The response speed should match the nature of the signal. A broken endpoint or repeated inference failure may require real-time technical support. A persistent increase in errors or downtime can then be reviewed as a governance issue, especially if the deployment has no practical fallback. This separation prevents a severe outage from waiting for a quarterly model review while also preventing every transient service event from being treated as evidence that retraining is needed.
Performance: are the outputs still accurate, equitable, and fit for use?
Performance monitoring asks whether the system's outputs remain reliable over time. For traditional predictive systems, the relevant measures depend on the task but may include sensitivity, specificity, positive predictive value, AUROC, AUPRC, calibration, and the rate at which the system flags cases. The earlier evidence shows why calibration should be included when probabilities or thresholds guide action.