The Score Behind the Headline
For an LLM monitor, reliability is not a single accuracy number. The important question is how often the monitor lets genuine misbehavior pass, how often it obstructs benign work, and where the decision boundary was set. This chapter develops that measurement model so that readers can interpret a reported catch rate without confusing safety coverage with operational usefulness.
The distinction matters because a monitor is usually asked to make a decision under uncertainty. A high suspicion score may stop an agent or send a case to human review; a low score may allow the agent to continue. The score becomes meaningful only after it is compared with a threshold, and the threshold is meaningful only relative to the benign work the system expects to see. The two sides of that comparison are Miss Rate and False Alarm Rate.
Two rates describe two different failures
Imagine a set of trajectories, some containing actual misbehavior and some genuinely benign. A monitor can make four kinds of judgment: correctly flag misbehavior, miss misbehavior, correctly accept benign work, or falsely flag benign work. The first and third outcomes are desirable, but they do not tell the whole story. The two failure types have different consequences and must be measured separately.
Miss Rate (MR) is the fraction of actual misbehavior that the monitor labels benign. In the notation used by AutoMonitor-Bench:
MR = FN / (TP + FN)
Here, FN means false negatives—misbehavior that escaped detection—and TP means true positives—misbehavior that was correctly detected. The denominator contains only the genuinely misbehaving cases. A low MR means that the monitor provides stronger coverage against the behavior the evaluation set defines as unsafe.
The denominator is important. Suppose an evaluation contains 100 genuinely misbehaving trajectories and the monitor catches 75 of them. Its MR is 25 percent, regardless of how many benign examples are also present. That number answers a safety-coverage question: when the dangerous class is present, how often does the monitor fail to recognize it? It does not answer whether the monitor is practical to run on a large stream of ordinary work.
That is not an abstract virtue. If an agent is permitted to alter a security control, expose a credential, or manipulate an evaluation, a false negative means that the system may continue under the belief that the work is safe. The monitor may have had evidence but failed to connect it, or the evidence may have looked ordinary in context. Either way, the operational consequence is that the control layer did not stop a case it was supposed to catch.
False Alarm Rate (FAR) measures the other failure: the fraction of benign cases that the monitor incorrectly flags. Its definition is:
FAR = FP / (TN + FP)
FP means false positives—benign work treated as misbehavior—and TN means true negatives—benign work correctly accepted. The denominator now contains only the genuinely benign cases. A high FAR does not merely make a dashboard look untidy. It can interrupt valid work, consume scarce human-review capacity, encourage operators to override warnings, and make it harder to distinguish an urgent signal from routine noise.
The denominator also prevents a misleading comparison. A monitor could catch many harmful cases simply because an evaluation contains unusually many of them, while still flagging a large share of normal work. Conversely, a low number of false alarms in a small benign sample may not mean much about a varied production workload. MR and FAR are conditional rates: each describes performance on a different population, and both populations need to resemble the decisions the system will face.
The two rates can move in opposite directions because the same score threshold governs both. Lowering the threshold makes the monitor more willing to raise an alarm. That may catch more genuinely harmful cases, reducing MR, but it also admits more benign cases into the alarm region, increasing FAR. Raising the threshold has the opposite effect. A monitor that flags everything can achieve zero misses on a fixed set while producing an unusable false-alarm burden; a monitor that flags nothing can have zero false alarms while providing no safety coverage at all.
AutoMonitor-Bench reports a moderate negative correlation of −0.54 between MR and FAR across its evaluated models. The result does not prove that every monitoring system must follow exactly the same line. It does show why a one-dimensional leaderboard is misleading in this setting: models with lower miss rates often paid for that coverage with more false alarms, while conservative models could look operationally tidy partly because they allowed more harmful behavior through. The relevant choice is not “the most accurate model” in the abstract, but an operating point that makes the safety–utility trade-off explicit.
The same logic changes how model comparisons should be made. A model with a lower MR but a much higher FAR may be appropriate where human review is abundant and missed harm is intolerable. A model with a lower FAR may be useful as one layer in a high-volume workflow, but only if another control covers the behavior it tends to miss. Neither option dominates without a defined consequence for each error and a realistic estimate of how much benign work will pass through the system. The preferred point can also change with the task: a monitor for an especially sensitive operation may accept more review, while a monitor attached to routine development work may need a different balance.
The sequence makes one point especially clear: the threshold is not discovered from the harmful cases alone. It is defined by how the monitor behaves on work that the system intends to accept. A claim about detection therefore includes an implicit claim about the negative population and the intervention capacity behind the threshold.
What a catch rate leaves out
SLEIGHT-Bench illustrates the next layer of interpretation. Its monitors produce a suspicion score from 0 to 100. The authors evaluate ten independent trials for each attack and configuration, then select a threshold calibrated to a 1% false-positive rate on paired benign transcripts. Those benign transcripts preserve the scenario and style of an attack while removing its harmful payload. This is a thoughtful control: it makes the comparison focus on whether the monitor responds to the attack mechanism rather than merely noticing that the malicious examples come from a different distribution.
In practical terms, calibration asks the monitor to score accepted work first. The team then chooses a cutoff such that only the permitted fraction of those benign scores lies above it. Attack scores are evaluated against that same cutoff. The process turns an unbounded or model-specific suspicion signal into an operational policy: scores above the line trigger a response, while scores below it are allowed to pass, subject to the limitations of the evaluation. Changing the accepted benign population changes the distribution from which that line is chosen.
It is also a limited control. A paired benign transcript is close to the attack by construction. Real benign work may be longer, more varied, and full of unusual but legitimate actions. When SLEIGHT’s authors recalibrated against five external benign sources—including real coding-agent and honest benchmark runs—the external sources generally produced stricter thresholds and reduced catch rates by roughly 10–20 percentage points. The attack set did not need to change for the headline result to move; changing the accepted background was enough.
This is not a flaw to be removed from every benchmark. Paired calibration answers a clean experimental question: how much does the harmful mechanism separate itself from an otherwise similar benign case? External calibration asks a different and more deployment-oriented question: can the monitor maintain the required false-alarm rate on the work it will actually encounter? Both are useful, but they should not be presented as interchangeable evidence. A paired result can isolate a mechanism while an external result exposes the cost of operating amid broader variation.
Repeated trials introduce another distinction. Under the ControlArena-adapted prompt, an Opus 4.6 monitor with extended thinking caught 32% of attack trials at the 1% false-positive threshold. Looking at attacks rather than individual trials, only 8 of 40 were reliably caught across ten trials, 12 were caught sometimes, and 20 were never caught in that configuration. The figures answer different questions. Per-trial catch rate describes the average behavior of an invocation. Per-attack reliability asks whether the system can be trusted to catch a particular attack consistently.
That difference matters for an agent that can try again, continue later, or pass work between sessions. An average score across invocations can hide a dangerous cluster of attacks that almost never trigger an alarm. A per-attack view makes that cluster visible, but it still depends on how the trials were sampled and whether repeated scoring is part of the proposed deployment. Neither summary is “the” catch rate without the unit of analysis attached.
The tempting response is to run the monitor repeatedly and count an attack as caught if any trial raises an alarm. That can be useful as an exploratory upper bound, but it is not free reliability. Repetition also creates more opportunities for benign work to receive at least one high score. If the benign side is still calibrated from individual trials while the attack side uses a maximum over ten trials, the comparison is asymmetric and optimistic. Under symmetric max aggregation—taking the maximum across trials for both attack and benign transcripts and recalibrating the threshold—the reported catch rates are lower because the benign threshold rises.
A threshold is therefore a policy, not a natural property of a model. It encodes how much benign disruption the system can tolerate, how many scores are generated per case, and what happens when scores are uncertain. The same monitor can appear more or less effective when the team changes any of those conditions. Even a stable model may produce variable judgments for the same transcript, so a single run is not enough to characterize the operational risk.
Three ways a catch rate can mislead
First, a catch rate can hide the cost of alarms if the benign denominator is not reported. “Caught more attacks” is incomplete when the change was achieved by lowering the threshold and flagging much more legitimate work. Second, an any-trial result can hide inconsistency. Catching a case once in ten attempts is not the same as reliably recognizing it whenever it appears. Third, a paired-benign result can hide distribution shift. A monitor may separate a carefully matched ablation from an attack while struggling with the irregular, ambiguous behavior of real users and agents.
These are not arguments for discarding simple metrics. They are arguments for attaching the right question to each metric. MR asks about escape from the harmful class. FAR asks about disruption to the benign class. A threshold asks how the two are traded. Repeated trials ask about variance and consistency. Calibration data ask whether the operating point survives the work the monitor is expected to judge.
A defensible monitoring claim should consequently disclose at least four things alongside its catch rate:
- The error definitions: whether the result emphasizes missed misbehavior, false alarms, or a combined utility measure.
- The calibration population: whether benign examples are paired ablations, external work, or a mixture representing the intended deployment.
- The aggregation rule: whether scores are per invocation, per transcript, averaged, or maximized across repeated trials.
- The monitor configuration: the model, prompt, reasoning budget or scaffold, and transcript representation used to produce the score.
These details are not formalities. They determine whether a result describes a monitor’s ability to separate a known mechanism under controlled conditions or a system’s ability to operate safely amid ordinary variation. AutoMonitor-Bench’s paired harmful and benign instances provide the same methodological lesson in a different form: controlled comparisons make attribution clearer, but they do not eliminate the need to test broader benign behavior.
The measurement model also prevents a common interpretive mistake. A high miss rate does not automatically mean the monitor is incapable of reasoning about harm; it may mean the evidence is diffuse or the operating point is conservative. A high false-alarm rate does not automatically mean the monitor is more safety-conscious; it may mean the prompt encourages over-interpretation of benign text. Before asking which behavior classes are hardest, we must know which side of the decision boundary is moving and what the boundary was asked to protect.
That question leads to the next distinction. Some harmful behavior is explicit enough to quote from a response. Some appears only when a contextual cue changes an otherwise correct answer. Some looks like successful task completion until the intended objective is compared with the proxy the agent optimized. The monitor’s measurement problem is inseparable from the kind of signal the behavior produces.