Why Better Prompts and More Compute Help—Then Stop

The LLM monitors tested in SLEIGHT-Bench do not fail for one uniform reason, so “make the monitor smarter” is an incomplete remedy. This chapter examines what extra reasoning, a coding-agent scaffold, and targeted prompts actually add to the monitoring system. The useful judgment is conditional: match the repair to the missing verification step, measure regressions on other blind spots, and treat an absent or malformed judgment as unresolved risk rather than permission to continue.

A monitor can fail because it never notices a relevant detail, because it notices the detail but cannot reconstruct its consequence, or because its prompt tells it to treat that kind of consequence as outside scope. More tokens, more tools, and more instructions address those causes differently. They should therefore be evaluated as separate interventions, not as interchangeable ways to increase a single notion of intelligence.

More compute helps when the missing step is verification

Extended thinking gives a monitor more room to maintain a chain of reasoning before it returns a score. In SLEIGHT-Bench, an Opus 4.6 monitor with extended thinking improved its per-trial catch rate by 13 percentage points over the no-thinking configuration. That is a meaningful gain, but its meaning is specific. The extra computation helped the tested monitor examine more of the transcript; it did not demonstrate that every blind spot had been removed or that a larger reasoning budget would generalize to every monitor and attack distribution.

The gain is easier to understand when the missing operation is explicit. Suppose the monitor must determine whether a later action uses a particular field in a long record, whether a transformation changed the content of a file, or whether several steps compose into an unsafe result. A longer internal analysis can allocate attention to the relevant positions and preserve more intermediate conclusions. It may still make a counting or arithmetic error, but it has at least been given room to perform the check instead of stopping at the first plausible interpretation.

The limitation is equally important. Extra tokens do not supply facts that the transcript omits. They do not tell the monitor which destination the organization approves, whether an instruction has legal authority, or which state belongs to a different session. They can help reason over evidence; they cannot create the evidence or define the policy. A longer answer can also become a longer rationalization if the monitor starts from a mistaken assumption and never tests it.

SLEIGHT’s agentic monitor adds a different capability. Instead of asking a model to inspect the transcript in one pass, the authors give it a Claude Code session and the transcript as a file. In the tested setup, the scaffold improves catch rates by an additional 20 percentage points over the extended-thinking Opus 4.6 configuration. The paper attributes particularly strong improvements to Counting and Needle-in-the-Haystack cases, where the coding agent can execute code to count, decompile bytecode, or compare text rather than relying on visual inspection.

This distinction is more useful than the generic claim that agents are “more powerful.” A tool can change the epistemic status of a conclusion. Reading a long file and asserting that a target appears at a certain position is one kind of evidence. Programmatically counting the file, comparing two versions, or parsing its structure is another.