A Reliability Vocabulary for Work That Takes Time

The reliability framework in Beyond pass@1 asks how to measure the behavior that a long-horizon terminal benchmark makes visible. Its focus is not to replace pass@1 with a more impressive-sounding number, but to separate several questions that an endpoint score conflates: how performance changes with duration, how outcomes vary across tasks, how much incomplete work is useful, and whether the trajectory has begun to lose coherence. This vocabulary helps readers interpret long-horizon agent reliability without treating one metric as a complete description of the system.

Duration and domain are inputs to reliability, not footnotes

The framework models an agent task as more than an instruction and an answer. It includes a workspace, weighted subtasks, available actions, a duration bucket, a domain, an estimate of the agent steps needed for a correct solution, and an evaluator that scores the resulting workspace. An episode is the sequence of action–observation pairs produced by one run. The model, the scaffold that mediates its tools, the task, and the evaluator jointly determine the outcome.

The duration buckets are based on estimated human completion time: short tasks take up to five minutes, medium tasks five to 30 minutes, long tasks 30 to 120 minutes, and very-long tasks at least 120 minutes. That choice is useful because it gives the study a common scale for work of different sizes. It is not a claim that human time is the same thing as agent difficulty. The framework explicitly keeps a separate estimate of the number of tool calls a successful agent is expected to need.

The distinction matters because people and agents pay different costs for the same task. A document-processing workflow that takes a human 45–60 minutes may require only four to eight tool calls when the transformation is structured and deterministic. A software-engineering task assigned a similar human duration may require 15–25 calls because the agent must inspect several components, preserve behavior across them, and run repeated checks. A human-time bucket can therefore contain tasks with very different opportunities for an agent to lose state, make a bad decision, or consume its budget.

Domain adds another source of structure. The study separates software engineering, web research, and multi-file document processing. These domains differ in determinism, feedback, external dependence, and the kind of state that must be maintained. A software repair can fail because several changes interact. Web research can be affected by live content and the need to synthesize information across sources. Document processing may be long for a person but relatively direct for an agent when schemas and checks are clear. If those domains are mixed into one curve, a difference in task structure can look like a pure duration effect.

This is why the framework treats reliability as a function of duration and domain rather than as a fixed property of a model. The same model can have a shallow decline in one domain and a steep decline in another. The distinction also prevents an apparent improvement at a longer bucket from being read too quickly. A different task pool may be easier for the agent even if humans would spend more time on it. The paper's non-monotone document-processing pattern is evidence for caution about the proxy, not evidence that long work is generally easier.