When an Agent Can Solve a Task but Cannot Be Relied On
This series examines a specific problem in the evaluation of language-model agents: an agent may solve a short, well-defined task and still be unreliable when the work stretches across many dependent decisions. The focus here is the gap between capability and reliability. The practical question is not only whether an agent can produce a correct result once, but whether it can preserve useful state, recover from mistakes, verify what remains, and reach a dependable finish under a realistic time horizon.
That distinction matters because the tasks people want agents to perform are rarely isolated questions. They look more like repairing a software pipeline, reproducing a research result, auditing a multimodal dataset, or carrying a multi-stage analysis from raw inputs to a checked artifact. Such work unfolds through a sequence of observations and actions. Each step changes what the next step means. A benchmark that sees only the final state may miss the most important fact about the run: how much coherent work happened before the endpoint failed.
The score that looks like competence
The familiar pass@1 metric is simple: did one attempt reach the fully correct final state? For a short task, that can be a useful capability signal. It tells us whether the model can understand a bounded instruction, choose an appropriate action, and produce an answer or artifact that satisfies the evaluator. What it does not tell us is how that performance changes when the task requires persistence, or whether the same result would appear across repeated invocations.
The difference becomes consequential in production. A system invoked thousands of times is not valuable because it succeeds on an especially favorable attempt. It must succeed often enough, and fail in ways that people and surrounding systems can manage. The reliability framework supplied for this series makes that distinction explicit. It treats repeated success as a separate question from one-attempt capability and asks how success changes across task durations. Its motivating example, drawn from earlier tool-agent evaluation, reports a large gap between a single-attempt success rate and the probability that all eight repeated attempts succeed. Even without treating those figures as universal, the logic is clear: more opportunities to fail expose a property that a single pass rate cannot measure.
A longer workflow adds another layer. Suppose an agent correctly inspects a data file, identifies a schema mismatch, repairs one transformation, and produces an artifact that fails only a final hidden check. Under strict pass/fail grading, that run is indistinguishable from an attempt that never understood the assignment. The endpoint is different, but the score is the same. That is not merely a statistical inconvenience. It makes it difficult to know whether an agent needs better local reasoning, better time management, better verification, or a different interaction design.
The distinction is therefore not between an easy metric and a sophisticated metric. It is between different questions. Capability asks whether a solution is within the model's reach. Reliability asks whether the solution remains reachable as the workflow becomes longer, the state becomes harder to track, and the agent must decide when to continue or stop.
Why long work changes the problem
Long-horizon work is not just a short task with more words in its instruction. It has a different operating structure. The agent must read and modify files, run commands, inspect partial outputs, revise a plan, and recover when an assumption proves wrong. Intermediate state is both a resource and a liability: it lets the agent build on earlier work, but a mistaken interpretation can persist and distort everything that follows.
The supplied Long-Horizon-Terminal-Bench (LHTB) makes this structure concrete. It places an agent in a containerized terminal environment with a natural-language goal, relevant tools and assets, and an oracle implementation or simulator used for grading. The agent works through the terminal rather than returning a detached answer. The tasks cover scientific and engineering workflows, software work, multimodal analysis, interactive environments, and professional scenarios. Their common feature is not a particular domain. It is that the final objective depends on a chain of intermediate objectives that must be discovered and completed over time.
This design changes what counts as a meaningful mistake. In a short coding task, a faulty edit may be visible in the next test result. In a long workflow, the agent may spend many later actions compensating for an early misunderstanding, repeatedly inspect the wrong artifact, or consume its remaining budget without noticing that the plan has stopped producing new evidence. A locally correct action can therefore sit inside a globally failing trajectory. Conversely, a run that does not finish may still have created valuable, verifiable intermediate results.
LHTB is designed to test that difference rather than reward superficial familiarity with visible examples. Its tasks begin as complete but deliberately broken terminal projects. Public checks cover basic command behavior, formats, or simple examples, while hidden stress cases vary inputs and conditions: wrappers may be nested, fields renamed, values missing, images altered, or noise injected. An agent that only patches the visible case can appear competent at the first boundary and fail when the underlying workflow is exercised. The benchmark thus treats robustness as the ability to carry a solution through changing conditions, not merely to make one demonstration pass.
What a final-state score hides
The answer proposed by LHTB is dense, subtask-based grading. A task is divided into semantically meaningful intermediate goals. Some checks are binary—for example, whether tests pass or a service responds. Others give a continuous or thresholded score for a metric, a reproduced figure, or a simulator outcome. Campaign-style tasks can aggregate performance across episodes. The overall reward combines these subtask results with weights, so the evaluator can record how far the workflow progressed while retaining a threshold for full resolution.
This does not turn the benchmark into a transcript judge. The grader still evaluates objective evidence in the final environment: files, outputs, test results, or simulator responses. That is an important boundary. A partial score is not a claim that every preceding thought was correct, nor is it a direct measure of time spent. It is evidence that specified pieces of the intended artifact or behavior were present at evaluation time.
The value of that evidence is diagnostic. If two runs both miss the final objective, their partial rewards can show whether one reached most intermediate targets while the other failed at the beginning. Near-complete runs may point toward missing final checks or weak self-verification. Very low rewards may suggest that the agent never established a workable plan. Neither conclusion follows from a binary failure label alone.
The same reasoning appears in the second supplied paper's Graceful Degradation Score. Its purpose is not to excuse incomplete work. It is to preserve distinctions that matter when a difficult task cannot be completed in one uninterrupted attempt. A system that can reliably complete eight of ten critical subtasks is operationally different from one that completes one, even if both receive zero under a strict endpoint definition. Whether the difference is valuable depends on the task and the consequences of using the partial artifact, but an evaluation should at least make the difference visible.
Two studies, one reliability question
The two papers are complementary rather than interchangeable. LHTB is the concrete stress test: it shows what happens when agents must sustain terminal work for tens of minutes or more, and it records partial reward, workload, termination cause, and hidden-verifier success. Beyond pass@1 supplies a broader vocabulary for asking how that behavior changes across duration buckets, domains, repeated attempts, and trajectories. It introduces measures for decay, variance, partial completion, and possible behavioral meltdown.
Together they move the subject away from the idea that an agent has one stable score called “intelligence.” The relevant system includes the model, the interaction harness, the task, the evaluator, the available budget, and the state carried from one action to the next. Change the domain and a reliability curve can change. Change the horizon and a model's rank can change. Add a memory mechanism and the overhead may outweigh the benefit. Let an evaluation endpoint fail and an infrastructure problem may be mistaken for a model limitation.
That broader view also keeps the evidence in proportion. The reported results are benchmark findings, not a guarantee about every production workflow. LHTB's terminal environments and hidden verifiers test important forms of sustained execution, but they do not cover every kind of autonomy. The reliability framework's duration buckets are based on estimated human completion time, which can diverge from the number of tool calls an agent actually needs. The papers themselves identify these limits. Reliability is a stronger subject of inquiry when its measurements expose their own assumptions.
The question to carry forward
A useful mental model is to see an agent run as a trajectory rather than a single answer. The trajectory begins with a goal and its dependencies, moves through actions and observations, leaves evidence in a changing workspace, and ends in a completion or termination decision. At each point, four questions can be separated:
- Can the agent take the next locally appropriate action?
- How much of the intended workflow has it completed and verified?
- Can it finish within the available horizon?
- Does it know when the work is actually complete?
Short-task pass@1 answers only part of the first question and the endpoint of the second. The LHTB design makes the missing progress visible. The reliability framework asks how to measure the same problem across durations and repeated attempts. The central question for the rest of this series is therefore straightforward: can an agent sustain verified progress to a dependable finish when the work is long, stateful, and imperfectly observed?
That question is more demanding than asking whether the model can occasionally solve a hard task. It is also more useful. A system that fails transparently at a verified boundary may be easier to improve or supervise than one that produces confident false finishes. A system that completes most subtasks but consistently runs out of time may need decomposition or better state management rather than a larger model. To judge those possibilities, we first need to understand how LHTB turns a long workflow into a measurable evaluation system.