Where the Long Run Breaks: Progress, Timeouts, and False Finishes
This chapter examines what happens inside the unresolved runs of Long-Horizon-Terminal-Bench (LHTB). The focus is not merely that current agents often fail the benchmark, but how far they get, how much time and computation they consume, and why the run ends. That diagnosis supports a more useful judgment than a leaderboard alone: a timeout, a premature finish, and an infrastructure interruption require different explanations and different remedies.
A hard horizon creates a measurable completion gap
LHTB makes the phrase “long task” concrete through the operating burden of its runs. Across the reported evaluation of 17 models, a task consumed an average of about 239 episodes, 9.79 million tokens, and 88.9 minutes of wall-clock time before exit under a 90-minute limit. The estimated average cost was about $10.8 per task. These figures are estimates under the paper's token-pricing assumptions, not universal costs for deploying an agent, but they establish the scale of the experiment: the benchmark is measuring sustained execution near the edge of a large time and context budget.
That burden is reflected in the completion results. Grok 4.5, the strongest reported model in the LHTB study, reached the paper's relaxed resolution threshold of reward at least 0.95 on 13 of 46 tasks, or 28.3%. At perfect reward, it resolved 9 of 46, or 19.6%. The mean pass rates across the 17 models were only 6.4% at the 0.95 threshold and 3.2% at perfect reward. The restrained conclusion is that long-horizon terminal execution remained far from saturated for the tested systems. It is not that every run was hopeless, and it is not that the benchmark proves a universal limit. It is that finishing the complete workflow within the available horizon was still a binding constraint.
Cost makes the result more revealing rather than less. The most expensive model in the table, GPT-5.4 at about $25.77 per task, did not lead the resolution rate. Grok 4.5 reached its stronger pass rate at an estimated $10.95 per task, while lower-cost models occupied other points on the cost–reward frontier. Token use and price therefore cannot stand in for reliability. Spending more may buy more attempts or longer reasoning, but it does not guarantee that the agent will convert that budget into a verified final state. The relevant quantity is useful progress per unit of time and cost, under the same completion standard.
The reward distribution shows why the headline pass rate is incomplete. LHTB reports 782 model–task runs: 50 pass the 0.95 threshold, 241 receive less than 0.05 reward, and 491 fall between those two outcomes. The middle group is not a rounding error. It represents 62.8% of all runs that achieve some measurable progress but would be labeled simply “failed” by a binary endpoint. A further 223 runs reach at least 0.5 reward. The paper also counts 90 near-misses between 0.75 and 0.95, nearly twice the 50 passes.
Those figures do not mean that a 0.8-reward artifact is safe to use. A final hidden check may protect a critical requirement, and a missing requirement can make an otherwise polished result unusable. The point is diagnostic: the reward tells us that these attempts crossed more of the intended workflow than the runs near zero. That distinction can guide improvement. A system that repeatedly reaches the final verification boundary may need better stopping or a small amount of additional time. A system that never establishes a productive trajectory may need a different planning or execution capability.
The LHTB figures also caution against interpreting a high average reward as consistent reliability. Mean reward aggregates partial work across tasks, while the thresholded pass rate asks whether complete resolution occurred. Two models can have similar mean reward and different numbers of finished tasks, or similar pass rates and different amounts of useful incomplete work. These are not contradictory rankings. They describe different parts of the long-horizon problem.
The averages also need to be read with the benchmark’s stopping rule in mind. An 88.9-minute mean under a 90-minute ceiling is not the time a successful solution necessarily requires; it is the observed time until a run exits, finishes, or is cut off. A timeout is therefore a censored observation of the agent’s attempt, not a measurement of the full time it would have needed to solve the task. The same caution applies to the average of 239 episodes and 9.79 million tokens. Those quantities describe the interaction burden in the reported runs, but they do not tell us that every model used the same number of useful steps or that more tokens were converted into more verified work. The per-model table shows wide differences in resource use, which is itself part of the reliability story.
Threshold choice creates a second interpretive boundary. Reward of 0.95 is a practical relaxed resolution rule in the paper, while reward of 1.0 means every required check reaches full score. The gap between the two thresholds is informative: a run can satisfy almost all weighted requirements and still fail a critical hidden condition. That is exactly why the benchmark reports both thresholds and mean reward. The relaxed rate describes how often a run gets very close to resolution; the perfect rate describes how often it clears every encoded requirement. Neither rate alone says whether the remaining defect is minor, safety-critical, easy to repair, or impossible to detect from the available evidence.
Cost should be interpreted through the same lens. A low-cost run that stops early may look efficient while producing little verified value; a high-cost run may spend its budget repeating an unproductive line of inquiry. The reported Pareto comparison is useful because it places cost beside resolution rather than treating price as a quality proxy. But even that comparison remains conditional on the paper’s model prices, token accounting, time limit, and harness. The meaningful question is not “which model is cheapest?” It is “under the same completion standard, how much verified progress and successful completion does the system buy for the resources it consumes?”
The unresolved runs do not all fail the same way
The termination analysis provides the next layer of diagnosis. Among 660 unresolved runs—those below the 0.95 resolution threshold—518, or 79%, ended because the 90-minute budget expired while the agent was still working. Early exits accounted for 19%, and harness errors for 3%. The dominant symptom is therefore timeout, but the distribution alone does not prove that every timeout was caused by the model's reasoning. A fixed horizon, a particular harness, tool latency, and repeated exploration can all contribute. The evidence does show that time management is not a marginal concern: most unresolved trajectories reach their termination boundary by exhausting the clock.
The reward attached to those timeouts helps distinguish two possibilities. Across models, timed-out runs have mean rewards ranging from 0.10 to 0.35. They usually contain some progress, but they are not generally near-complete artifacts that missed one final check. This weakens the explanation that the benchmark's low pass rate is mostly a collection of trivial last-step mistakes. The reported pattern is more consistent with a longer failure chain: agents can perform locally correct actions and still fail to turn them into a completed artifact before the horizon expires.
That diagnosis changes what “make the model better” should mean. If an agent spends too long exploring redundant paths, a larger context window alone may not solve the problem. The agent may need to recognize which observations are decision-relevant, preserve verified state, and stop reopening questions that have already been settled. If the workflow contains genuinely expensive computation, faster execution or a larger budget may matter. The benchmark does not isolate those causes completely, so these are candidate mechanisms rather than experimentally separated effects. What it establishes is the importance of distinguishing a work-in-progress timeout from an early failure.
Early exits expose a different boundary. An agent can terminate on its own before the budget expires, implying that it judged the work complete or no longer worth continuing. Some of these runs are simply abandoned attempts. Others are false finishes: the agent has completed much of the visible work, stops with substantial time remaining, and still fails the hidden verifier. LHTB identifies 14 false finishes with reward at least 0.75. Its examples include Kimi K2.7 Code stopping on duckdb-optimizer-closure at reward 0.92 and GLM 5.2 stopping on apex-ib244-matter at 0.90; several agents stopped on another task with roughly 20 minutes left.
The important evidence in a false finish is the combination of three facts: the agent stops voluntarily, the partial reward is high, and the hidden requirement remains unsatisfied. That combination points toward weak self-verification or miscalibrated stopping, not simply an inability to make progress. The agent may have fixed the public examples and inferred that the task was done, without searching for residual defects that the hidden stress suite was designed to expose. Under binary grading, this behavior looks identical to an attempt that failed immediately. Dense reward reveals that the agent was close enough for a different control policy to matter.
A false finish should not be confused with a near-miss by definition. The paper uses a thresholded reward interval to identify high-reward early exits, while a near-miss is a reward category that can include runs ending for other reasons. The categories overlap conceptually but answer different questions. Near-miss asks how close the final state was. False finish asks whether the agent stopped itself while still short of the verifier's standard. Keeping those questions separate prevents a high score from being treated as proof that the agent knew it was finished.
Harness errors form the smallest reported category, but they matter disproportionately for interpretation. An API exception, provider interruption, or verifier-side failure is not evidence that the model could not solve the task. If it is mixed into the model's failure count, a reliability study can mistake the execution infrastructure for an agent limitation. LHTB's 3% share is small enough not to explain the main completion gap, yet large enough to justify reporting it separately. Long evaluations increase exposure to rate limits, timeouts, tool failures, and environment faults; the measurement system must account for that exposure rather than assume it away.
The state model makes the attribution problem visible. “Partial” describes the state of the work; “timeout” and “early exit” describe how the run terminated; “false finish” adds an evidence-based classification to one kind of early exit. They are not interchangeable labels. A run can be partially complete and still time out. It can be near completion and still stop falsely. It can also be interrupted before the model's behavior can be judged.
What the benchmark can diagnose—and what it cannot
The strongest LHTB conclusion is not a single number. It is the pattern formed by workload, reward, and termination. Agents often make some progress, but the majority of unresolved runs do not complete within the budget. A smaller set stops too soon despite substantial progress. A still smaller set fails because the harness cannot sustain the evaluation. This pattern makes long-horizon completion a distinct engineering problem: it concerns the coordination of plans, state, verification, and time across a sequence, not only the correctness of the next command.
The distinction also creates a disciplined order for investigating a failed run. First ask whether the final state contains meaningful verified progress. Then ask whether the agent was still working when the budget ended or chose to stop. Finally check whether the environment and evaluator remained available. Those questions do not solve the failure, but they keep the proposed remedy connected to the observed boundary. More local reasoning is not the obvious answer to a provider outage. A larger budget is not the obvious answer to a false finish. A new memory mechanism is not automatically the answer to redundant exploration.
LHTB's evidence remains benchmark-specific. The 90-minute limit, containerized tasks, hidden verifier, model harness, and cost assumptions define the conditions under which the pattern was observed. The false-finish classification depends on the chosen reward threshold and the paper's definition of early exit. The results strongly support a need to measure partial progress and termination cause, but they do not by themselves prove that any particular recovery policy will improve production outcomes.
That qualification is part of the result, not a retreat from it. A reliability measure is useful when it makes the next uncertainty clearer. LHTB shows that “failed” is too coarse, then gives us enough structure to ask whether a run ran out of horizon, stopped without verification, or never received a fair chance to finish. The next step is to turn those distinctions into a broader vocabulary for reliability across different durations, domains, repeated attempts, and trajectory behaviors.