From Benchmark Scores to Reliable Autonomy

This series has followed one subject from two complementary directions: Long-Horizon-Terminal-Bench (LHTB), which measures agents working through difficult terminal environments, and Beyond pass@1, which proposes a broader science of reliability across duration, domain, repetition, partial completion, and trajectory behavior. The central question is now practical: what should a person conclude before trusting an agent with consequential long work? The answer is not a new universal score. It is a way to match measurements and controls to the actual horizon, state, evaluator, and cost of failure.

Reliability is a property of the whole long-horizon system

The two studies jointly support a stronger claim than “long tasks are harder.” Long-horizon reliability is the ability of an agent system to preserve verified progress through a changing workflow and reach a defensible finish under stated conditions. The system includes the model, the tool-using scaffold, the task and its domain, the workspace state, the evaluator, the time and context budget, and the infrastructure that keeps the run alive. A model can be highly capable in one part of that system and unreliable in another.

LHTB supplies the concrete environment in which this becomes visible. Its 46 tasks place agents in containerized terminal work involving code, data, scientific artifacts, multimodal inputs, simulations, and other technical or professional workflows. The agent must inspect the environment, choose commands, change state, study outputs, revise its approach, and pass hidden stress checks. The benchmark reports a workload of about 239 episodes, 9.79 million tokens, and 88.9 minutes per task on average across 17 evaluated models, with an estimated average cost near $10.8. The strongest reported model reaches the relaxed 0.95 reward threshold on 13 of 46 tasks, or 28.3%, and reaches perfect reward on 9 of 46, or 19.6%.

Those numbers do not mean that the model has a 28.3% chance of performing every possible long task. They describe a particular benchmark, harness, budget, and verifier. Their value is that they make the completion problem measurable. The reward distribution shows that an endpoint threshold hides a large middle: of 782 model–task runs, the paper reports 50 at or above 0.95, 241 below 0.05, and 491 between those levels. A system that repeatedly reaches meaningful intermediate conditions is not equivalent to one that fails before establishing a workable trajectory, even if both are counted as unresolved.

Beyond pass@1 adds the dimensions that a single LHTB-style endpoint does not fully capture. Its full study describes 396 tasks across four human-time duration buckets and three domains, 10 open-source or open-weight models, two interaction scaffolds, and three repeated episodes per model–task condition. The plan was 23,760 episodes, of which 23,392 were reported completed after deduplication. The study's aggregate ReAct pass@1 falls from 76.3% on short tasks to 52.1% on very-long tasks. Its GDS measure retains partial completion, while its other proposed metrics examine variation across horizons and changes inside trajectories.

The figures come from different experiments and should not be merged into one leaderboard. LHTB's reward is tied to its own subtask decomposition and hidden verifier. The framework study uses its own task registry, domain buckets, scaffolds, and evaluator. What can be combined is the measurement logic. One study shows why dense progress and termination evidence are necessary in a difficult workflow. The other shows why reliability must also be compared across duration, repeated attempts, and domains.

That logic can be stated as a set of distinctions:

• Capability asks whether the model can solve a task on an attempt.
• Reliability asks whether it continues to solve comparable tasks as the horizon and conditions change.
• Progress asks how much verified work exists when full completion is absent.
• Stopping judgment asks whether the agent stops because the task is complete or because it has misread its own state.
• Trajectory health asks whether the action sequence is still producing evidence or has begun to loop and lose coherence.
• System availability asks whether the model and evaluator were actually given a fair chance to run to termination.

These properties interact, but none can substitute for the others. High partial progress is not full completion. Low variance is not reliability when every run fails near the floor. A detected entropy spike is not proof of a model defect.