Overview
This series asks why agents that solve short tasks still fail at long, stateful work. LHTB exposes the gap with terminal workflows, dense subtask reward, hidden stress checks, and termination causes; Beyond pass@1 extends it across duration, domain, repeated attempts, partial completion, and trajectory behavior. Together they support task-matched, system-level evaluation—not a universal score—while keeping benchmark, verifier, infrastructure, and metric limits explicit.
Learning outcomes
- Short-task pass@1 shows capability, not whether long workflows remain reliable.
- Dense, verifier-grounded partial reward exposes meaningful progress hidden by endpoint-only grading.
- Reliability depends on horizon, domain, task structure, repetition, and the model–scaffold system.
- Timeouts, false finishes, trajectory disorder, and outages need different diagnoses and recovery controls.
Keywords
Preview every chapter · Unlock full text, audio, and video from chapter 4
PlansFAQ
Why is pass@1 not enough for long-horizon agents?
Pass@1 is not enough because it records one attempt’s final success and hides duration effects, repeated variation, partial progress, and termination cause.
What does dense reward add to long-horizon agent evaluation?
Dense reward adds evidence about verified intermediate subtasks, distinguishing near-complete work from runs that never establish useful progress; it does not make incomplete artifacts safe.
Why measure agent reliability by domain and duration?
Reliability should be measured by domain and duration because human-time length does not capture agent-relevant steps, dependencies, feedback, or state burden.
How should evaluators interpret incomplete or interrupted runs?
Evaluators should separate verified progress from termination cause, stopping judgment, and infrastructure availability, then assess recovery from durable checkpoints rather than treating every failure alike.