Why Duration Is Only Half the Difficulty

The full-study results in Beyond pass@1 show a real decline in long-horizon reliability, but they also show why “longer” is not a complete explanation. This chapter reads the reported curves by domain and by metric, then follows the models whose rankings change when the horizon expands. The practical judgment is specific: select an agent for the duration and structure of the work it will actually perform, not for its short-task rank alone.

The aggregate curve is real, but its slope belongs to the domain

Across the study's ReAct results, mean pass@1 falls from 76.3% on short tasks to 59.8% on medium tasks, 50.5% on long tasks, and 52.1% on very-long tasks. The last step rises slightly, but the net change from short to very long is a 24.3-percentage-point decline. The paper reports that every model has a net decline across the full range. That is strong evidence, within this task suite, that extending the horizon exposes failure that short evaluation misses.

The corresponding GDS curve is less severe in absolute terms: the mean falls from 0.81 to 0.66, 0.61, and 0.59 across the same four buckets. The difference between GDS and pass@1 is important. Pass@1 records whether the entire task reached the exact full-success state. GDS preserves credit for weighted subtasks that were completed even when the final condition was not. As the horizon grows, those two views separate: an agent may continue to produce correct intermediate artifacts while becoming less likely to assemble every requirement into one finished result.

The aggregate curve should not be read as a law that fails at a particular number of minutes. The domain breakdown is more informative. Software-engineering GDS declines from 0.90 on short tasks to 0.59 at medium, 0.57 at long, and 0.44 at very long. Web-research GDS falls from 0.80 to 0.72, 0.59, and 0.63. Document-processing GDS moves from 0.74 to 0.69, 0.66, and 0.71, a nearly flat short-to-very-long change compared with software engineering.

The contrast is not a contradiction in the duration hypothesis. It shows that the hypothesis needs a second variable. Software engineering often requires a chain of interacting edits whose correctness depends on preserving behavior across components. The agent must keep track of assumptions, run checks, and repair regressions that can appear several decisions after the original change. A document-processing task can take a human substantial time because of volume or careful reading while remaining relatively direct for an agent once schemas and transformations are identified. Web research sits between those cases in the reported study: it combines multi-step synthesis with live or changing information, but its failure pattern is not identical to software repair.

These are interpretations of the task structure, not experimentally isolated causes. The paper also notes that human-time duration and agent tool-call complexity can diverge. A task pool can therefore make a longer bucket more tractable for an agent than a shorter one if its operations are more structured. That is how a non-monotone point, such as the document-processing recovery at very long duration, should be read: as evidence that task composition and agent-relevant steps qualify the duration proxy, not as evidence that the horizon stops mattering.