How Long-Horizon-Terminal-Bench Turns Work Into Evidence
This chapter treats Long-Horizon-Terminal-Bench (LHTB) as an evaluation system, not simply as a collection of difficult prompts. Its specific question is how a benchmark can preserve the dependencies and messiness of real terminal work while still producing evidence that an agent made meaningful progress. Understanding that boundary helps readers interpret later results: a score is only as informative as the workspace, interaction loop, stress conditions, and verifier that produced it.
The task is a controlled terminal system
An LHTB task begins with an overall goal expressed in natural language. The agent sees that instruction as the specification, then enters a containerized terminal environment containing the relevant files, data, tools, and helper material. The environment is defined consistently enough that different model runs can be compared, but it is not reduced to a toy question-and-answer exchange. The agent must inspect what it has been given, issue commands, edit artifacts, run programs, examine outputs, and decide what to do next.
That arrangement creates three distinct responsibilities.
The workspace owns the starting conditions. It contains the project or data, the available utilities, the initial defects, and the constraints within which the task must be solved. In a software repair task, that may mean a working application with an important cross-cutting defect. In a scientific or multimodal task, it may mean code and artifacts that must be analyzed together. The workspace is not merely a storage location; it defines what evidence is available to the agent and what state can change as work proceeds.
The agent owns the sequence of decisions. It chooses when to inspect, which command to run, what to modify, how to interpret an intermediate result, and whether the current evidence justifies moving on. No single action is expected to solve the entire assignment. The agent has to maintain a plan while learning more about the environment, revise that plan when observations disagree with it, and eventually decide whether to continue or finish.
The verifier owns the final judgment. LHTB tasks include an oracle implementation or simulator that evaluates the resulting environment. The verifier can inspect outputs, files, test results, or responses from a simulation. This division of responsibility matters because a failure can occur at any boundary. The workspace may make a dependency hard to discover. The agent may misunderstand a schema or spend its budget exploring a dead end. The verifier may intentionally test a condition that was not visible in the public examples. Treating all three as one undifferentiated “benchmark” hides where reliability is being tested.
The task lifecycle is consequently iterative rather than linear. An instruction enters once; the agent then cycles through observation, action, execution, and interpretation. An intermediate output is not just a result to read. It becomes part of the state that informs the next action. If an inspection reveals a schema mismatch, the agent may need to trace the mismatch into a transformation, repair the transformation, rerun the pipeline, and inspect a new artifact. If a result is plausible but incomplete, the agent must find out what evidence would distinguish a genuine fix from a visible-case patch.
This is why LHTB starts from complete but deliberately broken terminal-only projects. Asking an agent to provide an abstract solution would test whether it can describe a method. Giving it an environment with code, data, and a failure to localize tests whether it can turn that method into a sequence of state-changing actions. The benchmark's setup resembles professional work more closely because the agent must first discover the shape of the problem. The difficulty is not added by making every command obscure. It comes from making the final outcome depend on several linked decisions.
The suite contains 46 tasks spanning technical and professional settings, including software engineering, research reproduction, scientific computing, multimodal analysis, interactive environments, climate and earth-science work, and other workflows. That variety is useful only because the same underlying interface is preserved: an instruction, a terminal workspace, an agent trajectory, and an evaluator. The benchmark is asking whether an agent can sustain a method across different kinds of artifacts, not whether it has memorized one domain-specific trick.
The environment also gives the task a meaningful stopping boundary. The agent is not asked to narrate indefinitely. It works until it reaches a satisfactory state, calls the available completion action, or exhausts the allowed execution budget. A correct local change that leaves the final artifact unverified is therefore different from a completed task. The interface makes that difference operational: state must cross from the agent's decisions into the workspace, and then cross again into the verifier's evidence.
The sequence shows an important asymmetry. Public checks can provide feedback while the agent is working, but the hidden verifier owns the high-stakes judgment at the end. The agent's path is open-ended; the evaluator's observation is deliberately constrained to objective evidence. LHTB relies on that asymmetry to test whether the agent can infer and implement the underlying requirement rather than optimize only for what it can see.
Robustness lives between visible checks and hidden stress
Public checks serve a practical purpose. They let an agent determine whether a command-line interface behaves, whether an output has a required format, or whether a simple example works. They make the environment navigable and provide feedback during a long run. But the LHTB design gives these checks relatively low reward weight. Passing them is evidence that a basic boundary has been crossed, not evidence that the general problem has been solved.
The hidden stress suite tests a deeper boundary: whether the implementation captures the rule behind the examples. The paper describes cases in which wrappers are nested, data fields are renamed, values are missing, noise is injected, or images are rotated or cropped. These variations matter because a superficial patch often works by assuming the exact shape of the public input. A robust solution must parse the relevant structure, preserve meaning across variation, and generate the expected end-to-end artifact.
Consider the difference between fixing a displayed example and repairing a pipeline. If a visible input always names a time field time, an agent could change one line to read that exact name. A hidden case that uses valid_time asks a different question: did the agent identify the semantic role of the field, or did it memorize the spelling? Similarly, a single clean image can make an image-processing procedure appear correct even if it fails when a frame is rotated, cropped, or noisy. The hidden check is not arbitrary punishment. It is an attempt to move evaluation from surface compliance toward generalization within the task's stated problem.
That design also changes how an agent should use public feedback. A failing public test is useful evidence about a defect. A passing public test is only a local confirmation. It should update the agent's confidence, not terminate the investigation. In a long workflow, the agent has to ask what the visible check does not cover and whether the implementation would survive variations that are natural consequences of the requirement. This is one reason LHTB measures sustained execution: robustness often requires a second cycle of inspection after the obvious error disappears.
The hidden verifier does not evaluate every thought or every command. It evaluates the final container state through objective evidence. This is a strength because deterministic checks reduce the variability of a human or language-model judge. It is also a boundary. A final file can pass a test suite through an accidental shortcut or an overfit condition if the suite is weak. The LHTB paper addresses that risk by placing most reward in hidden stress cases and requiring the official solution to reach full score across the complete suite. Even so, programmatic evaluation proves conformity to the encoded checks; it does not prove that the agent's internal reasoning was sound at every step.
Partial credit follows the workflow, not the transcript
LHTB's second design choice is to decompose the overall goal into semantically meaningful subtasks. Each subtask has its own check and contributes a normalized result to the overall reward. Some checks are binary, such as whether a test suite succeeds or a service responds. Quantitative goals can receive continuous or thresholded scores, such as how closely a reproduced metric matches a reference. In repeated or campaign-style environments, a subtask can aggregate performance over multiple episodes.
The important word is meaningful. A subtask should correspond to an intermediate condition that matters to the intended workflow, not to an arbitrary command count. “The agent issued three commands” would say little about whether a repair occurred. “The pipeline accepts the required schema variations” says something about a real property of the resulting system. Good decomposition makes progress observable without pretending that each step has equal value.
The verifier combines subtask results using weights. Equal weights are the default, but the design allows a final goal to receive more weight when that reflects its criticality. This provides two kinds of information at once. A threshold can answer whether the task counts as resolved. The weighted reward can show how much of the intended workflow was achieved when the threshold was missed. The two outputs serve different decisions and should not be collapsed into one.
There is a subtle but important limit to what partial reward means. It is evidence about the final state, not a percentage of the agent's thoughts, time, or moral effort. If a file exists and a check passes, the evaluator can credit that condition. It cannot infer that every earlier hypothesis was correct. If a subtask receives no credit, that does not prove the agent did nothing useful; it may mean that the required evidence was not present in the state the verifier inspected. The score is valuable precisely because it is anchored to observable conditions, but its interpretation must remain anchored too.
The same principle explains why hidden checks and subtask grading belong together. A benchmark could add many intermediate labels and still be weak if those labels reward visible rituals. Conversely, a single final check can be rigorous but too sparse to distinguish an almost-complete repair from an abandoned attempt. LHTB combines workflow decomposition with stress variation so that partial credit reflects both progress and robustness. An agent receives credit for crossing real intermediate boundaries, but high reward requires crossing them in a way that generalizes across the hidden cases.
This arrangement preserves realism while adding observability. Real engineering work rarely has a clean score after every action; people infer progress from tests, artifacts, and intermediate outputs. LHTB formalizes a selected subset of those signals so that different runs can be compared. It does not reproduce all the judgment involved in professional work, and it cannot guarantee that the chosen subtasks represent every stakeholder priority. It does make the benchmark's priorities explicit enough to inspect.
The system-level consequence is that “the agent failed” becomes an incomplete diagnosis. The workspace defines the problem and the state. The agent changes that state through a long sequence of decisions. The verifier samples the resulting evidence under visible and hidden conditions. A low final score may reflect a failure to form a plan, a brittle implementation, a missing final check, a budget limit, or a mismatch between the agent's stopping decision and the verifier's standard. Later results from LHTB become useful only because this evaluation system preserves those distinctions well enough to measure them.