What a Coding-Agent Score Actually Measures
Coding-agent evaluation becomes more intelligible when the system around the model is made explicit. This chapter focuses on the boundaries between an agent harness and the wider system harness, and on the feedback loops that connect them. The goal is not to catalogue components, but to give readers a causal model for explaining why the same named model can produce different benchmark results—and for deciding what additional evidence would make a result useful.
The model sits inside two nested harnesses
An agent harness is a language model working toward a task through tools, prompts, and context. It may inspect files, run a command, read the output, edit code, and try again. The harness determines how those actions are presented and sequenced. In ordinary discussion, this is often what people mean by a “coding agent.”
A system harness is the larger orchestration around one or more agent harnesses. It turns a higher-level goal into concrete work, dispatches tasks, manages the environment in which changes occur, and routes outputs through feedback that approximates whether the work is acceptable. The distinction is not a matter of terminology. It identifies two different places where a benchmark result can be shaped.
Consider a request to improve a software product. The system harness might split it into vertical slices, assign each slice to an isolated agent session, and send the resulting pull request to tests, reviewers, or other agents. The individual agent harness then handles the local loop of repository exploration and editing. A benchmark that starts with a prewritten issue and ends with a patch may exercise the local loop while saying little about decomposition, coordination, or the decision to send the patch back for revision.
The position paper describes five recurring parts of this broader arrangement. The first is the task: the unit of work derived from a larger goal. The second is the agent harness: a configurable executor made from a model, prompt, tools, and action loop. The third is the environment: the repository and runtime under change, together with services such as the issue tracker, continuous integration, or deployment surface. The fourth is context: a curated projection of the environment and of material created by the harness, such as specifications, project guidance, or skills. The fifth is feedback: anything the harness reads to refine the current solution or to refine itself.
These parts have different relationships to state and authority. The environment is where the repository and runtime state actually change. An agent can propose an edit, but the environment determines whether that edit applies, compiles, runs, or affects a test. Context is not a second repository; it is a selected view of the project and its rules. The harness controls what is placed in that view, so omissions in context can be a system-design failure even when the model could have reasoned correctly with the missing information. Feedback travels back across the boundary: a test result, review comment, or production incident becomes information that can alter the next action or the harness itself.
A model remains central, but it is not the sole owner of the outcome. It generates decisions inside an interface. The agent harness decides which observations become available and how tool calls are executed. The system harness may decide how work is decomposed and which checks block progress. The environment supplies the behavior that those checks observe. A score produced after all of these interactions belongs, at minimum, to the composition.
The nested relationships can be represented like this:
The important interface is not just the arrow from model to answer. The task crosses from orchestration into an agent session; actions cross from the session into the environment; observations cross back as feedback; and the system harness decides how that feedback is used. If a benchmark fixes most of these interfaces, its score may be perfectly reproducible while still measuring a narrow configuration of the system.
Ownership also makes hidden protocol choices easier to see. Imagine two evaluations that present the same repository issue. In one, the context already includes the project conventions the agent needs; in the other, the agent must discover them from the repository. In one, the environment has a dependable dependency installation and a clear test command; in the other, setup failures consume the available interaction budget. In one, a failed tool call can trigger a retry; in the other, the current patch is submitted immediately. The final verdict may be recorded as the same kind of failure, but the corrective action is different: improve context selection, repair the environment, change the interface, or change the model's reasoning. The score alone cannot choose among those diagnoses.
The boundaries can also move between benchmark families. A task may be supplied as a carefully written issue rather than generated by a system that first decomposes a product goal. A benchmark may provide a verifier that is always available, while a deployed workflow must decide which checks to run and how to react when they disagree. A result can therefore be reproducible without being portable. Reproducibility means that the stated setup can be rerun; portability asks whether the measured behavior survives different repositories, contexts, resources, or orchestration choices. Those are related properties, not synonyms.
This also explains why model–harness fit matters. A model may have learned or been tuned around particular conventions for tool use, context, or reasoning effort. A harness that presents the same underlying repository through a different interface can make the model more or less effective. The position paper cites a fixed-model Terminal-Bench comparison in which success rates vary by more than 20 percentage points across harnesses, and it discusses additional movement from retries, orchestration, container allocation, evaluation seeds, and infrastructure. The exact contribution of each factor is not isolated by that comparison. The broader conclusion is more modest and more useful: the model name does not identify the whole measured system.
Feedback turns a run into an improvement loop
The system-harness model matters most when feedback is treated as a structured flow rather than as a single green or red light. The position paper distinguishes three feedback tiers by scope, latency, and trust.
Inner-loop signals arrive within seconds or minutes. Tests, type checks, linters, and compilation tell an agent whether the current change is executable, well-typed, policy-compliant, or locally worth another iteration. They are fast and comparatively cheap, but narrow. A passing test can confirm a behavior it was written to observe; it cannot by itself establish that the architecture fits the project or that the patch will survive a maintainer's review.
Middle-loop signals take minutes to hours and cover a wider slice of work. Reviewer requests, simulations, maintenance agents, score rubrics, and recurring trace analysis can expose duplicated abstractions, drift from project conventions, repeated inefficiencies, or quality problems that a unit test does not express. These signals are useful precisely because they aggregate across a change or a set of related changes. They may be judgment-dependent, so they are not all suitable as hard gates.
Outer-loop signals arrive over days or weeks, after a change encounters human review, production, or users. Pull-request acceptance, revert rate, incidents, customer feedback, and code longevity are closer to the outcomes a software organization ultimately cares about. They are also delayed and confounded. A revert may reflect a product decision, a change in requirements, or a problem in the patch; customer feedback rarely points cleanly to one prompt or task. Outer-loop evidence is therefore valuable for calibrating faster proxies, not a simple replacement for them.
A verifier is the strict subset of feedback that returns a pass/fail verdict suitable for blocking: a test, type check, linter, or binary rubric. The broader category includes qualitative review and outcome signals that inform rather than gate. Keeping the terms separate prevents a common mistake. If every observation is turned into a binary score, the evaluation may become easier to aggregate while losing information about why the system should change.
These tiers also clarify ownership. The agent can often respond directly to an inner-loop result: inspect the failure, edit, and rerun. The system harness may schedule a middle-loop review, decide whether the work needs a new task, or update a rule that will guide later sessions. An outer-loop result may trigger a change to task decomposition, context, or verification policy, but not necessarily a local patch. The feedback is connected, yet the response belongs to different layers.
A single end-to-end benchmark score flattens this structure. Suppose the final patch fails. The failure may come from the model's reasoning, an omitted project rule in context, a tool interface that obscured an error, a faulty task decomposition, an environment with the wrong dependency, a verifier that did not represent the requirement, or premature termination after a failed call. The score tells us that the configured run did not reach the benchmark's accepted state. It does not tell us which boundary failed.
That is the difference between realism and diagnosticity. An end-to-end run is realistic when it exercises the system in much the way it will be used. It is diagnostic when its output helps locate the cause of success or failure. Integration tests in conventional software have the same property: they reflect how components work together, but a failing integration test does not automatically identify the broken component. Unit tests sacrifice some deployment realism to make local behavior easier to inspect. Agent evaluation needs both kinds of evidence.
The position paper therefore argues for evaluating system components separately as well as in composition. A context artifact can be tested for the lift it provides under a fixed agent. A verifier can be examined for whether it catches meaningful failure modes rather than merely matching a reference patch. A maintenance process can report before-and-after changes in scoped metrics such as complexity, duplication, dead code, dependency cycles, or churn hotspots. These proposals do not form a complete validated standard; they are ways to recover signal that a final pass rate cannot supply.
Reporting metadata is the minimum version of the same idea. A useful benchmark submission should identify the model, agent-harness version, environment or infrastructure configuration, and dataset version. The paper also recommends at least one ablation along a non-model axis against a fixed baseline. Such an ablation does not magically establish causality: changing a harness component can interact with the model, task, and budget. It does, however, turn an unexamined attribution into a more informative comparison.
The practical judgment is consequently two-part. If the question is whether a configured system can complete a particular workload under a fixed protocol, an end-to-end score is the right kind of evidence. If the question is why the system performs as it does or how to improve it, the same score is insufficient. We need to know which context was supplied, what the environment allowed, which feedback was available, and where the failure appeared in the loop.
This boundary is especially important as agentic systems become longer-running and more orchestrated. The more components participate, the less defensible it becomes to attribute the final result to one component by default. The next issue is therefore not merely who owns the score, but how the benchmark's definition of success narrows the claim. A system can be measured faithfully against an oracle that still observes only one implementation or one slice of the intended engineering behavior.