The Number on the Leaderboard Is Not the System
The subject of this series is the evaluation of coding agents: systems that use language models, tools, repository context, execution environments, and feedback to carry out software work. The opening question is why one benchmark score can be both accurate for its test and misleading as a statement about general engineering ability. The practical value of answering it is simple: before comparing two numbers, readers should be able to identify what was measured, what counted as success, and which parts of the system the number leaves unexplained.
A benchmark score looks like a property of a model because the model's name is often the most visible label. “Model X solved 65% of tasks” sounds like a stable capability statement. But a coding agent is not a model answering in a vacuum. It is a configured arrangement that receives a task, gathers context, edits a repository, runs tools, responds to failures, and submits an outcome to a grader. Change the arrangement and the number can change even when the model name remains the same.
That does not make end-to-end scores worthless. A score produced by a fixed task set, fixed environment, and fixed evaluation procedure can answer a real question: how often did this configured system complete these tasks under these conditions? The problem begins when that bounded result is silently promoted into a broader claim—about the model alone, about every kind of software work, or about performance in a live engineering organization.
A high score can be accurate—and still incomplete
Traditional coding benchmarks were designed around a relatively narrow object: a model generating code for a self-contained problem, usually judged by hidden tests. Repository-level benchmarks made the setting more realistic by asking an agent to inspect an existing codebase, interpret an issue, modify files, run tests, and produce a patch. Yet the headline structure often remained the same: one agent configuration, one environment, one task distribution, and one end-to-end pass rate.
That structure compresses several questions into one outcome. Did the model reason correctly? Did the prompt supply the right project context? Did the tool interface make repository navigation easy or awkward? Did the agent recover from a failed command? Did the environment contain the dependencies and runtime behavior needed to expose the defect? Did the test suite check the requirement or only a convenient consequence of it? A successful run does not separate these causes. A failed run does not identify which one broke.
The distinction matters because practical software engineering is a system activity. A deployed coding workflow may involve an issue being decomposed into tasks, an agent working in an isolated environment, tests and type checks providing fast feedback, reviewers inspecting the patch, and production outcomes arriving much later. The benchmark may touch only one slice of that workflow. Its score is therefore a measurement of the slice and its surrounding protocol, not a complete reading of the agent's place in the whole system.
The position paper that motivates this series names the larger arrangement a system harness. It includes the model and its tool-use loop, but also the task definition, environment, curated context, and feedback mechanisms. That vocabulary changes the question from “How smart is the model?” to “What does this configured system accomplish, under which conditions, and with what evidence that the result is acceptable?”
Here is the basic chain:
The arrows are not merely a diagram of execution. They mark choices that define the meaning of the result. A real engineering goal must be represented by tasks. The system must be given some context and resources. The verifier must decide which behaviors count. The score then summarizes the verdicts. Deployment outcomes—review acceptance, reverts, incidents, customer effects, or code longevity—may provide valuable evidence later, but they are usually outside the benchmark's immediate field of view.
The number moves when the named model does not
The fixed-model comparison in the position paper makes the hidden arrangement visible. On one Terminal-Bench task distribution, the same Claude Opus 4.6 model appears with several agent harnesses. The reported success rates range from 79.8% for ForgeCode to 58.0% for the Anthropic Terminal-Bench entry, a spread of more than 20 percentage points. The comparison is drawn from a reported leaderboard rather than a new experiment by the paper, so it should not be read as a precise decomposition of variance. It does establish the central fact: the model label alone cannot explain the result.
A harness changes how the model encounters the work. It can alter the system prompt, the available tools, the representation of terminal output, the number and timing of retries, the way context is selected, and the way intermediate failures are handled. It can also be better or worse matched to the conventions under which the model was trained or tuned. In some interfaces, the caller can change inference effort, trading time and cost for more deliberate reasoning across tool calls. Infrastructure choices, container allocation, orchestration, and evaluation seeds can add further movement.
These variables do not all represent the same kind of capability. A better context builder may help the system find the relevant code. A more reliable retry loop may prevent a correct idea from being lost to a malformed tool call. A larger budget may permit more investigation. A stronger model may still be important, but the observed outcome is produced by their interaction. Calling the whole result “model performance” hides the very mechanisms an engineering team would need to improve.
The verifier creates a second source of ambiguity. In many repository benchmarks, the tests are derived from the pull request that originally fixed the issue. That is convenient: the benchmark inherits a concrete task, a known patch, and tests that distinguish the pre-fix and post-fix states. But the original patch also embodies one decomposition, one API shape, and one set of implementation decisions. A different patch may preserve the intended behavior through another abstraction and still fail a test that implicitly expects the reference's structure.
This is not an argument against reference patches. When a specification is tight enough that only one shape is reasonable, a reference can be an appropriate gold standard. The concern is about open-ended engineering work, where defining an API, reorganizing abstractions, choosing a dependency, or preserving a project's design conventions may have several acceptable answers. Local hidden tests can show that some observed behavior holds while missing code reuse, architectural fit, dependency health, or maintainability.
The position paper cites secondary analyses that illustrate the danger of treating a passing result as settled correctness. One reported differential-testing study found that 7.8% of patches counted as resolved failed developer-written tests, while 29.6% diverged from the gold patch's runtime behavior. Those figures belong to the cited studies and are not a reanalysis here; their importance is the kind of gap they expose. A benchmark can certify success according to its oracle while leaving a meaningful question about the software itself unanswered.
Three questions before trusting a score
The first question is what system produced the number? The answer should include more than the model family. A useful comparison identifies the agent or orchestration harness, tool interface, context, environment, inference setting, retry or timeout behavior, and resource budget. If those details are missing, a difference may still be real, but its cause is not identified. Two scores may compare complete systems rather than models, and that can be exactly what a practitioner wants—provided it is described honestly.
The second question is what counted as success? A benchmark's task statement and oracle define its construct in practice. Did the agent need to make a failing test pass while preserving existing tests? Did it need to generate a test that rejects plausible incorrect fixes? Was a review judged by wording similarity, location, an LLM's judgment, or a later behaviorally correct change? Were design and repository conventions observable at all? The more a verifier depends on one reference implementation, the narrower the claim about engineering alternatives may be.
The third question is what signal explains the result? An end-to-end score is useful for knowing whether the configured system completed the benchmark workload. It is much less useful for deciding what to repair when performance changes. Component-level measures can ask whether context helped, whether the agent followed an invariant, whether a test suite discriminated among failure modes, or whether a maintenance process improved a scoped health metric. Broader signals such as review rounds, reverts, incidents, and customer feedback arrive later and are confounded by many causes, but they connect evaluation to the outcomes that matter after a patch leaves the sandbox.
These questions do not demand that every benchmark become an enormous evaluation program. They demand that the claim stay attached to its evidence. “This configured system passed this task distribution under this budget and verifier” is a useful statement. “This model is a 70% software engineer” is not, unless additional evidence supports the leap across systems, tasks, or deployment conditions.
The rest of this series follows the same discipline. It first examines the system around the model, then the oracle that turns engineering work into a verdict. From there, two benchmark designs show why breadth and behavioral grounding add different kinds of information. The point is not to replace one leaderboard with another. It is to learn how to read a score as a bounded observation—and how to ask for the next measurement when that observation cannot answer the decision at hand.