How to Read—and Build—an Evaluation That Matters
The subject of this series is not a contest between model names; it is the measurement of coding agents as systems that act on software under defined conditions. The central question is why one benchmark score cannot carry the whole claim of coding ability, and the answer is now precise: a score belongs to a configured system, a particular workload, a particular oracle, and a particular resource protocol. The practical value of that answer is a disciplined way to read results, choose complementary evidence, and stop a conclusion at the boundary the evidence can support.
Why one number is not enough
A benchmark percentage is an observation, not an essence. It records how often a configured system reached the evaluator's accepted state on a selected task distribution. That observation can be valuable. It can compare two complete systems under a common protocol, reveal that a task is difficult, or show that a change improved a workflow. What it cannot do by itself is identify the model's general ability across other harnesses, tasks, languages, environments, budgets, or definitions of correctness.
The distinction follows from the system around the model. A coding agent acts through prompts, tools, context, an action loop, and an execution environment. A broader system harness may also decompose goals, dispatch tasks, route feedback, and coordinate multiple agent sessions. The model generates decisions within those boundaries; it does not control all of them. A test result crosses back from the environment as feedback, and a verifier turns selected behavior into a verdict. The headline number is produced only after those interactions have already shaped the opportunity to succeed.
The fixed-model Terminal-Bench comparison makes this visible. The position paper reports the same Claude Opus 4.6 model appearing with harnesses whose success rates range from 79.8% to 58.0% on one task distribution. That spread does not prove the exact causal contribution of every prompt, tool, retry policy, or infrastructure choice. It does show that the model label cannot identify the measured object. A statement that omits the harness and protocol is not necessarily false; it is incomplete in a way that matters whenever the result is used to select or improve a system.
The oracle creates a second boundary. A reference-derived test may be exactly right for a tightly specified interface or protocol. It becomes narrower when it encodes the decomposition, API shape, or abstraction choice of one original patch even though the task permits alternatives. Passing such a test establishes compatibility with the tested consequences and perhaps with incidental reference assumptions. It does not automatically establish architectural fit, code reuse, maintainability, dependency health, or the quality of a different solution that the oracle never observes.
This is why the strongest answer to the title question is not “benchmarks are unreliable.” It is “benchmarks are conditional measurements.” Their reliability must be judged relative to their construct. The relevant questions are whether the task represents the work, whether the oracle represents the intended behavior, whether the system and resource protocol are known, and whether the result is being used for a decision that resembles the measured condition.
The missing component is diagnosticity. An end-to-end pass rate can tell a team that the system did not reach the accepted state; it usually cannot tell whether the failure came from reasoning, context, task decomposition, tool use, environment setup, verifier design, or premature termination. The position paper therefore proposes evaluating components separately as well as in composition, much as software teams use unit and integration tests together. This proposal is not a completed universal standard, but it identifies the information a single end-to-end number discards.