Checkpoint: What the Score Can—and Cannot—Claim
Coding-agent evaluation now has three connected facts in view: the measured object is a configured system, the benchmark turns work into a verdict through an oracle, and the final percentage compresses both into a summary. The useful judgment is not whether a score is “real,” but whether its scope matches the claim attached to it. This checkpoint turns those facts into a compact way to read a result before comparing the broader task and review benchmarks that follow.
A score has five coordinates
A headline number becomes interpretable when five coordinates remain attached to it.
First is the system. Is the result for a model in a limited generation setting, an agent harness operating tools, or a broader orchestration that decomposes work and routes feedback? The label matters because a model's decisions are mediated by prompts, tools, context, retry behavior, environment handling, and the surrounding control loop. A result for one composition should not be silently attributed to one component.
Second is the task distribution. What did the system have the opportunity to do? A self-contained function, a repository-level issue, a test suite, and a review response are not interchangeable units. The issue description, repository state, language, interaction horizon, and difficulty determine which capabilities are exercised. Even a carefully curated set of tasks estimates performance on that set first; generalization is an additional claim.
Third is the oracle. What evidence made the output count as correct? A hidden test may check a local behavior. A reference-derived test may also encode choices made in the original patch. A behavioral test may state an invariant or expected outcome without requiring one implementation shape. These oracles answer different questions, so their scores should not be treated as though they share one meaning.
Fourth is the protocol. What resources and conditions governed the run? The model and harness label are not enough if the budget, inference effort, environment, tool interface, retries, or infrastructure differ. A fixed protocol can make a comparison reproducible, but it does not remove the need to state what was fixed.
Fifth is the outcome. Was the result a final pass rate, a graded quality measure, a review-alignment rate, a cost-adjusted measure, or a delayed operational signal? The outcome tells us what was observed, not automatically why it occurred or what will happen in production.
These coordinates form a claim boundary. A well-formed statement might say: a particular configured coding system achieved a particular result on a named workload under a specified oracle and resource protocol. That statement can be useful for choosing whether to investigate or deploy the system in a similar setting. It does not by itself establish a model-intrinsic capability, performance on unrepresented work, architectural quality, or production reliability.
The five coordinates are not a bureaucratic preamble. They are the minimum information needed to distinguish a change in the measured system from a change in the measurement. If the task distribution changes, a score change may reflect a new denominator. If the oracle changes, it may reflect a new definition of success.