How a Benchmark Turns Work into a Number
A coding-agent benchmark does not discover a capability in the way a thermometer discovers temperature. It constructs a situation in which a system is asked to act, defines an accepted outcome, and compresses the resulting verdicts into a score. This chapter examines that construction—task, environment, candidate output, oracle, and aggregation—so readers can tell which part of an engineering goal a number represents and where its claim stops.
The measurement chain: from engineering goal to verdict
An open-ended engineering request is too ambiguous to score directly. “Make this library safer,” “improve the parser,” or “clean up the service” could lead to many valid decisions, different amounts of work, and different evidence of completion. A benchmark must narrow the request into an observable task. It supplies a description, a repository state, an environment in which code can be changed and executed, and some rule for deciding whether the result counts.
That narrowing is not a flaw. Every measurement needs an operational definition. The issue is whether the definition preserves the part of the construct that matters. If the intended construct is “can this system resolve a well-specified repository issue under realistic constraints?”, a task with the issue description, repository, dependencies, and regression checks may be a reasonable operationalization. If the intended construct is “can this system perform software engineering across unfamiliar projects?”, the same task may be too narrow, because it does not observe project selection, task decomposition, design discussion, or adaptation to a different environment.
The chain has several transitions. A task setter converts a goal into an instruction and a fixed starting state. The agent receives that instruction through its harness, gathers context, and produces edits or other artifacts. Those actions cross into an execution environment, where the repository and runtime state determine what actually happens. The resulting state is passed to an oracle: tests, a rubric, a reference comparison, or another verifier. Finally, individual verdicts are aggregated into a pass rate or another summary. Every transition can preserve information, discard it, or introduce a new source of variation.
The environment is not a neutral container around the “real” test. It determines which dependencies are present, how commands behave, how much interaction time is available, and whether a failure is attributable to the agent or to setup. A benchmark that carefully builds a reproducible environment has made a valuable design choice. It has also fixed a condition that may be variable in practice. A result then answers how the configured system worked in that environment; it does not automatically answer how the same system will behave when installation, access, or runtime conditions are less controlled.
The denominator is another transition that can disappear in casual reporting. A pass rate over short, self-contained functions is not interchangeable with a pass rate over repository-level issue fixes. A score across one language is not interchangeable with one spanning several languages. A task set selected for clean issue descriptions and runnable dependencies may estimate performance on that selected workload more reliably than performance on messy production work. The numerator and denominator must travel together.
The same caution applies to an agent's output. A benchmark may record only the final patch, not the route by which the agent reached it. That choice makes aggregation simpler, but it hides useful distinctions. A patch that passes after a careful sequence of test-guided corrections and a patch that happens to pass on the first attempt receive the same binary outcome. If cost, time, retries, or intermediate failures matter to the operational decision, the headline score is not enough evidence even when the final verdict is correct.
This is why an end-to-end number should be read as a conditional statement. Its strongest defensible form resembles: under this task distribution, starting state, environment, agent configuration, resource protocol, and oracle, the system achieved this observed result. The statement may be highly useful. It becomes misleading only when the conditions are silently removed and the result is presented as a general property of the model or as a forecast of every engineering workflow.
There is a useful way to locate the information lost at each stage. The task statement loses part of the original goal when it turns a broad request into a fixed issue. The starting state loses part of the surrounding organization when it supplies one repository snapshot instead of a living codebase with changing requirements. The execution trace loses information when only the final patch is retained. The oracle loses information when it tests a few observable consequences rather than the full set of behaviors and constraints that maintainers care about. Aggregation loses information again when all verdicts become one percentage.
None of these reductions is automatically unacceptable. A benchmark needs a manageable task, a stable starting point, an executable environment, and a rule that can be applied consistently. The design question is whether each reduction is aligned with the decision the score is meant to inform. If a team is choosing a system for a narrow, well-specified repair workflow, a fixed repository snapshot and regression suite may be appropriate. If the team is deciding whether an agent can maintain a changing product over time, the same setup is evidence about repair under a snapshot—not about long-term maintenance.
The aggregation step can conceal this mismatch because percentages look comparable even when their units differ. A score can be averaged over tasks that vary in complexity, over tests that vary in importance, or over outcomes that have different costs when missed. Without the task distribution and the aggregation rule, readers cannot tell whether a change in the percentage reflects more difficult work, a different mix of cases, a different notion of failure, or a genuine improvement on the same cases. The number has not become meaningless; its denominator has become invisible.
A disciplined evaluator therefore treats the task and the oracle as a pair. First state the behavior or engineering judgment the task is intended to represent. Then ask whether the oracle observes that behavior across the situations that matter and whether it admits acceptable alternative implementations. Finally, check whether the reported result preserves enough information about case difficulty and failure severity for the intended decision. This sequence does not require a perfect measure. It makes the remaining uncertainty explicit.
The oracle decides what “correct” can mean
The most consequential choice in the chain is often the oracle. The oracle is not merely a final quality check; it is the operational definition of success. A reference-derived test can be an excellent way to verify a tightly specified behavior. It becomes a poor proxy when the reference patch contains incidental decisions that the task never required.
Repository benchmarks in the SWE-Bench family illustrate the tension. The position paper explains that FAIL_TO_PASS and PASS_TO_PASS sets are derived from tests changed in an original pull request. This connects the benchmark to a real issue and a concrete fix, and it gives the evaluator a practical way to check regressions. But the tests can also carry the reference patch's decomposition: which functions exist, where an interface is placed, or which abstraction boundary the original author chose. A system that fixes the underlying behavior through a different but reasonable design may be judged against choices that were never part of the stated requirement.
A reference is a fair gold standard when the specification is tight enough that alternative shapes would violate the task. For example, a compatibility contract may require an exact public interface, or a protocol may require a precisely defined output format. In such cases, matching a particular shape is not incidental; it is part of the behavior being measured. The fairness boundary moves when the task permits refactoring, API redesign, dependency changes, or architectural alternatives. Then the evaluator should distinguish required behavior from properties that merely happened to appear in the reference implementation.
The difference can be seen in a simple thought experiment. Suppose two patches both prevent an invalid input from causing a runtime failure. One adds a guard at the entry point; another reorganizes validation into a shared abstraction used by several callers. A test that invokes the invalid input may accept both. A test that inspects a particular helper or assumes the original call structure may accept only one. If the task required the guard's behavior, the latter test has narrowed the construct from “handles invalid input safely” to “handles it through this implementation route.” That may be deliberate, but it must be stated.
Local functional tests also see only what they exercise. They may establish that a function returns an expected value, while failing to observe duplicated logic, an abstraction that conflicts with the repository, a dependency graph that has become unsound, or a design that makes the next change harder. The position paper calls these design-level properties part of what good agentic software engineering should eventually measure. It does not claim that every benchmark can capture them cheaply. The important distinction is between a property being unmeasured and being absent from the system's quality requirements.
This is where behavioral, multi-shape, property, and differential verifiers become useful. A behavioral verifier states what must happen across relevant inputs or states without requiring one source-code arrangement. A property test checks an invariant rather than one example. A differential test can compare behavior against another implementation or oracle. A multi-shape evaluation can admit several implementations when they satisfy the same contract. These approaches do not eliminate the work of writing a good specification. They move the evaluator's attention from resemblance to a particular patch toward evidence that the intended behavior holds.
The position paper presents this move as a proposed remedy, not a finished universal recipe. The difficult part is operationalization: describing what the system should do in terms a grader can apply without encoding how the agent must do it. A verifier that is too loose may accept a superficial workaround. One that is too tight may reject a sound alternative. A good oracle therefore has two obligations: it must be sensitive to the failure the task is about, and it must avoid treating irrelevant implementation details as requirements.
c-CRAB offers a concrete example of this principle in code review. Instead of asking whether an automated comment resembles a human comment, it converts the human-identified concern into an executable test. The original pull request should fail the test, and a corrected version should pass it. During evaluation, an automated review is given to a coding agent, whose revised patch is then run against that test. The oracle is consequently tied to whether the feedback led to a behaviorally correct correction, not whether the words or location look similar.
That design is more independent of wording, but it is not independent of every choice. c-CRAB's test still represents a human-selected concern, and the final outcome depends on a downstream coding agent being able to act on the review. It measures alignment with a defined review intent, not all useful issues an automated reviewer might find. This is precisely why the example belongs in a discussion of operationalization: a stronger oracle can improve a claim without making the claim universal.
Read the task and oracle as separate evidence
A benchmark comparison is incomplete if it reports the oracle but not the task, or the task but not the oracle. Two systems can receive the same repository issue and be judged by different tests; two benchmarks can use similar issue descriptions while testing very different notions of correctness. A change in either side can move the score.
The task determines the opportunity to demonstrate a capability. Its language, repository scale, issue clarity, interaction horizon, complexity, and available resources shape what the agent must do. The oracle determines which parts of the resulting work are visible to the evaluator. A narrow task with a broad behavioral oracle may test a precise capability well. A broad task with a reference-shaped oracle may still report a number whose meaning is narrower than the work appears.
For readers, the practical habit is to ask two questions before comparing results. What was the agent asked to do? Then: What evidence was allowed to make that work count? Only after both are clear does the percentage become interpretable. A score is not a property waiting to be extracted from the model; it is the endpoint of a chain of design decisions. The next step is to use that chain to state what a result can—and cannot—claim before looking at broader benchmark profiles.