The Contract Is the Feedback Loop

The live benchmark, the repository harness, and the performance workload described so far are not three unrelated testing tricks. They are three ways of instantiating the same behavior-to-signal loop for AI programming agents. This checkpoint connects them so readers can make a practical judgment: what behavior is specified, how is it observed, what decision does the signal support, and what important behavior remains outside the measurement?

The value of that judgment is greater than remembering the names of individual benchmarks. It lets a reader inspect a new agent evaluation—or a proposed training environment—and ask whether its executable contract is valid for the claim being made. The common structure is simple. Intent defines a target; a candidate acts; an executable artifact observes selected consequences; a controller turns those observations into a score, filter, repair signal, or review decision. The engineering difficulty lies in making each connection faithful.

The same loop under different operating conditions

Consider the three cases as changes to the boundary around the candidate. A live algorithmic benchmark controls the problem distribution and release date, then uses a reference solution and tests to observe correctness across several tasks. Its distinctive concern is whether the evaluation is fresh enough to measure capability rather than familiarity, and whether the scenarios expose more than direct code generation.

A repository harness controls a different boundary. It preserves a function’s surrounding dependencies, setup, and context, then compares a candidate with the original implementation on selected executions. Its question is not simply “can the agent write a plausible function?” It is “can the agent change this function inside this environment while preserving the behavior that the environment makes observable?” The harness also changes the agent’s interaction: a failure can become feedback for another repair attempt.

A performance workload changes the target again. Functional equivalence becomes a condition, not the whole objective. The candidate must preserve the measured behavior while improving a quantitative property relative to an expert target. The benchmark can then expose a capability gap that a bug-fixing score would hide: locating a bottleneck, choosing the right abstraction level, making a low-level change, and verifying that the improvement is real rather than specific to one input pattern.

The shared loop can be represented without pretending that the three cases are identical:

flowchart LR I["Intent and task context"] --> C["Candidate behavior"] C --> O["Executable observation"] O --> Q["Signal calibration and blind-spot check"] Q --> D["Score, repair, filter, or review"] D --> C I --> Q O --> U["Unmeasured behavior"]

The arrow from observation to decision is the point at which evaluation becomes system design. A test result is not self-interpreting. The evaluator decides whether one failed case rejects the candidate, whether several timing measurements are aggregated into a relative target, whether an error is exposed to the agent for repair, or whether the result is only recorded for later analysis. The same observed event can support different decisions in different roles.

This also explains why a benchmark score cannot be detached from its construction. LiveCodeBench’s release dates address contamination; they do not turn competition problems into repository engineering. R2E’s context and equivalence harness address interfaces and dependencies; they do not measure performance optimization. GSO’s quantitative target addresses runtime improvement; it does not settle maintainability, memory use, or idiomaticity.