Beyond Pass Rates: Building a Useful Evaluation Portfolio
The evidence from task-diverse and behaviorally grounded benchmarks points to a practical problem: an engineering decision rarely asks only whether an agent can pass a particular test set. It asks whether the system can do a specified kind of work, at an acceptable cost, with a tolerable failure mode, and with enough signal to improve when it falls short. This chapter turns that problem into a layered evaluation portfolio, showing what each layer adds and why the layers should be matched to the decision rather than averaged into a universal score.
Start with the decision, not the leaderboard
A portfolio begins with the decision it must support. “Should this system help repair issues?” is different from “Can it generate tests that catch plausible wrong fixes?” “Can it review pull requests?” is different from “Can it replace human review?” The wording determines the construct, and the construct determines which task, oracle, and operational evidence are relevant.
A useful portfolio has an end-to-end layer, but it does not stop there. The end-to-end result answers whether the configured system reached an accepted state on the selected workload. It preserves the interactions among model, harness, context, environment, tools, retries, and verifier that a deployed workflow will experience. This is the closest benchmark layer to the complete system, and it should remain visible rather than being replaced by isolated component tests.
The second layer is a set of capability slices. Split the workload by the task the system performs, the language or repository context, and—when evidence supports it—the complexity or interaction horizon. OmniCode demonstrates why this matters: bug fixing, test generation, review response, and style fixing produce different results, and their correlations are imperfect. A task slice does not tell us why a system succeeds, but it tells us whether a score is being carried by one activity while another remains weak.
The third layer is the oracle layer. Align the verifier with the behavior being claimed. A bug-fixing result needs checks for the requested behavior and regression preservation. A test-generation result needs a discriminating criterion that rejects plausible incorrect patches, not only a known correct patch. A review result needs evidence that the feedback leads to a correction of the underlying concern, not merely similar wording. The oracle should be as independent as the construct requires while still being feasible to run.
The fourth layer records the operating protocol. Name the model, harness, context, environment, dataset, budget, inference effort, timeout, and retry conditions that materially shape the run. The purpose is not to force every benchmark into one standardized configuration. It is to make clear which comparison is being made. If the question is whether a harness improves a fixed model, hold the model and relevant task distribution steady. If the question is whether a model transfers across harnesses, vary the harness deliberately and report the interaction rather than hiding it.
The fifth layer is component diagnosis. A context builder can be evaluated for the lift it provides under a fixed system. A verifier can be checked for whether it catches meaningful failure modes. A task-decomposition policy can be compared with another policy while the downstream agent is held as constant as practical. These measurements are not a replacement for end-to-end success; they explain where to look when end-to-end success changes.