Verify Before You Trust the Best Candidate

Once an AI programming agent can generate several plausible patches, reliability changes from a generation problem into a selection problem. A pool may contain a correct candidate even when the first attempt fails, but the system still needs to identify which candidate to submit without access to the final gold tests. This chapter compares two inference-time verifiers—one that executes synthesized tests and one that judges trajectories without execution—so readers can distinguish generation capacity from verification capacity and understand why no single signal wins everywhere.

The distinction is captured by two related measures. Pass@K asks whether at least one correct solution exists among K sampled candidates. Best@K asks whether the verifier actually selects a correct one. The gap is not a minor accounting detail. It measures the value of the selection mechanism, and it creates a new failure surface: a verifier can reject a correct patch, prefer an incorrect patch, or fail to distinguish candidates that look different to a human.

The candidate pool is produced before the verifier makes its choice, so the two stages should not be credited for the same success. If every sampled patch is wrong, no ranking method can recover the task. If a correct patch is present but the verifier selects a weaker one, the generation stage created potential that the selection stage failed to realize. The reported experiments hold a 26-candidate pool fixed while comparing verifier designs, which isolates the scoring question: under the same opportunities, which evidence channel chooses more of the candidates that actually pass the held-out correctness condition?

This separation also changes how extra compute should be spent. More editor rollouts may increase the chance of including a correct patch. More testing-agent rollouts may increase the chance of generating a distinguishing test. More judge capacity may improve ranking resolution. These are different investments. Calling all of them “test-time scaling” can hide whether a system improved by creating better candidates, producing better evidence, or simply spending more resources on selection.

Two ways to interrogate a candidate pool

The execution-based verifier treats the candidate patch as a program that should be run. A specialized testing agent receives the issue and repository state, explores the code, and synthesizes reproduction tests intended to cover corner cases, input types, and boundary behavior. The verifier then runs those tests against each candidate. Existing regression tests can first filter out patches that break behavior already preserved by the repository; among the survivors, the number of generated tests that pass supplies the execution-based score.

This is the most direct kind of evidence available at inference time. The candidate either passes a runnable test or it does not. The verifier does not have to infer correctness from how confident the agent sounded or how orderly its trajectory appeared. It can expose a concrete mismatch between the patch and an observed behavior, which is exactly the kind of signal that made repository environments useful during repair and training.

But directness does not guarantee resolution. If several patches pass the same generated tests, they receive the same integer pass count. The verifier has no further information with which to separate them. A test can also be valid yet uninformative: it may not reproduce the issue, may fail against the correct patch, or may exercise a path that every candidate handles equally. In the reported analysis, most generated tests provided little discriminative signal across the candidate pool. Execution is grounded, but the ground may be too flat for ranking.

There is a more dangerous case than a tie. A toxic test favors an incorrect patch over a correct one.