How AIxCC Turned Security Work into Evidence
AIxCC’s competition design is the focus here: it defined several moments in an ordinary software workflow and asked autonomous cyber reasoning systems to produce specific, judgeable evidence at each one. Understanding that contract lets us distinguish a full codebase search from a pull-request investigation, a suspicion from a proof, and a fast answer from a useful security result.
The design choice was more consequential than it may first appear. “Scan the code for vulnerabilities” sounds like one task, but the question changes with the event that starts the scan. A new release asks whether anything in the full project needs attention. A pull request asks whether a limited change introduces a problem. A static-analysis alert asks whether an existing report is valid. A collection of findings asks whether separate pieces of evidence describe the same underlying vulnerability. AIxCC made those differences operational rather than leaving them as vague variations on an agent prompt.
Four moments, four questions
The final competition connected its systems to a GitHub-centered development workflow. The first trigger was a new release. At that point, a CRS could conduct a full scan, searching across the project rather than assuming that the relevant code was already known. Full-scan autonomy has breadth as its defining difficulty: the system must decide where to spend attention, find useful inputs, and keep enough evidence to distinguish a genuine issue from the many ordinary failures that occur in a large software project.
The second trigger was a pull request, which initiated a delta scan. Here the input was not simply “the repository,” but a change and the code around it. A delta scan narrows the search space, yet it does not reduce the reasoning problem to reading the changed lines. A modified function may affect callers, data formats, error paths, or assumptions elsewhere in the project. The advantage is that the change provides a directional clue. The cost is that the system must decide how far beyond the diff to follow that clue.
This distinction affects both efficiency and evidence. A full scan can discover a problem whose location is not known in advance, but it must manage a much larger search. A delta scan can focus fuzzing or code reasoning on a smaller region, but it can miss an issue if the system treats the diff as a hard boundary. The competition therefore did not make “full” and “delta” two names for the same procedure. They were different operating modes with different starting information and different failure risks.
The two modes also imply different ownership of context. In a full scan, the CRS has to create a useful search agenda from the project and its harnesses. In a delta scan, the development system has already supplied a hypothesis about where change occurred, but the CRS owns the decision about whether that hypothesis is security-relevant and how much surrounding code to inspect. This is a general pattern in autonomous systems: narrowing the input does not eliminate judgment; it moves judgment from broad discovery toward scope selection and impact analysis.
The third trigger was a security alert in SARIF, the standard format used by static-analysis tools to describe a potential vulnerability. SARIF review asked the CRS to classify the alert as valid or invalid. This is a triage problem, not a fresh search over the entire project. The system receives a proposed location and explanation, then has to examine enough code and execution context to decide whether the warning corresponds to a real security issue. A good triage result can save a maintainer from investigating a false alarm; a bad one can either waste time or cause a real issue to be dismissed.
The fourth capability was report synthesis. After the systems had generated findings, they had to correlate them into per-vulnerability reports. This is where the identity of a finding becomes important. Several inputs can trigger the same bug, and several reports can refer to different bugs in the same function. A system that cannot preserve those distinctions may appear productive while making the maintainer’s work harder. Synthesis is therefore not decorative summarization. It is the act of deciding which evidence belongs together and which claims can legitimately be connected.
These four moments create a useful map of autonomy. Full scanning tests breadth. Delta scanning tests targeted reasoning and the ability to follow effects beyond a code change. SARIF review tests judgment about an external claim. Report synthesis tests whether the system can preserve identity and relationships across its own work. None is a complete measure of security ability, but together they define a more realistic operating surface than a single prompt to “find bugs.”
The workflow was also event-driven rather than a single batch request. Organizers could dispatch full or delta challenges, sometimes alongside SARIF broadcasts. The CRS then had to manage evidence arriving from different paths and decide how those paths should inform one another. A static alert might guide proof generation; a fuzzer-found crash might help assess a static report; a patch could connect both. The design left the systems room to combine signals, but it made the final responsibility clear: submitted artifacts had to retain the relationships that made them meaningful.
The diagram also shows an important ownership boundary. The development workflow supplies the event and, in some cases, a narrowed input. The CRS owns the analysis and the decision to produce evidence. The evaluation system owns the consequences of what is submitted. A pull request does not become a validated fix merely because it triggered an analysis, and a CRS does not control the value of its output after it has submitted an incorrect or late artifact.
Artifacts, incentives, and the limits of a score
AIxCC’s scoring rules gave those operating modes a common evidence vocabulary. A proof of vulnerability, or PoV, was an input that triggered abnormal behavior such as a crash. It converted a claim about a flaw into an executable demonstration. That matters because a static warning or an agent’s explanation can be plausible without showing that the alleged path is reachable under the project’s actual runtime conditions.
A PoV is still not a repair. A patch had to resolve the vulnerability while preserving functionality. This made it a stronger artifact and a more demanding one. The system had to identify what to change, produce a usable edit, and supply enough validation for the result to count. In the competition’s scoring, a patch was worth more than a PoV because it could save a developer the work of designing and implementing a fix. The distinction also reveals a practical limit: generating a patch-shaped text is not the same as showing that the underlying defect has been removed.
A SARIF assessment was a judgment about whether a static-analysis report was valid. Its value came from reducing triage effort, but the evidence available to make that judgment differed from the evidence produced by a fuzzer. The CRS might have a reported location, a description, and a weakness classification, yet still need to inspect data flow or runtime behavior before deciding. A correct “invalid” judgment prevents wasted investigation; an incorrect one creates a false sense of cleanliness.
A bundle connected related evidence into a coherent vulnerability report. It could link a PoV with a patch, a PoV with a SARIF assessment, or a patch with a SARIF report. The relationship was valuable because it told the evaluator—and, by extension, a developer—which artifacts referred to the same issue. But the risk was asymmetric: an incorrect pairing could make an otherwise useful set of artifacts misleading. AIxCC therefore allowed a bundle to earn a substantial positive score for correct linkage and a substantial negative score for an incorrect one.
The scoring ranges made those priorities explicit. In the final rules, PoVs could receive one or two points, patches three or six, and SARIF assessments a half or one point, depending on the relevant challenge. Bundle scores could range from a large penalty to a comparably large reward. The exact values mattered less than the ordering: direct repair was valued above proof, proof above a judgment on an existing alert, and correct relationships could be worth more than merely producing another isolated artifact. That ordering encoded an estimate of how much work each result removed from a developer’s queue.
These artifacts are best understood as a chain rather than a menu. A PoV provides runtime evidence that a behavior can be triggered. A root-cause analysis, whether explicit or embedded in the patching process, interprets that behavior. A patch proposes a change. Build checks, PoV replay, and project tests provide layers of validation. A bundle then connects the validated pieces to a single report. At every transition, information can be lost or misinterpreted. The scoring system made some of those losses visible, but the underlying dependency exists whether or not a contest assigns points to it.
The chain also creates a deduplication problem. Multiple inputs may reproduce one vulnerability, so sending a separate patch or report for every input can waste time and confuse the relationship among artifacts. Conversely, grouping too aggressively can collapse distinct flaws into one claim. The contest’s bundle and accuracy incentives pushed teams toward a middle position: preserve enough identity to avoid duplicate work, but do not infer a relationship merely because two failures look similar. That is a systems problem involving state and evidence, not a formatting problem at the end of the run.
Timing added another dimension. AIxCC used time decay so that a correct submission made early in a challenge received more value than the same result made at the last minute. That rule represented a real operational preference: a maintainer can act on an accurate result sooner, and an autonomous system that discovers a flaw but cannot finish before the useful window closes has delivered less value. Yet speed could not be separated from correctness. A fast wrong answer is not a fast security improvement.
The accuracy multiplier formalized that trade-off. A CRS could explore many hypotheses, but its per-challenge result was adjusted by how accurate its submissions were. This discouraged a strategy of sending every plausible candidate and asking the evaluator to sort them out. In an open-source project, that strategy would transfer the system’s uncertainty directly to maintainers. The competition made the cost explicit: an agent had to decide not only what it believed, but when its evidence was strong enough to submit.
This is why the AIxCC score is more informative than a raw count of generated findings. It reflects several mediated outcomes: a technique must reach a candidate, a candidate must become the right kind of evidence, the evidence must be linked to the right repair or report, and the submission must arrive with acceptable accuracy and timing. A system can be strong at one stage and still score poorly if a later stage drops or corrupts the result.
At the same time, the score has a clear boundary. The contest’s weights are proxies for developer effort, not a complete theory of production safety. They do not fully measure whether a patch is maintainable over time, whether its behavior is secure under inputs not represented by the PoV, how a project maintainer would review the change, or how a system behaves when its cloud environment disappears. The scoring contract rewards evidence that can be evaluated within the competition; it does not settle every question required before a patch enters a critical codebase.
There is a second boundary in the word “correct.” Competition correctness can mean that an artifact matches the evaluator’s challenge data and rules. Production correctness may additionally require a complete root-cause repair, acceptable performance, compatibility with project policy, a reviewable diff, and responsible disclosure. AIxCC’s contract could make some of those properties visible through PoV replay, tests, and linkage, but it could not compress every maintainer decision into a point value. A score is consequently a measurement of performance under a defined interface, not a certificate that a repository should merge every winning patch.
This boundary is not a flaw in the design so much as a condition for interpreting it. A competition needs observable outputs and consistent rules. AIxCC made a serious attempt to evaluate more than model fluency by assigning value to proofs, repairs, relationships, timing, and accuracy. Its results should therefore be read as evidence about autonomous systems operating under a particular contract. They should not be read as a direct measurement of a model’s general ability to secure software.
The contract now gives us a basis for comparing the finalists. They all had to produce recognizable evidence, but they did not build the same route from a development event to that evidence. The next question is how they arranged fuzzing, analysis, language-model reasoning, patching, and validation—and what those arrangements reveal about the trade-offs inside a CRS.