What the Scoreboard Hides
The AIxCC scoreboard is most useful when read as a diagnosis of sustained system behavior, not as a simple ranking of who had the smartest vulnerability technique. The focus here is the gap between technical potential and visible performance: stability, accuracy, resource use, and submission strategy determined how much of each finalist’s work reached the score. That distinction lets us interpret the results without turning an outage into a capability judgment or a high score into a universal guarantee.
The final ranking was striking. ATLANTIS finished first with 392.8 points, well ahead of BUTTERCUP at 219.4 and ROBO DUCK at 210.7. Those numbers establish that ATLANTIS was the most effective integrated system under the competition’s rules. They do not, by themselves, explain whether the gap came from better discovery, better patching, more reliable orchestration, more disciplined submissions, or some combination. The post-competition analysis is valuable because it looks beneath the total.
The largest gap was continuity
The final evaluation lasted roughly 143 hours across seven phases. That duration changed the meaning of competence. A CRS did not merely need to solve a case once; it needed to keep analyzing as new challenge projects arrived, preserve its state, and continue producing acceptable evidence. Every additional phase was an opportunity to show capability, but also an opportunity for a build process, scheduler, disk allocation, or service dependency to fail.
ATLANTIS sustained activity across all seven phases. BUTTERCUP and ROBO DUCK were competitive early but plateaued after several phases, while LACROSSE was largely inactive after its early participation. If two systems have similar ability to solve an individual vulnerability but only one remains operational, the long-running evaluation will separate them sharply. The score then measures a property that is easy to understate in a model-centered discussion: the ability to keep a complex pipeline alive.
The SoK analysis identifies several plausible operating causes. A wireshark challenge produced roughly 20.9 GB across 47 harnesses, exposing systems to heavy disk and memory demands. The authors suggest that this environment likely contributed to the plateaus of BUTTERCUP and ROBO DUCK, although the available telemetry does not establish every causal link. ROBO DUCK also had a challenge-cleanup issue in its telemetry. LACROSSE’s postmortem identifies an unrecoverable out-of-memory crash in its master scheduler, after which the system’s only output was more than 1,200 PoVs against one challenge-project vulnerability. That is not a small decline in quality. It is a loss of the system’s ability to allocate attention coherently.
Other systems also encountered operational failures. BUG BUSTER’s submission bug cost a substantial amount of patch scoring. ARTIPHISHELL reported issues including an incorrect language-model budget configuration. ATLANTIS silently failed on all Java challenges involving one project, even though its broader system remained active. These cases matter because they resist a neat distinction between “the algorithm worked” and “the infrastructure failed.” The algorithm is part of a system whose output is conditional on the infrastructure executing the right work against the right target and preserving the result.