When a Warning Becomes a Vulnerability

CULPA’s value is easiest to judge at the point where analysis leaves the abstract graph and meets real packages. This chapter asks what the study’s numbers actually measure, then follows the arrow-arith case from an inconsistent metadata contract to an unchecked write and a maintainer decision. The aim is a practical distinction: a tool warning, a manually validated bug, a developer-confirmed vulnerability, and a disclosed advisory are different states of evidence.

Read the evaluation in layers

The study scanned the top 1,000 Rust packages from crates.io, selected by download count and by recent update activity before December 1, 2025. That is a meaningful test of ecosystem exposure: popular packages can influence many downstream builds. It is not, however, a random sample of all Rust code, and it does not estimate how common each bug class is across the entire ecosystem.

The scan ran with 16 parallel processes and a ten-minute timeout for each package. It completed in 18 minutes, but not every package reached CULPA’s analysis pipeline. Two hundred seventy packages failed before analysis began. The paper attributes those failures to internal compiler errors, packages that defined macros without producing analyzable code, dependency conflicts, missing analyzable targets, and other causes. The exact breakdown is useful because it separates a tool’s inability to process a package from a conclusion that the package contains no relevant bug. A package that never reaches the detector contributes no evidence about the detector’s performance on that package.

The paper reports timing for the 730 packages that it successfully analyzed. The mean was 9.0 seconds per package. Half finished within 4.9 seconds, 90 percent within 20.2 seconds, 95 percent within 29.4 seconds, and 99 percent within 60 seconds. Those figures support the claim that the analysis can operate at a useful package scale under the study’s conditions. They describe completed analyses, not the 1,000-package population as a whole, and they say nothing by themselves about correctness.

The detection results require another separation. CULPA reported 107 potential zero-day bugs. Manual inspection classified 55 as true positives, 23 as false positives, and 29 as out-of-scope foreign-function cases. The paper calculates 70.5 percent precision from the 55 true positives and 23 false positives—the 78 warnings considered in scope for that calculation. The 29 FFI cases remain important as evidence about scope and triage, but they are not silently counted as either confirmed bugs or ordinary false positives in that denominator.

A reader can therefore ask four different questions. How much of the package set did the tool process? The answer is 730 of 1,000 under the reported setup. How quickly did the completed analyses run? The answer is expressed by the mean and percentiles for those 730 packages. How many candidate findings did the detector surface? The answer is 107. How many survived manual inspection within the stated memory-safety scope? The answer is 55, with 23 in-scope false positives. No one of these answers substitutes for the others.

The 270 pre-analysis failures are not just an operational footnote. They define the population to which the timing and finding counts apply.