What the Numbers Actually Measure
The empirical subject here is the security-smell evidence from agent-assisted coding pull requests. The immediate question is not whether a percentage sounds alarming, but which changes entered the study, how the researchers counted a finding, and what the resulting rates can support. Reading the numbers this way lets us judge the evidence without either dismissing a real pattern or turning a scoped measurement into a universal claim about every coding agent.
The study begins by narrowing the surface
The study uses the AIDev dataset, a collection of pull requests associated with five coding agents. That description alone is not yet the unit of analysis. The researchers first narrowed the material to paths where a small change can influence building, packaging, deployment, execution, or the handling of identity material. The retained surface included continuous-integration definitions, container files, infrastructure-as-code, secret-bearing and other configuration files, shell scripts, and computational notebooks.
This choice follows the security question. A pull request that changes an ordinary application function may contain important defects, but the study was designed to examine structural security smells in files with a particularly direct relationship to execution and access. The researchers therefore excluded lock files, minified assets, vendor directories, and binary formats. They also reduced each patch to lines added by the change, discarding removed lines and diff metadata for the count. A finding in this analysis is consequently a risk pattern introduced on an added line in a retained high-risk file.
That boundary has two consequences. First, it makes the result more specific: the study can ask where agent-assisted changes introduce visible security-relevant structure instead of vaguely asking whether an entire repository is safe. Second, it makes the result incomplete by design. A weakness in an excluded path, or a problem that appears only through the interaction of new and unchanged files, is outside the measurement. The absence of a flag is not evidence that those other surfaces are clean.
The researchers organized the retained patterns into six categories. Secrets identity covers hard-coded keys, tokens, private keys, and credential-bearing connection strings. Supply-chain integrity covers mutable action or image tags and unpinned global installs. Over-privilege execution covers choices such as root containers, world-writable files, sudo, shell installers, and broad continuous-integration scopes. The remaining categories capture cleartext transport, permissive network exposure, and misconfiguration or hardening failures such as debug modes, logged secrets, or disabled encryption. The categories are not a complete security ontology; they are a practical lens grounded in secure-coding, benchmark, and hardening guidance.
The path from source material to result is therefore a sequence of filters and interpretations rather than a direct reading of all agent output:
The diagram also clarifies why a number from this study should carry its boundary with it. The dataset is not transformed into a security verdict in one step. A change must first be in the selected corpus, then have an added line in the retained surface, then receive a parseable label. Only after that can the result be aggregated by file, pull request, category, or severity. Manual validation checks the judge on a sample; it does not turn the entire corpus into a manually confirmed audit.
The study began with 16,370 file changes across 4,064 pull requests. The judge returned valid, parseable labels for 16,112 file changes spanning 4,022 pull requests. Those figures should not be collapsed into one denominator. The initial corpus describes the material selected for analysis. The valid set describes the material on which the reported judge-based rates were calculated. The difference is a reminder that data-processing failures are part of the evidence boundary, not an invisible detail to be rounded away.
The filter also determines what kind of security question each category can answer. A supply-chain finding concerns what a future build may trust or retrieve. An over-privilege finding concerns what an execution environment may do. A secrets finding concerns identity material that could authorize access. Network, transport, and hardening findings concern who can reach a service, whether a connection is protected, or whether a system exposes information it should not. These categories sit at different points in an action chain. Counting them together is useful for showing the overall burden, but it does not make their remediation interchangeable.
The added-line rule is equally important. It ties the smell to content introduced by the pull request, which is helpful when asking what an agent-assisted change brought into the repository. It does not establish that the new line created the entire risk. A newly added permission may activate an older component; a new configuration reference may interact with an unchanged workflow; a credential-shaped value may be a harmless example. The study’s design identifies where to investigate and prioritize, not every condition needed to determine impact.
Nor should the unparseable portion be silently treated as clean. The judge did not return a usable label for 258 of the selected file changes. Without a separate missing-data analysis, the safest interpretation is that those items are outside the valid-rate denominator. They may contain no smell, or they may contain findings the process failed to classify. This is a small but important habit for operational dashboards: “not measured” and “measured clean” are different states, and merging them can make a safety net appear stronger than it is.
Denominators change the meaning of the result
Within the valid labels, the judge flagged 3,807 file changes as containing at least one smell. That is a file-level rate of 23.6%. At the pull-request level, 1,563 of the 4,022 valid requests contained at least one flagged change, or 38.9%. These figures answer different operational questions.
The file-level rate asks: when a retained, changed file is examined, how often is it flagged? The pull-request rate asks: when a request is considered as a unit of review, how often does at least one of its retained changed files contain a smell? A pull request can change several files, and a single file can contain several distinct smells. The request-level measure will therefore be higher than the file-level measure when findings cluster inside requests. Neither measure says that the same proportion of all files or all pull requests in software development would fail a security audit.
The researchers also counted 8,701 distinct smells after deduplication. That count is not another prevalence denominator. It describes the volume of findings and makes category composition visible. A request with one flagged file and a request with many distinct supply-chain or privilege findings both contribute to the PR-level rate, but they do not contribute the same amount to the smell count. This is why a useful reading keeps three questions separate: how many requests were touched, how many files were flagged, and how many patterns were found.
The category distribution shows a strong concentration. Supply-chain-integrity findings numbered 7,160, or 82.3% of all detected smells. Over-privilege execution accounted for 835 findings, or 9.6%, and secrets identity for 294, or 3.4%. Misconfiguration and hardening, permissive network exposure, and cleartext transport made up the remainder. The result says that supply-chain configuration is where debt accumulates most often within this scoped surface. It does not say that supply-chain findings are the only meaningful risk or that each one has the same consequence.
Severity changes the picture again. Of the 8,701 findings, 6,292 were labeled major, 2,156 minor, and 253 critical. Two hundred and fifty-two of the 253 critical findings—99.6%—belonged to secrets identity. In other words, credentials were a small share of the total smell count but almost the entire critical tail. The 294 secret-identity flags should still not be read as 294 live credentials; the later manual check distinguishes genuine values from placeholders and other false positives. Even the severity label is a classification produced by the judge, not a demonstrated exploitability ranking or an incident forecast.
This contrast is central to interpreting security debt. Frequency identifies a broad maintenance and control problem. Severity identifies findings where the potential consequence is especially direct. A team that focuses only on the largest category may leave credential exposure under-protected. A team that focuses only on critical secrets may allow thousands of lower-severity supply-chain choices to accumulate until they become difficult to inventory, update, or reason about. The evidence supports a portfolio of attention rather than one universal threshold.
The study reports other differences that are useful as signals but unsafe as rankings. The proportion of flagged pull requests rose with request size: 16.2% for changes of one to nine lines, 29.8% for 10–49 lines, 42.9% for 50–199 lines, 42.9% for 200–999 lines, and 53.6% for changes of at least 1,000 lines. The spread between the smallest and largest groups was 37.4 percentage points. A larger request may expose more paths, combine more tasks, or simply give a reviewer more material to miss. The stratification shows where review pressure rises; it does not prove that line count by itself causes smells.
Agent and language comparisons require the same discipline. The reported PR-level rates were 45.5% for Copilot, 41.2% for Claude Code, 40.6% for Cursor, 39.7% for Devin, and 34.9% for OpenAI Codex. Codex’s flagged files carried the highest reported mean number of smells per file, 0.62. Among the listed repository languages, JavaScript was highest at 55.3%, while Python was lowest among the major languages at 31.8%; TypeScript sat at the 38.9% corpus average. These figures could reflect task mix, repository selection, file mix, user behavior, or agent configuration. The study does not control those factors well enough to turn the comparisons into a product or language safety leaderboard.
File categories give a more actionable signal because they align directly with system authority. GitHub Actions workflows and Docker files together held 87.6% of all detected smells, with file-level dirty rates of 36.3% and 36.4%. Application configuration files had a 2.2% dirty rate. This does not make application configuration irrelevant, nor does it prove that a workflow or container change will be unsafe. It does show that the places where software is built and executed deserve review attention that is proportional to their ability to affect downstream behavior, not merely to the number of visible lines.
The measurement itself has an error boundary. In a manually annotated sample of 376 file changes, the human coders reached a Cohen’s kappa of 0.929, indicating strong agreement under the study’s labeling process. The judge achieved precision of 0.908, recall of 0.775, F1 of 0.836, and kappa of 0.789 against the human gold labels. Precision answers how often a judge flag was supported by the reference labels; recall answers how much of the reference-labeled smell set the judge found. A judge can therefore be useful for large-scale screening while still missing a meaningful share of smells. Here, the recall of 0.775 implies that the reported frequencies likely understate the true prevalence within the study’s scoped corpus.
That limitation should be combined with the sampling boundary rather than used as a reason to discard the result. The study uses selected coding agents and open-source GitHub repositories. It does not establish how the same patterns would distribute in private enterprise repositories, under different agent scaffolds, with different security bots, or with stronger deployment guardrails. It also does not compare the observed rates causally with human-authored code. The evidence is strongest for a narrower conclusion: in this selected, high-risk, added-line surface, agent-associated pull requests contain a substantial and unevenly distributed burden of security smells, and the instruments used to detect them are informative but incomplete.
That conclusion is already enough to guide the next question. Numbers can tell us that debt is concentrated in certain paths, that large requests create more review pressure, and that supply-chain patterns dominate by count. They cannot, by themselves, explain how a mutable reference, a broad permission, or a credential becomes consequential. To understand that, we must examine what these files control and why a small configuration choice can cross from a pull request into the build, deployment, or runtime system.