From 98,380 Skills to 157 Confirmed Threats
The subject here is not a frightening number but the measurement process behind it. The USENIX study of malicious agent skills asks how a large public registry snapshot can be narrowed from suspicious patterns to behaviorally confirmed abuse, and what confidence each stage deserves. Understanding that funnel lets a reader use the findings responsibly: take the 157 confirmed cases seriously, without treating them as either the total number of threats or the probability that any arbitrary skill is malicious.
The funnel answers different questions at each stage
The collection began with 98,380 skills indexed by two community registries in January 2026. One registry contributed 25,187 skills from 2,337 repositories; the other contributed 73,193 skills from 8,909 repositories. Together, they offered a large view of public, indexed activity, but not a universal directory. The study excluded private or unindexed repositories, skills shared directly inside enterprises, MCP servers, and official platform-curated collections. That scope is not a footnote. It determines what the denominator means.
The first reduction was static triage. Researchers scanned the skill documents and bundled scripts for patterns associated with credential harvesting, remote execution, data transmission, hidden instructions, privilege abuse, and related behaviors. This stage identified 4,287 candidates, or 4.4% of the full snapshot. A candidate is not a verdict. It is a package whose text or files contain something worth examining more closely.
Static triage is valuable because it makes a registry-sized problem tractable. It can search many artifacts quickly and preserve a reproducible reason for selecting them. But it sees resemblance rather than intent. A network request can support a legitimate feature or covert transmission. A command that changes permissions can be part of a setup routine or an attempt to gain unauthorized access. A credential-shaped string can be a test fixture rather than a secret. The same pattern can therefore be a useful search signal and a poor final label.
The next stage looked for runtime indicators. The 4,287 candidates were executed in isolated environments with monitoring for file, network, process, and credential-related behavior. The study also used synthetic invocations derived from each skill’s description and repeated execution with varied inputs. This narrowed the set to 762 candidates that produced at least one indicator requiring manual review. The transition matters: the question has changed from “does this package resemble a known pattern?” to “does this package actually cause a security-relevant effect under the tested conditions?”
The final stage produced 157 behaviorally confirmed malicious skills. They represented 0.16% of the full 98,380-skill snapshot and 3.7% of the static candidates. The confirmed set contained 632 labeled vulnerability instances across 13 patterns and six attack phases. Those figures describe different units: 157 is a count of skills, while 632 is a count of labeled behaviors that can coexist inside those skills. Treating them as interchangeable would make the result look either smaller or larger than it is.
The funnel is best read as a sequence of claims. The first claim is about the size of the public snapshot. The second is about how many artifacts matched the researchers’ search patterns. The third is about how many of those produced observable runtime signals. The fourth is about the cases that survived behavioral and human review. The fifth describes the multiple vulnerabilities found within the confirmed cases. No stage retroactively changes what an earlier stage measured.
Notice what the funnel does not contain: a step that declares every package outside the final set safe. The branches are not simply “malicious” and “clean.” A candidate can fail to trigger under the test conditions, be a legitimate dual-use utility, or remain ambiguous because the evidence is insufficient. The study’s dynamic environment and monitoring made those distinctions possible, but they did not remove the need for judgment. A measurement system is strongest when it makes uncertainty explicit instead of forcing every item into a binary label.
The shape of the funnel also reflects an engineering trade-off. A broad first filter is affordable enough to run across a registry, but it will include benign packages that happen to use sensitive operations. A narrow confirmation stage is more expensive because it requires execution, observation, and human comparison, but it protects the credibility of the final label. Expanding the first stage to eliminate all false positives would risk missing unfamiliar behavior; expanding the last stage without changing its test conditions would not solve dormant or environment-specific attacks. Each stage has a different job, and quality depends on not asking one stage to do another stage’s work.
This separation also explains why two apparently contradictory statements can both be true. Static analysis may flag a noticeable share of a fast-growing ecosystem, while only a small fraction of those candidates are confirmed malicious. That is not evidence that static analysis is useless. It is evidence that triage and ground truth serve different purposes. A broad filter accepts noise to avoid overlooking candidates; a confirmation process accepts a narrower view to keep a malicious label defensible.
It is useful to imagine the alternatives. If the researchers had published all 4,287 static matches as malicious, the headline would have been larger, but the label would have mixed intentional abuse with legitimate tools, examples, configuration fragments, and incomplete code. If they had published only the 157 cases without explaining the funnel, readers might mistake the result for a random sample or a prevalence estimate. By retaining the intermediate stages, the study shows both the scale of the search problem and the narrower evidentiary basis for its strongest conclusion.
The stages also expose where a future improvement would have to operate. Better static analysis could reduce the review queue or catch patterns that the first filter misses. Longer and more varied execution could expose delayed or conditional behavior. More context about publisher history and intended use could improve the judgment of intent. None of these improvements makes the others unnecessary, because they address different uncertainties: search coverage, runtime activation, and interpretation. The funnel is therefore not only a count of cases; it is a map of the system’s remaining failure modes.
What the confirmed set proves—and what it leaves open
The study defines maliciousness through observable behavior combined with evidence of intentional abuse, rather than through the presence of a suspicious string. The researchers looked for behaviors such as credential theft, unauthorized transmission, remote code execution, privilege abuse, instruction-level manipulation, and hidden functionality. They then compared what a skill did with what its public description said it did. A dual-use security tool could trigger a static pattern and still remain outside the confirmed set if its behavior was legitimate in context.
That distinction is the difference between detection and accusation. A scanner can say, “investigate this artifact.” A runtime harness can say, “this artifact made an unexpected connection or accessed a monitored resource under these conditions.” A reviewer can then ask whether the behavior was authorized, related to the stated function, and deliberate. The paper required two researchers to independently assign at least a probable-malicious rating before confirmation. Their agreement was high: 94.5%, with Cohen’s kappa of 0.89. Vulnerability labeling also achieved a kappa of 0.91.
The reported 99.6% precision belongs to a separate evaluation set of 300 skills, split evenly between benign and malicious examples, and assessed with five-fold cross-validation. In plain language, among the items the combined pipeline labeled positive in that evaluation, almost all were genuinely positive under the evaluation’s labels. That supports confidence in the pipeline’s positive findings. It does not tell us that 99.6% of all malicious skills in the wider ecosystem were found.
It also does not turn the evaluation set into a model of the registry. The evaluation is designed to test whether the detector can distinguish known benign and malicious examples under a controlled label, not to reproduce the proportions of an open marketplace. A detector can have high precision on the items it flags and still miss a class of attacks that never reaches its rules, never activates during execution, or appears only after a state change. A reader should therefore ask two questions of any security benchmark: “How reliable are the positives?” and “Which relevant cases had a chance to become positives at all?”
That second question is recall: how much of the relevant threat population escaped detection? The authors prioritize precision because a benchmark polluted with false accusations would be difficult to use for later research. But the cost of that choice is that the confirmed set is a lower bound. The 60-second execution limit can miss a payload that waits, checks its environment, requires a particular user interaction, or stores a trigger for a later session. A skill may behave normally inside a test container and differently on a production host. An attack may also be distributed across several invocations rather than completed in one.
The study’s review of candidates that passed static analysis but failed dynamic verification illustrates the uncertainty. In a 10% sample of 4,130 such candidates, researchers classified 93.2% as containing dormant or conditional malware, 4.9% as legitimate security tools, and 1.9% as genuinely missed malicious cases. The percentages come from a sample and do not convert directly into a corrected ecosystem-wide total. They do, however, establish why “not confirmed” cannot be casually translated into “benign.” Some unconfirmed artifacts were safe; some were quiet under the test conditions; some attacks required a detection path the study did not exercise.
This is a particularly important distinction for an agent skill because activation may depend on context. A helper can wait for a production hostname, a particular permission level, an unusual time, or a later interaction. An instruction can influence the agent only when a certain request creates the right conversational setting. A one-minute, one-session test is a useful safety boundary for handling untrusted code, but it is not an oracle for every future state. The study’s choice to call the confirmed set a lower bound is therefore a statement about the measurement design, not modesty for its own sake.
There is a second source of uncertainty in the label itself. “Malicious” combines an observed effect with a judgment that the effect is intentional and outside the stated function. That is appropriate for a ground-truth dataset, because a benchmark needs a defensible target. It also means readers should not transfer the label mechanically to every package with similar syntax. A legitimate security tool may look alarming at the static stage; the contextual comparison is what keeps the final category meaningful.
The same caution applies to the study’s 100% removal result. After responsible disclosure, registry maintainers removed or de-indexed all 157 reported skills. That is valuable external evidence: independent operators examined the reports and judged the cases actionable enough to remediate. It supports the practical value of the classifications. But removal is a response after discovery. It does not show that users were protected before installation, that every malicious skill was found, or that a registry without equivalent review would produce the same outcome.
The denominator also carries a time boundary. The sample represents a public ecosystem at one moment early in the life of agent skills. It cannot establish a stable prevalence rate across later registry versions, other platforms, enterprise deployments, or private sharing channels. The paper’s own results are additionally skewed by a single publisher cluster that accounts for more than half of the confirmed skills and a larger share of labeled vulnerabilities. That concentration is itself a finding about industrialized campaigns, but it means the average case should not be imagined as an independent, typical attacker.
Reading the numbers correctly therefore requires keeping three layers separate. Observed scale tells us that malicious skills were present in a large public snapshot. Measurement confidence tells us that the 157 cases were supported by runtime evidence, manual review, and high agreement rather than by pattern matching alone. Uncertainty tells us that time limits, dormant behavior, taxonomy choices, and excluded populations leave threats outside the confirmed set. A strong conclusion includes all three layers.
This way of reading the evidence also changes the practical question for a registry operator. The goal is not to drive the static-candidate percentage to zero by suppressing every sensitive operation. That would hide legitimate tools and encourage a false sense of cleanliness. The goal is to use static analysis as an efficient queue, runtime behavior as a source of observable facts, and contextual review as the boundary for a serious maliciousness judgment. A detector succeeds when its claim matches the evidence it actually collected.
The study’s headline numbers are consequently neither a reason for complacency nor a license for panic. They show a public ecosystem in which intentional abuse was measurable, multi-stage, and actionable, while also showing how much depends on the conditions of observation. The next question is what those confirmed skills did after crossing the installation boundary. To answer it, we need to stop counting cases as isolated files and trace the phases they combined: access, execution, concealment, exfiltration, and impact.