Checkpoint: What the Numbers Are Really Measuring

Agent skills now have two properties that must be considered together: they can arrive through an ordinary distribution channel, and they can influence both an agent’s instructions and a host’s execution environment. This checkpoint connects that trust-boundary model to the evidence funnel, asking what the confirmed cases establish and what they do not. The practical judgment is simple to state but easy to lose: a small, high-confidence set of observed abuses can reveal a broad architectural exposure without becoming a prevalence rate for every skill.

Authority and evidence are different dimensions

The first distinction is between authority and certainty. Authority is what a skill could reach after installation: the agent may interpret its persistent instructions, and local helpers may operate with the permissions of the host. Certainty is how strongly an investigator knows what a particular skill did: a registry listing supplies one kind of information, a static match another, a runtime indicator more, and a behaviorally confirmed case the strongest claim in the study. These dimensions do not move together automatically.

A skill can have substantial authority and very little evidence about its behavior. That is the ordinary condition of a newly listed package: it is available to the agent, but the user may know only its name and description. Another skill may have an obvious static signal but limited authority in the environment where it is tested. A third may produce a strong runtime signal but still require contextual review to determine whether the action was an intentional abuse or a legitimate function. Treating authority as proof, or suspicion as certainty, collapses two different questions.

The first three stages of the study’s funnel make this separation visible. The 98,380-item snapshot describes the size of the public indexed ecosystem. The 4,287 static candidates describe the output of a search strategy. The 762 runtime-flagged candidates describe packages that produced an indicator under the study’s tested conditions. Only the 157 confirmed cases support the paper’s strongest maliciousness claim, because the final category combines observed behavior with comparison to the stated function and a judgment about intentional abuse.

flowchart LR P["Publisher claim"] --> R["Registry listing"] R --> I["Installation and host authority"] I --> B["Code plus agent instructions"] B --> S["Static suspicion"] B --> O["Runtime observation"] S --> M["Contextual manual judgment"] O --> M M --> C["Confirmed behavior or unresolved case"] R -.-> Q["Public snapshot scope"]

The diagram’s crossing lines matter. A package’s authority is established around installation and runtime, before an investigator has necessarily observed its behavior. Static suspicion and runtime observation are complementary routes to evidence, not two names for the same event. Manual judgment connects them to the public function and the question of intent. The final box must retain both outcomes: some cases become confirmed; others remain unresolved because the conditions did not reveal enough.

This framework explains why the headline can be important without being universal.