Before You Install an AI Skill, What Are You Actually Trusting?

SkillSpector is a security scanner for AI-agent skills, and its central question is deceptively simple: is a skill safe to install? The difficult part is that an agent skill is not only a program and not only documentation. It may contain instructions that shape an agent’s decisions, metadata that influences tool selection, executable files, dependencies, hooks, and configuration. This series examines how SkillSpector turns that mixed object into evidence for a pre-install trust decision—and where that evidence stops short of a safety guarantee.

The distinction matters because installing a skill can grant an agent a new source of instructions and capabilities. A reviewer therefore needs more than a reputation signal or a list of suspicious words. They need to know what the source contains, what the code can do, what data can reach sensitive operations, whether the declared purpose matches the implementation, and whether the scan actually saw the relevant material. SkillSpector’s design is best understood as an attempt to make those questions explicit and bounded.

When a Skill Looks Like Text but Behaves Like Code

A conventional software review begins with an executable artifact. An AI-agent skill complicates that starting point. A Markdown file can be read as instructions by an agent, even when a human sees it as ordinary prose. A description or parameter field can influence how an agent chooses and calls a tool. A directory can carry scripts and dependencies that become active only after installation. A settings file can attach a handler to a lifecycle event. The effective behavior is distributed across these surfaces.

That is why SkillSpector does not limit its stated purpose to finding malware in scripts. The repository describes the project as an Apache-2.0 Python tool that scans skills for vulnerabilities, malicious patterns, and security risks before installation. It supports local and remote inputs and produces a risk assessment with recommendations. It is also presented as one part of the NVIDIA Verified Skills pipeline, where skills can be scanned, evaluated, and signed before publication. That ecosystem connection is useful context, but it is not the same as saying that every local scan verifies a signer or a certificate chain.

The first principle is therefore simple: installation is the trust boundary. Before installation, a reviewer can still refuse the skill, ask its author for clarification, or isolate it for deeper examination. After installation, the skill may be operating inside an agent environment whose permissions and context are valuable. A scanner placed before that transition can reduce risk; it cannot undo the consequences of trusting the skill later.

This also explains what SkillSpector deliberately does not do. The repository says that it never executes the scanned skill. Its analysis is static—using text and pattern checks, Python AST analysis, YARA, and optional LLM evaluation of file contents. That makes the scanner safer to run on an untrusted bundle than a tool that imports or launches it. But static inspection is not a sandbox. If an operator installs a skill despite a warning, SkillSpector does not contain the skill, restrict its runtime permissions, or observe its subsequent behavior.

Consider a hypothetical skill described as a local document formatter. Its visible instructions may be harmless, while an accompanying script reads environment variables and sends them to a remote endpoint. A text-only review might miss the relationship between the description, the script, and the transmitted data. Conversely, a security document might contain words such as “credential” or “shell” while discussing threats rather than performing them. A useful scanner must detect both concrete behavior and context, and it must avoid pretending that either one is sufficient alone.

The companion Skill Inspector included in the repository makes the same point in operational terms: static evidence and source-aware semantic review are independent lines of inquiry. A low score can miss semantic risk. A high score can sometimes be explained by sensitive behavior that is documented, necessary, bounded, and under user control. The score is a posture signal; the decision still requires judgment.

A Trust Decision Is Made From Several Kinds of Evidence

SkillSpector’s overall model becomes clearer if its evidence is grouped by the question being asked rather than by the name of an analyzer.

The first question is, “What exactly arrived?” Input handling is part of the security model. SkillSpector can resolve a Git repository, a direct file URL, a ZIP archive, a single Markdown file, or a local directory. Its input handler validates permitted hosts, rejects destinations that resolve to private or reserved addresses, rejects unsafe local links, and bounds downloads, clones, and archive members. Those controls do not prove that an allowed repository is trustworthy. They prevent the act of collecting evidence from becoming an uncontrolled network, filesystem, or archive operation.

The second question is, “What does the bundle contain?” The build-context stage treats raw bytes as the source of truth. It classifies artifacts as text, binary, or opaque based on their bytes and decodability instead of trusting an extension. It builds an inventory, a bounded cache, manifest metadata, references, and an inspection ledger. Security-normalized text views can remove or interpret certain Unicode and formatting tricks while retaining offsets back to the original material. That is important for a finding: a reviewer needs both the normalized signal and a location they can inspect.

The third question is, “What can the code do?” Deterministic analysis answers this in several ways. Text patterns can identify prompt injection, data transmission, credential access, unsafe defaults, trigger abuse, or supply-chain behavior. AST analysis can recognize execution sinks such as exec, eval, dynamic imports, subprocess calls, reflective access to command sinks, and unsafe deserialization. Taint analysis asks a more specific question: does an environment value, file read, network input, or user input flow into a network, execution, write, or deserialization sink? A call to a network library is not the same claim as proof that credentials reach that call.

The fourth question is, “Can the declared identity and dependencies be trusted enough for the intended use?” Dependency checks distinguish a package name from the version that will actually be installed. Lockfiles can provide exact versions; a name-only advisory does not prove that the release selected by a broad version range is affected. SkillSpector can query OSV.dev for known vulnerabilities and falls back to a small built-in list when the service is unreachable. It also uses YARA signatures and checks for concealed executables or shipped Python bytecode. These are different forms of supply-chain evidence, and collapsing them into one label would overstate what any one check established.

The fifth question is, “What does the surrounding context mean?” Some findings require understanding purpose and intent. A natural-language instruction may be a paraphrase of a jailbreak without using obvious keywords. A script may perform a network request that is expected for a web-search skill but unjustified for a local formatter. A comment may claim that an operation is read-only while the code writes to disk. SkillSpector’s optional semantic analyzers and meta-analyzer address these residual questions. They add context to deterministic findings; they do not replace the underlying evidence.

flowchart LR N01["Untrusted skill source"] --> N02["Bounded evidence construction"] N02 --> N03["Deterministic analysis"] N02 --> N04["Optional semantic analysis"] N03 --> N05["Findings and provenance"] N04 --> N05 N05 --> N06["Coverage and completeness"] N06 --> N07["Allow, warn, or block"] N08["No containment after install"] -.-> N07

This arrangement is more than a convenient pipeline diagram. It prevents different kinds of certainty from being mixed. A static match is an observation about content or structure. A taint result is a bounded inference about a path through code. An LLM explanation is a contextual interpretation. A coverage record says whether relevant work completed. A recommendation is a policy projection over all of them. Each layer can be useful without being promoted into a universal proof.

Read the Result as Posture, Not as Permission

A report becomes dangerous when its summary is easier to read than its qualifications. SkillSpector’s design tries to keep both visible. It computes a score and severity, but also records the files and work items that were partially inspected, skipped, failed, opaque, or outside the content-analysis scope. If a relevant limit is reached, the result is incomplete. The repository’s report logic can raise an otherwise SAFE recommendation to at least CAUTION, and the MCP interface requires successful, complete inspection before it returns safe_to_install: true.

That does not mean every incomplete scan is malicious. A binary file may simply be an image. A recognized signing bundle may be intentionally excluded from content analysis. A nested archive may exceed a cumulative inspection limit. The point is different: the scanner must not turn “not inspected” into “clean.” A zero-finding result answers only what the retained evidence allowed the analyzers to see.

The data path also needs to be part of the decision. LLM analysis is optional, but when it is enabled, analyzer-eligible file contents are sent to the configured provider. Hidden, nested, binary, and other local-only material is excluded from that external-model cache, and environment-file values are redacted for model use. Static-only mode keeps file contents local, yet dependency checks can still send package names and versions to OSV.dev. “No LLM” therefore does not mean “no network activity.” It means a different egress and coverage profile.

The practical questions for every report are consequently more useful than the headline score:

  • Does the skill’s stated purpose fit the capabilities visible in its files?
  • Are permissions and allowed tools narrow enough to make that purpose testable?
  • What evidence supports each high-severity finding, and where is it located?
  • Were hidden, nested, binary, or oversized artifacts fully inspected?
  • What content or dependency coordinates left the machine, and to whom?
  • Does the chosen integration treat CAUTION, incomplete work, and execution failure as distinct states?
  • If a baseline suppressed a finding, what was reviewed and why should that decision remain valid?

Those questions turn SkillSpector from a badge generator into a disciplined pre-install checkpoint. The scanner’s value is not that it can pronounce an unfamiliar skill harmless. Its value is that it can make the trust decision more concrete: acquisition is constrained, evidence is located, multiple forms of behavior are tested, semantic uncertainty is surfaced, and incomplete work is not silently rewarded.

The next question follows naturally from that model. Before asking whether a particular pattern is dangerous, we must understand how an untrusted source is acquired, bounded, classified, and preserved as evidence in the first place.

References