The Semantic Layer: Context Without Surrendering Authority
SkillSpector’s deterministic analyzers can show that a suspicious phrase, execution sink, or data path exists. They cannot always decide what that signal means in the skill’s stated context. The optional LLM layer addresses that gap by examining intent, paraphrase, narrative, and purpose-versus-behavior mismatch. This chapter explains how that semantic help is bounded so readers can choose when it adds value, understand what content leaves the machine, and recognize when a static-only or degraded result is not a full semantic scan.
The important word is “optional.” SkillSpector does not make an external model a prerequisite for every security conclusion. Its design gives deterministic findings priority and uses the model to interpret or supplement them. That division of labor is a response to two different failure modes: literal checks can overreact to benign context or miss a novel phrasing, while a language model can produce a plausible explanation without having reliable evidence behind it.
What Semantic Analysis Adds
A pattern matcher is good at recognizable forms. It is less good at a sentence that politely reframes a jailbreak, a story that gradually normalizes a dangerous request, or a description that omits the behavior most important to the user. These cases require connecting words to purpose. SkillSpector’s semantic security discovery analyzer is designed for that residual gap. It asks an LLM to look for intent and attack phrasing that evade obvious keywords: indirect instructions to redirect an agent, paraphrased jailbreaks, natural-language data-leak directions, and gradual or narrative deception.
Other semantic analyzers ask related questions. Developer-intent analysis compares the skill’s manifest and documentation with its actual code, looking for a materially different purpose, an unjustified capability, scope beyond declared permissions, or documentation that contradicts what the code does. Quality and policy analysis examines concerns such as vague activation triggers, missing user warnings, or natural-language policy violations. MCP tool analysis can compare a declared description with executable behavior. These are not alternative names for the same check. They operate at different relationships between the source’s words, the code’s capabilities, and the agent’s context.
The semantic question is also deliberately narrower than “does this skill feel suspicious?” The analyzer prompts tell the model not to report benign security discussion or obvious static patterns already covered elsewhere. It should report a residual semantic concern only when the risk depends on meaning, context, or cumulative intent. This precision-over-recall posture is important for a pre-install gate: a long list of speculative findings can make reviewers ignore the evidence that matters.
To make the model’s output usable, SkillSpector’s shared LLM analyzer base turns a file into a bounded unit of work. A small file can be sent as one batch. A larger file is split into line-aligned chunks that fit the model’s input budget, with overlapping lines so a relevant statement near a boundary does not lose all surrounding context. The prompt numbers source lines with an L prefix, and the returned finding carries a source line rather than an approximate location. A reviewer can therefore move from a semantic claim back to the material that prompted it.
The output is structured rather than accepted as free-form prose. A discovery finding contains a rule identifier, message, severity, location, confidence, explanation, and remediation. Pydantic validation checks that shape before the result enters the graph state. A model that returns malformed data does not get to invent an arbitrary report structure. The analyzer can retry bounded validation failures, while a failure that persists is isolated to that batch.