What Counts as a Vulnerability When the System Is Probabilistic?
This chapter examines the meaning of “vulnerability” in AI vendors’ disclosure policies. The question is narrower than whether an AI system can behave badly: it is whether a condition in the system lets someone violate a security policy or cause a material, unauthorized effect, and how the AI mechanism makes that claim testable. Keeping those questions separate gives researchers a more durable way to describe findings as models, prompts, and product architectures change.
The distinction is necessary because AI systems produce behavior rather than simply exposing a visible defect in a line of code. Their outputs can vary with wording, conversation history, model version, retrieved context, tool access, or deployment settings. That variability complicates evidence, but it does not make the concept of vulnerability meaningless. It changes what must be shown before a vendor can decide whether a report belongs in a security program, an abuse or safety channel, or neither.
A vulnerability is a relationship between a condition and a security policy
The CERT Guide to Coordinated Vulnerability Disclosure defines a vulnerability as a set of conditions or behaviors that allows an explicit or implicit security policy to be violated. The condition might be a software defect, a design choice, a configuration, an unexpected interaction, or an environmental change. The guide also offers a practical description: a vulnerability is an undesirable, exploitable, and likely unintended feature that lets an attacker perform an action that would not otherwise be available to them.
Several parts of that definition do important work. “Condition” means the vulnerability is not identical to the visible effect. It may depend on a combination of input, privilege, configuration, component interaction, or deployment context. “Allows” points to capability or opportunity, not necessarily to a completed attack. And “security policy” can be explicit, such as an access-control rule, or implicit, such as the expectation that one customer cannot retrieve another customer’s private information.
That framing is useful for AI systems because it starts with the boundary that matters. If a chatbot produces an incorrect answer, the relevant policy may be accuracy, reliability, or product quality rather than security. If the same model is connected to a private knowledge base and can be induced to disclose another user’s records, the behavior crosses a confidentiality or authorization boundary. The model’s language output is the visible mechanism, but the security judgment rests on what the system allowed the requester to do.
The distinction is also useful when the behavior looks deliberately adversarial. A researcher may use an unusual sequence of instructions to make a model ignore a constraint. That sequence is evidence about how the system behaves under pressure; it is not yet a complete vulnerability claim. The report needs to say what the bypass permits. Does it reveal a hidden system prompt? Expose another customer’s data? Invoke a tool with the wrong permissions? Alter an account or a transaction? Or does it only produce prohibited text within the researcher’s own session? Those outcomes have different security significance even if they are all described as “jailbreaking.”
The CERT guide separates vulnerability from three nearby concepts. An exploit is software or a method that uses a vulnerability to achieve an effect. A vulnerability can therefore exist before anyone has demonstrated a polished exploit, provided the relevant condition and impact are sufficiently evidenced. An incident is a violation or attempted violation of a security policy. It may involve a vulnerability or exploit, but it may also result from misuse, a configuration problem, or another event that does not establish a reusable product-level weakness. A harmful output is even broader: it may be a safety concern, a quality failure, an incident, or a vulnerability depending on the system and the policy at stake.
Those distinctions prevent two opposite errors in a vendor’s scope. The first is over-inclusion: every surprising, offensive, or inaccurate response is treated as a security vulnerability. That floods a security queue with reports that require product, safety, or policy decisions rather than vulnerability triage. The second is over-exclusion: anything that does not resemble a conventional software flaw is dismissed, even when it enables unauthorized access or action. Both errors arise when the label replaces the relationship between condition, policy, and effect.
This relationship also explains why disclosure scope should not end at the word “eligible.” Coordinated vulnerability disclosure is a process that includes reporting, validation and triage, remediation, public awareness, and deployment. A report is valuable because it can lead to action that reduces risk, not simply because it qualifies for a reward. If a vendor accepts a finding but does not know which team owns the relevant model or integration, the scope decision has not yet produced protection. If it rejects a safety-related report from the bounty queue but provides no other channel, the category distinction has become a dead end.
The word “security” does not make the boundary simple. Security policies may be implicit, and AI products often combine model behavior with ordinary software, private data, external tools, and user accounts. A report can therefore contain several layers: an AI behavior, a system condition that makes the behavior reachable, a violated property, and an effect on a user or asset. The job of scope is to keep those layers visible long enough for the right decision to be made.
The diagram is intentionally not a hierarchy in which every model behavior becomes a vulnerability. It shows possible relationships. An observed behavior may instantiate a vulnerability when it creates an exploitable policy violation. An exploit may use that vulnerability, and an incident may result from it, but neither relationship is guaranteed. The CVD response is what turns a credible claim into coordinated action.
Mechanism tells you how the boundary is crossed
The current debate over AI vulnerability scope often appears as a choice between two positions. One position emphasizes mechanism: AI systems learn probabilistic rules, so prompt injection, model extraction, adversarial examples, and jailbreaks may require categories and procedures that ordinary software policies do not mention. The other emphasizes outcome: if a system leaks sensitive data or permits unauthorized action, it should not matter whether the path used SQL injection, an LLM injection, or another mechanism.
The useful answer is to keep both layers. The outcome explains why the report matters; the mechanism explains how the condition arises, how it can be tested, and what kind of repair might be possible. An outcome-only policy can recognize a data-access violation, but it may leave researchers uncertain about what testing is authorized, how much variation is expected, or which model and application states must be reproduced. A mechanism-only policy can name prompt injection or jailbreaks while failing to distinguish a harmless demonstration from a cross-user compromise.
Consider prompt injection. The phrase describes a way of influencing a model by placing instructions in user input, retrieved content, a document, or another data source. That description identifies a mechanism, not a complete impact. In one application, the result might be an altered answer in the requester’s own conversation. In another, the model may have access to a private repository, email account, or transaction tool. If the same influence causes the system to disclose protected information or perform an action beyond the user’s authority, the security property is clear even if the model’s response is probabilistic.
Jailbreaking raises a related but different question. A jailbreak may bypass a model’s built-in safety instructions and generate content that a vendor does not want it to produce. The USENIX study found that jailbreaking was among the categories most often treated as out of scope in the policies it examined. The study also records vendor rationales that a jailbreak may affect only the requester’s own session or may be difficult to mitigate completely. Those rationales do not prove that every jailbreak is harmless. They reveal that vendors are asking whether the behavior changes a protected capability, affects another party, or produces a demonstrable security impact that their program is equipped to handle.
Hallucination illustrates the other side of the boundary. An inaccurate or fabricated response may be a serious product or safety problem, especially when users rely on it in a consequential setting. But it does not automatically establish that an attacker can violate a security policy. The study reports that hallucination was rarely accepted as a vulnerability in the public policies it analyzed, partly because some vendors saw no clear application-security impact. That is a scope choice, not a finding that the behavior lacks social importance. It may justify a separate safety-reporting, monitoring, or model-quality process rather than a silent rejection.
Probability changes the evidence question but not the need for evidence. A deterministic reproduction is convenient, but a behavior can still be security-relevant when it occurs under identifiable conditions with a meaningful probability. Conversely, a single dramatic output may be insufficient if the vendor cannot determine the preconditions, affected versions, reachable data, or likely consequence. The report should make the uncertainty part of the claim: how often the behavior appeared, what conversation or state preceded it, what privileges were available, what the researcher could observe, and which effect was actually demonstrated.
This is why the AI mechanism cannot be treated as decorative detail. It may determine whether a finding is reproducible, whether testing risks exposing real data, whether the vendor can isolate the relevant component, and whether a patch, retraining step, guardrail, access-control change, or monitoring rule is plausible. The mechanism can also reveal that the receiving vendor does not own the vulnerable layer. A model extraction report, for example, may concern a provider’s intellectual property even when the exposed interface is embedded in another application.
A strong report can keep the layers distinct without becoming cumbersome. It should identify the observed behavior and its preconditions; state the security property or material harm at issue; describe the affected asset, user, or state; explain the AI-specific mechanism; and show the evidence and limitations. It should also say what the researcher did not access or change. That structure helps a vendor ask the right follow-up question instead of arguing about whether a fashionable term appears in the policy.
The same structure gives a vendor a better basis for scope decisions. If the property is unauthorized data access, a security program may have a direct eligibility rule. If the effect is harmful content confined to the researcher’s session, the vendor may decide that the issue belongs in a safety or abuse process. If the effect depends on a downstream tool or a foundation-model provider, the vendor may need to coordinate rather than simply accept or reject the report. In each case, the mechanism, context, and evidence explain why the route differs.
The key distinction is therefore not “traditional vulnerability versus AI vulnerability.” It is between a label that ends the analysis and a description that exposes the system boundary being crossed. AI vendors need mechanism-aware policies because the route to impact can be probabilistic, indirect, or shared across components. Researchers need outcome-aware reports because a mechanism name alone does not show why anyone should act.
A useful definition can hold both truths at once: an AI behavior becomes a vulnerability when a sufficiently evidenced condition in an AI-enabled system can be used to violate a security property or cause material, unauthorized harm. The behavior’s mechanism, context, reproducibility, and limits remain part of the claim. This definition will not settle every case automatically. It does something more valuable: it tells the parties what must be examined before a report is dismissed, rewarded, redirected, or escalated.