How a Poisoned Item Gets Through the Gate
The retrieval problem in a RAG system is more specific than “can an LLM be tricked?” This chapter examines how one item placed in an external corpus can compete for a natural user query when the attacker cannot change the base model or inspect the whole corpus. The useful judgment is not a recipe for reproducing an attack. It is an understanding of the separate work required to cross the retrieval boundary: making content appear in the model’s context is one problem, and persuading the model to act on that content is another.
That separation corrects an easy assumption. If a malicious instruction is written into a document, it may look as though the attack is complete. In a retrieval-based application, however, the document must first win a ranking contest against legitimate material. A user may ask about a routine process while the poisoned item sits among thousands or millions of ordinary records. Unless it becomes one of the items selected for that request, its instruction-like language never reaches the model through the retrieval path.
The attacker must solve two different problems
The study’s threat model gives the attacker a constrained but meaningful capability. The attacker can insert one malicious item into an external corpus. They do not know the corpus contents in advance, do not have the parameters of the retriever or the LLM, and do not control the user’s natural query. They can query the embedding model through a standard interface and use the resulting similarity information. This is a black-box setting: the attacker can observe scores through access to the service, but cannot open the model and calculate gradients through its internals.
The single-item assumption matters. It asks whether a stealthy insertion can be enough, rather than allowing the attacker to flood the index with many copies or coordinate a large collection of documents. The model also targets arbitrary natural queries instead of requiring the user to type a special sequence. Those restrictions make the retrieval question harder and make a successful result more meaningful. They do not describe every real deployment. A system may prevent untrusted parties from writing to its corpus, may use a different search architecture, or may expose no embedding API to an attacker. Each of those conditions changes the threat model.
The malicious item itself has two conceptual parts. The attack fragment carries the instruction or objective that the attacker wants the downstream model or agent to follow. The trigger fragment has a different job: it changes the item’s representation so that the whole item is more similar to the target query. The trigger is not valuable because a human reader finds it persuasive. It is valuable because it affects the retriever’s comparison.
This division exposes why a plain malicious document usually fails. The attack fragment may contain language that is highly relevant to the attacker’s goal but unrelated to the user’s question. A document that says, in effect, “change your behavior” may be semantically distant from a request for a reimbursement procedure, a scientific fact, or a summary of a business topic. The legitimate corpus documents already discuss those topics, so they normally dominate the top results. The instruction becomes consequential only if a retrieval-oriented addition helps the document enter the model’s context.
The attack therefore crosses an initial condition and a termination condition. It begins with an item that exists in the corpus but is not yet exposed. It succeeds at the first gate when the item ranks within the system’s selected top-K results. It can fail because the item never enters the candidate set, because it ranks below the cutoff, or because a later stage rejects or ignores it. Retrieval success is necessary for this path, but it is not sufficient for an external effect.
The diagram is deliberately modest. It does not imply that the trigger controls the model’s later behavior. It shows only how the retrieval objective is attached to the item that contains the downstream objective. Once the item reaches context, another set of conditions takes over: how the LLM weighs the text, whether the application separates it from higher-priority instructions, and whether tools or approvals constrain the result.
This two-part view also changes how a defender interprets an evaluation. A test that places the attack fragment directly in context measures the second problem. A test that measures whether a constructed item appears in top-K measures the first. An end-to-end test needs both. Reporting one number as though it represents the entire path can hide whether a system is hard to retrieve from, hard to influence after retrieval, or merely protected by a final action control.
Optimizing a black-box retrieval signal
The paper adapts the Cross-Entropy Method, a classic black-box optimization approach, to search for a short trigger sequence. The important idea can be understood without following a token-by-token procedure. Instead of choosing one obvious prefix—such as repeating the user’s query—the method maintains a probability distribution over possible tokens at each position. It samples candidate sequences, measures how well each combined item matches the target query, keeps an elite group of high-scoring candidates, and updates the distributions to favor the choices that appeared in that group.
The loop turns a difficult discrete search into repeated evidence gathering. At the beginning, the search has little preference among candidate tokens. After scoring a batch, it learns which choices tend to improve the item’s similarity. The next batch is more concentrated around those choices, but smoothing prevents the search from discarding all alternatives too quickly. Repeating the process can produce a compact trigger that is difficult to infer from the attack fragment’s human meaning.
This is the key difference from naïve query repetition. Repeating the query chooses one human-readable sequence and assumes that the embedding model will combine it with the rest of the document in a favorable way. The CEM search explores a larger space of possible sequences under the model’s actual scoring behavior. The improvement comes from using feedback from the target embedding interface, not simply from adding more words. The paper reports that the optimized trigger can be shorter and more effective than the repeat-query baseline in the same retrieval setting.
The method also explains why access to the embedding API matters. The attacker does not need the model’s parameters if they can submit candidate text and observe a similarity score. That does not make the attack free. The search requires many black-box evaluations, and an attacker may need to guess which embedding model the target uses. The paper’s default experiment uses a ten-token trigger and a large bounded query budget; it reports commercial costs ranging from about $0.21 to $0.76 in the tested settings. Those figures are experimental estimates for the named services, not universal costs or a promise that every target can be optimized this way.
The distinction between practical evidence and theoretical guarantee is important here. The paper gives a utility guarantee for a linear form of the score function: under that condition, the sampling procedure can approach the best sequence in a bounded space with stated probability and query complexity. Real embedding systems often pool information across tokens, which motivates the study’s formalization, but the theorem does not prove that every deployed encoder has the required structure. The observed retrieval results, rather than the conditional theorem alone, support the paper’s practical claims.
The search space is bounded for another reason. An embedding model accepts a finite input length, and an unrestricted prefix would make the optimization question both unhelpful and unrealistic. A longer prefix can give the attacker more room to improve similarity, but it can also increase visibility, alter document behavior, or exceed system limits. The paper’s results show strong performance with short prefixes in many settings; they do not establish that longer is always better, or that a defender can identify the useful tokens by inspecting their ordinary language meaning.
Transfer is another condition that must remain visible. A trigger optimized for one embedding model may not work equally well on another architecture. The paper reports good transfer in some related model families and weaker transfer across other families. This limits the attacker when the target model is unknown, but it is not a security guarantee. An attacker may learn or infer the likely service, optimize several variants, or target a common model family. Conversely, an organization that changes embedding providers should not assume that a trigger disappears without testing the new stack.
Position and placement introduce similar nuance. The study reports that many models remain vulnerable when optimized tokens are moved or dispersed, while some models are more sensitive to token position. A location-sensitive encoder can make a previously optimized sequence less effective when moved, but the paper also reports that optimizing for the changed position can recover effectiveness. The general lesson is not that hiding a trigger in one location solves the problem. It is that the embedding model’s treatment of position is part of the actual attack surface and must be evaluated rather than assumed.
The same caution applies to the attack fragment. Reusing a trigger with a different downstream objective can reduce the attacker’s optimization cost if the trigger encodes a general similarity signal rather than a meaning specific to one payload. The paper reports evidence for that kind of reuse in its experiments. But this does not mean a trigger is universally portable across corpora, query distributions, models, or downstream applications. The trigger still has to win the ranking contest defined by the target system.
What this mechanism changes for security review
The central design failure is not simply that an LLM can misunderstand an instruction. It is that a system has allowed an embedding similarity score to control exposure without an independent statement of trust. The retriever asks which text resembles the query. It does not ask whether the text is authorized to change the model’s behavior. The trigger-fragment result demonstrates how those questions can be pulled apart: content can be made relevant to the ranking function without becoming trustworthy evidence or an authorized command.
That observation suggests several distinct checks. First, the application needs to know who can write to the corpus and what provenance is retained when an item is admitted. Second, it should measure what kinds of untrusted items can reach top-K under natural queries, not only whether a model resists text that has already been placed in context. Third, it should preserve the distinction between the retrieved item and the instructions that govern the application. Finally, the tool or action layer must not treat the model’s interpretation of a retrieved item as permission to perform a high-impact operation.
These checks also help interpret failure. If a poisoned item is never retrieved, the retrieval boundary has prevented this path for that query, although other queries or other documents may behave differently. If it is retrieved but the model ignores it, the second gate held in that test. If the model changes its answer but no external action is permitted, the system has experienced influence without the same level of operational consequence. If a tool call is accepted without independent validation, the original ranking error has crossed several additional boundaries.
The paper’s controlled results make the first gate a serious engineering concern, but they do not erase the conditions around it. The evaluation focuses on embedding-based retrieval, not every hybrid or reranking pipeline. The researchers use benchmark corpora and controlled agent workflows rather than probing live organizations. Cross-model transfer is incomplete, and the attacker’s ability to inject an item is an assumption rather than a universal fact. A responsible conclusion is therefore narrower than “all RAG systems are compromised.” It is that systems with a similar retrieval surface should test whether relevance can be manipulated before relying on downstream model behavior as their main defense.
Once the trigger and attack fragments are separated, the path from corpus to consequence becomes easier to analyze. The trigger addresses exposure. The attack fragment addresses interpretation. The application’s permissions, validation, and approvals determine whether interpretation can become action. Treating those as separate responsibilities is the foundation for judging the evidence—and for asking, at the next level, what happens after the poisoned item actually reaches context.