Overview

This chapter follows a retrieved poisoned item beyond RAG exposure. In ordinary RAG, it can change a controlled answer, measured as attack success rate (ASR), which is distinct from Recall@5. In agentic workflows, query reformulation, context loss, and handoffs can carry external text toward tools or other agents. Controlled evaluations used 11 datasets and 11 language models for RAG, and public Enron email workflows with AutoGen and Magentic-One for agent behavior; reported outcomes depend on models, tasks, orchestration, and random seeds. The central boundary is authorization: a generated answer or tool request remains a proposal until independent validation, least privilege, provenance, and approval determine whether an external effect is allowed.

Learning outcomes

Recall@5 measures retrieval exposure, while ASR measures whether poisoning changed the defined model answer; the metrics are not interchangeable.
For GPT-4o’s tested code-execution task, ASR was 2–4% in single-agent versus 72–80% in multi-agent workflows under those conditions.
Generated tool requests should remain proposals subject to provenance, independent validation, least privilege, and approval.

Keywords

RAGAgentsASRContext lossTool callsProvenance

When Retrieved Text Becomes Action

The retrieval-based LLM has crossed its first gate when a poisoned item reaches the model’s context, but exposure is not yet the same as impact. This chapter follows that item through two downstream settings: ordinary RAG, where it can steer an answer, and agentic workflows, where it can travel through query rewrites, tool calls, and multiple model contexts. The practical judgment is to identify the second gate clearly: what does the model do with retrieved content, and which application controls keep that interpretation from becoming an unauthorized action?

The distinction matters because retrieval metrics can sound alarming while still leaving the consequence unspecified. A malicious item may appear in the top five and have no observable effect. It may alter a generated answer but never reach an external system. Or it may be passed between agents until a tool treats it as an approved instruction. These are different states in the same lifecycle, and the study’s end-to-end evaluations are valuable because they examine more than whether the item was ranked.

RAG: retrieval changes the answer

The RAG case study keeps the downstream task deliberately simple. A single malicious document is inserted into each benchmark corpus, and the language model is asked to answer the user’s query normally. The attack’s target is a fixed phrase selected by the attacker. A trial counts as successful when the clean corpus does not already produce that phrase but the poisoned corpus does. This comparison isolates the effect of the injected item instead of counting answers the model would have generated anyway.

That success measure is called attack success rate, or ASR. It is not Recall@5. Recall@5 asks whether the malicious item entered the five retrieved results. ASR asks whether the poisoned setting changed the model’s answer under the defined task. Retrieval is therefore a prerequisite in this experiment, while answer manipulation is a downstream outcome. The two values can move together, but they need not be equal.

The paper evaluates the RAG case across the same broad family of 11 information-retrieval datasets and across 11 language models. The models include Qwen3, Llama 3, and Vicuna families, with both base and instruction-tuned variants. Results are averaged over five random seeds. Across nearly all combinations, the optimized malicious document can reliably coerce the model into the target answer once retrieval has brought it into context; the reported ASR is often close to one.

The result is not that every answer in every RAG application will be replaced. It is that the second gate can be surprisingly permeable when the context contains instruction-like material and the task allows the model to use it. The model is not merely retrieving evidence and copying a citation. It is deciding how to respond while the retrieved text is present. If that text tells the model to prioritize a different response, the model’s normal ability to follow instructions can become the mechanism that turns retrieval exposure into answer manipulation.

The result also challenges a simple “bigger model means safer application” story. Instruction-tuned models are generally more willing to follow the injected instructions in the paper’s experiments. That behavior is useful in ordinary interaction: users want a model to follow a task and use supplied context. Under indirect injection, the same responsiveness can make the model more faithful to the wrong source.

Published by DeepMox

FAQ

How is ASR different from Recall@5 in RAG?

ASR measures whether the poisoned setting changed the model’s answer, whereas Recall@5 measures whether the malicious item entered the five retrieved results.

Why can multi-agent RAG be more vulnerable after retrieval?

Multi-agent RAG can be more vulnerable because handoffs may strip away the original request and source labels, letting a downstream executor treat retrieved material as trusted workflow input.

What keeps a retrieved instruction from becoming a tool action?

Independent validation, provenance, least privilege, and approval keep a retrieved instruction from becoming an authorized tool action. A model’s request is a proposal, not permission.

Previous12 minRAG Retrieval Evidence: What 94.1% Means

Next12 minRAG Defense: Why Simple Filters Fall Short

RAG Agents: When Context Becomes Action

When Retrieved Text Becomes Action

RAG: retrieval changes the answer

FAQ