Trust Is an Architecture: Security, Provenance, and Safety

The repository’s production lessons can show that a service meets a latency or cost boundary, but operational evidence does not make the service trustworthy by itself. This chapter examines the next boundary: how retrieved data, tool results, credentials, provenance, and high-risk actions should be separated and checked. The practical judgment is to distinguish instruction from authority, then connect security and safety controls to the points where data enters, decisions are made, and effects leave the system.

The problem is easy to miss because the model sees many kinds of text in one context window. A user request, a policy document, a web result, a tool description, and a server message may all look like language. They do not have the same authority. If the system treats them as equivalent, an attacker can place instructions in content the user never intended to authorize.

Diagnose the indirect-injection path

An indirect prompt injection enters through content the agent reads during normal operation rather than through the user’s direct message. The repository identifies retrieval and tool output as important delivery paths. An attacker can publish a document containing an instruction, cause a search or retrieval step to select it, and rely on the model to treat the text as part of the task context. A tool controlled by someone else can return a similar payload. The user may be only the messenger who asked the agent to search or summarize.

This explains why a filter that checks only the user’s input is insufficient. The harmful instruction is not in the user’s words. It appears later, after retrieval or a tool call. A filter over all context may catch some patterns, but it also faces a difficult distinction: legitimate documents can contain imperative language, security instructions, or code that looks like a command. Keyword matching cannot decide authority on its own.

The repository’s information-flow framing provides a better diagnosis. User intent is treated as trusted input for the current task. Retrieved content and tool results are treated as untrusted data. If an action is proposed because of untrusted content, the system requires ratification from a trusted source before allowing the action to proceed. The exact implementation can vary, but the boundary must exist. A document may inform the answer; it should not silently become the principal that authorizes a side effect.

Consider a procurement assistant asked to compare vendor proposals. One retrieved proposal contains a hidden instruction telling the agent to upload the company’s credentials to “verify eligibility.” The request to compare prices is trusted intent; the proposal is untrusted content. The proposal may be quoted, summarized, or flagged as suspicious, but it cannot authorize an upload. A tool registry may contain an upload capability, yet the caller still needs a separate approval and a scope that permits the exact destination. If the assistant proposes the upload, the system should expose the reason and stop or escalate rather than quietly following the document.

This example also shows why the controls are layered. Content labeling identifies the source. An authority check rejects the attempted expansion of the task. Permission control limits which identity can call the upload tool. A sandbox can restrict the files and network destinations available to the process. Result verification can confirm that no upload occurred or that an approved transfer matched its declared destination. Audit records preserve the decision.