From Model Behavior to Agent Safety: Permissions, Observability, and Runtime Controls
The instruction hierarchy improves one crucial decision inside an AI agent: whether a piece of text should influence the plan as an authorized instruction or be handled as untrusted data. This chapter focuses on what surrounds that decision—how inputs are transformed, how plans reach tools, how memory persists information, and how outputs are rendered—and on the controls that contain mistakes when the model is fooled or simply wrong. The practical judgment is where to place human control, dynamic permissions, policy checks, and observation so that better model behavior reduces risk without becoming the only barrier.
The security boundary follows the agent’s lifecycle
An agent is more than a model answering a prompt. It receives a user request and may gather context from documents, email, websites, application state, or sensors. It transforms those inputs into a form the reasoning model can use, develops a plan, invokes tools, stores or retrieves memory, and renders a result in an application interface. The instruction hierarchy sits inside this flow, but authority can be lost before the model makes its decision and consequences can appear after it has made one.
Input and perception: preserving origin
The first boundary is the input path. A user may type a command or speak a request, while the agent also receives contextual data from an email, webpage, file, or external system. Both may be converted into text or another model-readable representation. If the transformation preserves their origins and roles, the model has a basis for applying the hierarchy. If it merges them into one undifferentiated prompt, an instruction hidden in an email can appear to have the same status as the user’s request.
This is the agent-level form of indirect prompt injection. The attacker does not need to control the application’s system instruction. They need only place content in a source the agent is expected to inspect. Personalization introduces another version of the same boundary: preferences or facts learned from one interaction can be manipulated or contaminated, and in a multi-user system they must not cross from one person’s context into another’s.
The input boundary also has to identify the controlling human. If several people can access an agent, the application must know which user is giving the instruction and which permissions apply to that user. A correct interpretation of the words is not enough if the system has attributed them to the wrong person. The instruction hierarchy can rank a user message below a system message, but it cannot infer identity from language alone.
Reasoning and planning: turning interpretation into a proposal
The reasoning core combines the system instructions, the user’s query, retrieved content, and possibly memory. It interprets the goal and develops a plan, often as a sequence of retrieval and tool calls. The plan may be revised after each tool result. That iterative loop creates repeated opportunities for a lower-trust result to redirect the next step or for an ambiguous goal to drift away from the user’s intent.
The instruction hierarchy is designed to make the model resist one class of drift. It should treat instructions in a webpage or tool response as misaligned under the paper’s current conservative rule, while still using relevant facts from the response. But a model can also fail without malicious input. If “email Mike about the update” refers to several contacts, the model may select the wrong one. The resulting harm is a misinterpretation of intent, not necessarily an injection, yet it still shows why model judgment cannot be the sole authorization mechanism.
Autonomy magnifies these risks. A one-step assistant can make one mistaken choice. An agent that repeatedly plans, retrieves, and acts can compound an early error. A mistaken interpretation may change memory, influence the next plan, and eventually trigger a state-changing tool call.