Why Secure AI Agents Need More Than a Smart Model
Google’s Approach for Secure AI Agents examines a specific change in computing: language models are becoming systems that can act. This opening focuses on why that change turns security into a problem of delegated authority and control boundaries, not just a problem of producing accurate text. The reader should come away able to distinguish an answer from an action, see why autonomy enlarges both value and risk, and understand why Google argues for a hybrid defense-in-depth approach.
From generating answers to exercising authority
A conventional language model mainly transforms an input into an output. The output may be wrong, biased, or unsafe, but the model does not necessarily have a direct path to a bank account, an inbox, a file system, or a browser. An AI agent adds that path. It can perceive its environment, interpret a goal, make a plan, call tools, use the results to choose another step, and present or store what it has done.
That difference is easy to underestimate because an agent’s decisions may still arrive as ordinary-looking text. Consider an assistant asked to research a topic and draft an email. A text model might produce a draft for a person to inspect. An agent may search several sources, read documents, retain facts in memory, choose a recipient, and send the message. The visible request is small; the delegated workflow is not. Each additional capability creates another place where an ambiguous instruction, a misleading document, or a mistaken assumption can influence the next decision.
Google describes agents as useful precisely because they can handle multi-step work. They may categorize service requests, gather information, summarize it, and take follow-up actions. The same features that make them valuable also increase the possible blast radius of failure. A model that misunderstands a sentence can produce a poor paragraph. An agent that misunderstands a sentence can choose the wrong contact, delete information, make a purchase, submit data to a website, or expose material that the user expected to remain private.
The security question is therefore not simply, “Can the model give a safe answer?” It is also, “What authority does the system give the model’s interpretation, and what happens when that interpretation is wrong?”
Why autonomy creates a control problem
The risk does not depend only on a malicious user. Current AI models are non-deterministic: the same input does not guarantee a perfectly repeatable behavior. Agents can also produce emergent behavior that was not explicitly programmed. More importantly, an agent must interpret human goals that are often incomplete. “Email Mike about the project update” leaves room for uncertainty if several people are named Mike, if the update contains confidential information, or if the user expected to approve the final message first.
An agent can also encounter instructions that were never written by the controlling user. A webpage, an email, a retrieved document, or a tool response may contain text that looks like a command. If the model treats that text as authoritative, data intended as background can redirect the plan. This is the basic shape of prompt injection. The problem is not that the text is linguistically unusual. The problem is that the system has failed to distinguish who is allowed to instruct the agent from what the agent is merely supposed to inspect.
Iterative planning makes this harder. An agent may reason, call a tool, receive new information, revise the plan, and call another tool. A mistaken assumption can survive into the next cycle. So can a malicious instruction hidden in one of the results. The risk accumulates not because every step is independently catastrophic, but because each step can change what the agent sees, remembers, or is prepared to do next.
This is why autonomy should be treated as a risk-and-value decision rather than as a simple measure of product sophistication. More autonomy can remove useful friction for routine work. It can also move decisions away from immediate human oversight and give a mistaken plan more time, more context, and more authority to develop. The relevant design question is how much freedom is justified by the task, and what hard limits remain if the model’s reasoning is compromised.
A boundary around the agent, not a promise about the model
Google’s answer is a hybrid approach. It combines traditional, deterministic security controls with reasoning-based defenses that use models to recognize context and manipulation. The distinction matters. A model-based defense can improve the quality of judgment, but it remains part of the probabilistic system. An external control can enforce a limit more predictably, but a simple rule may not understand the meaning of an action in context.
Imagine a system that can send email after reading a document. A static rule could block every such message. That would reduce one route to exfiltration, but it would also prevent a legitimate workflow in which the user asks the agent to summarize a document and send the summary. A model may be better positioned to interpret the difference. Yet if the same model is fooled by an instruction inside the document, trusting its interpretation as the only safeguard leaves the email capability exposed. The useful design is not to choose rules or reasoning in the abstract. It is to put a dependable decision boundary around the side effect while using contextual defenses to make the decision more informed.
The boundary begins with three relationships that Google presents as core principles for agent security.
First, an agent needs a well-defined human controller. The system must know whose authority it is using and must be able to distinguish that person’s instruction from external content. Critical or irreversible actions should require explicit confirmation. This is not merely a user-interface preference. It determines who can authorize a financial transaction, a security-setting change, or a large deletion, and whether that authorization can later be inspected or revoked.
Second, the agent’s powers must be limited. An assistant designed for research should not automatically be able to modify financial accounts. Google extends ordinary least privilege into runtime context: a general-purpose agent should have only the permissions relevant to the current task. The system should define and enforce a maximum authority level, prevent the agent from escalating itself, and use mechanisms such as granular authorization and scoped credentials to keep delegated power bounded.
Third, actions and planning should be observable. Users and security teams need to know which inputs were received, which tools were called, what parameters were passed, and what outputs were generated. Action metadata—such as whether an operation is read-only or changes state—helps automated controls and human reviewers assess consequences. Logs and planning information must themselves be protected, because visibility into an agent can expose sensitive data if it is collected carelessly.
These principles do not make the model infallible. They change what an infallible model would have to accomplish. Instead of asking it to understand every possible attack and never make a consequential mistake, the surrounding system can limit the actions available to it, pause before high-impact effects, and preserve evidence for correction. The model still needs to recognize trustworthy instructions and reason about context, but its failure no longer has to equal unrestricted authority.
The point of this model is not to imply that one box solves security. It is to show where responsibility must remain visible. The user supplies a goal, but the agent also receives surrounding context. The reasoning core proposes a path, but a boundary should stand between that proposal and an external side effect. Memory can carry consequences forward, and rendering can create new exposure after the model has finished generating text. Security is distributed across those transitions.
The question that governs trustworthy autonomy
Google’s introduction calls the two most important outcomes rogue actions and sensitive-data disclosure. A rogue action may be a harmful command inserted into a document, but it may also be an ordinary misunderstanding of an ambiguous request or a complicated website. Disclosure may occur because a tool was used to retrieve private information, because a later action carried it out of the system, or because a rendered URL or piece of markup exposed it. These outcomes overlap, and their severity depends heavily on what the agent was allowed to access and do.
That leads to a more useful standard than “the model is aligned”: can the system keep an understandable human in control, restrict authority to what the current task needs, and make the path from input to consequence inspectable? A secure agent is not one that never encounters hostile content or never makes a wrong prediction. It is one whose design prevents every wrong prediction from becoming an unbounded event.
The next step is to follow that design through the agent’s operating path. Once perception, prompt assembly, reasoning, orchestration, memory, tools, and rendering are treated as separate transitions, it becomes possible to ask where trust changes—and where a control boundary has to intervene.