Designing the Control Boundary: Humans, Powers, and Visibility

The failure paths around AI agents point to three practical questions: who is actually in control, what authority is active for this task, and what evidence will show what happened? This chapter applies Google’s three security principles—well-defined human controllers, limited powers, and observable actions—to those questions. The goal is not to turn them into a checklist, but to show how they work together when an agent is shared, operating asynchronously, or asked to perform a consequential action.

Control begins with an identifiable human

An agent may act like an assistant, but it does not become accountable simply because a person initiated the conversation. Google’s first principle is that every agent should have a well-defined controlling human user or set of users. The system must reliably distinguish an authorized controller’s instruction from text the agent encounters in an email, document, webpage, or tool result. That distinction establishes the source of authority before the agent starts interpreting the request.

This is more demanding than attaching a username to a session. A real controller needs a meaningful relationship with the agent’s actions. They should be able to understand what the agent is allowed to do, approve an operation when its consequences warrant personal judgment, and revoke authority that is no longer appropriate. If the person cannot see the delegated permission or cannot withdraw it, the account is an attribution label, not effective control.

The threshold for direct involvement depends on consequence. Google gives large data deletions, significant financial transactions, and security-setting changes as examples of critical or irreversible actions that should require explicit human confirmation. The point is not that every tool call should interrupt the user. Requiring approval before a low-risk read operation could make an assistant too cumbersome to use. The point is that an agent should not silently convert a probabilistic interpretation into a high-impact commitment when a human could still inspect the proposed action.

Confirmation also needs to describe the action in terms a person can judge. “Continue?” is weak if the interface does not identify the recipient, the data being shared, the amount of money, or the state that will change. Google’s principle does not prescribe a particular interface, but its emphasis on consent and user understanding implies that approval must attach to an intelligible consequence, not merely to the existence of a model-generated plan.

The distinction becomes harder when several people share an agent. A team assistant may receive a request from one employee but have access to resources belonging to the team, another employee, or the organization. The system needs distinct identities and a clear authorization model so it can tell which user is acting and whose data or authority is involved. A shared agent may need more granular permissions than a single-user assistant. Otherwise, one person’s ordinary request could trigger an action that affects another person without a clear basis for that authority.

Imagine a shared project assistant asked to send a status report. One employee may be authorized to read the underlying project files but not to disclose them outside the organization. Another may be allowed to approve an external message but not to edit the source records.