One Goal, Bounded Execution
This chapter follows one agent.act call inside tester-army/e2e: what enters the step, how a model choice becomes an authorized engine action, why the runner waits for a settled screen, and how failure states stop the model from claiming more than the evidence supports. The practical result is a way to write goals and budgets that are finite, diagnosable, and safe to hand back to a human when the app or the model leaves the expected path.
The semantic observation from the previous chapter is the starting state. It gives the agent names, roles, values where safe, states, node IDs, and optional pixels. It does not give the agent permission to act freely. The runner uses that observation to build a controlled loop in which every action has an owner, a deadline, and a known consequence.
From opening observation to a step verdict
An act step begins with a goal such as “open the Billing tab” or “get the app to greet {name}.” The configured agent receives the instruction, its parameters, the current observation, any bounded ledger of earlier step summaries, and the agent context that explains application-specific vocabulary. The observation is redacted and size-bounded before it reaches the model. The model does not receive raw HTML, cookies, headers, environment values, or the plaintext of a secret. If the semantic tree cannot describe the screen well enough, the built-in agent may request a permitted screenshot, but visual evidence follows the same masking and taint rules as other runner evidence.
The opening observation is not a promise that the model can see everything. It is a statement of what the runner has safely established. A truncated tree tells the model that the listing is incomplete. A screen with no useful interactive nodes may trigger a pixel request. An unavailable semantic capture is not turned into an empty screen, because an empty list could be mistaken for proof that no control exists. This is the first lifecycle boundary: the agent can reason only from the evidence the runner has admitted.
The built-in agent then works through a deliberately small grammar. It can observe again when work is still in progress, tap a listed control, type into an input, press a key, select an option, set a checkable state, scroll, navigate, drag, use a point action when the engine supports one, or call a permitted project tool. The grammar describes intent in terms of the current screen rather than exposing arbitrary selectors. A model response does not become executable code. It becomes a proposed tool call whose arguments the harness validates.
A targeted action goes through several owners before it changes the app. The dispatcher checks the argument shape and the action budget. It resolves the model’s target against the newest observation. The engine receives the target reference and the action kind under an operation deadline. If the engine accepts the operation, the dispatcher records the committed action and arms the appropriate settling policy. The action’s summary is generated from what was actually resolved and performed, not from a model claim that the action probably happened.
The distinction between a target and an action is important. The model can say “tap the Save button,” but the runner must connect that phrase to one semantic node in a particular observation. The engine then decides whether that node is actionable on its surface. On a browser, this may involve visibility, attachment, and page actionability.