From Task to Action: Inside the Agent Loop
Browser Use’s distinctive behavior comes from how its Agent moves a task through a sequence of owned states. This chapter follows that sequence—from configuration and browser startup to model output, action results, history, and completion—so the reader can tell which part of the system is observing, deciding, executing, or recording at each point. That distinction is useful when a task succeeds, but it becomes essential when the browser changes, a tool fails, or the final result is incomplete.
A run begins by assembling a control system
The first important event is not the model’s first answer. It is construction of the runtime around the task. An Agent receives a natural-language task and can receive a language model, a browser session, a tool registry, sensitive data, file paths, output schemas, and settings for vision, retries, timeouts, planning, and history. If the caller does not provide a model, the constructor selects a configured default or creates the project’s Browser Use model wrapper. If the caller does not provide a browser session, the agent creates one from the browser profile.
That arrangement gives each major object a different responsibility. The agent owns the objective and the control state. The browser session owns the live connection to pages, tabs, profiles, navigation, and browser events. The tool registry owns the actions that the model is allowed to request and the schemas used to validate them. The message manager turns the current task, prior results, browser state, and available capabilities into model messages. History preserves a trace of the interaction. The model interprets the task and proposes the next action, but it does not own the browser or decide what information is technically available.
This assembly is also where the first boundary decisions are made. Vision can be enabled, disabled, or made conditional. A caller can provide a fallback model, a final structured-output schema, a custom system message, or a maximum number of actions per step. A browser profile can determine whether the session is local or remote and which domains are permitted. The result is not a model attached to an arbitrary browser; it is a configured control system whose model, environment, action surface, and state store are meant to agree.
Before the main loop starts, Browser Use may detect a URL in the task and turn it into an initial navigation action. Initial actions are executed before the model begins its normal reasoning cycle and are saved as a minimal history item. That choice matters for continuity: opening the starting page is still part of the run’s trace, but the agent does not need to spend a model step deciding to perform an action that the task already made explicit.
The main lifecycle can be seen as a set of transitions rather than a list of functions:
The diagram’s ownership is the key. The browser supplies observations; the message manager packages them; the model proposes; the registry validates; the browser effect happens through a tool; and post-processing decides what persists into the next iteration. This division does not remove judgment from the model. It prevents the model’s judgment from being mistaken for direct ownership of every layer beneath it.
One step has four different jobs
A normal step begins by preparing context. The agent asks the browser session for a current state summary, typically including a screenshot even when vision is not ultimately sent to the model, then checks for newly available downloads and stop or pause signals. It updates action models for the current page, obtains any page-specific tool descriptions, incorporates skills if configured, and renders the current plan. The message manager then combines the task, the previous step’s model output and results, the browser state, available files, sensitive-data context, and other stateful material into the messages sent to the model.
This phase is more than prompt assembly. It is the point at which the runtime decides what the model is allowed to know about the current environment. The previous action result can include a short-lived extraction, persistent memory, or an error. A screenshot can provide visual information that the serialized page does not express well. A plan can keep a long task oriented toward a larger goal. Context compaction can reduce old message history when the conversation grows. Loop-detection and budget warnings can be added as nudges. The next action is therefore conditioned not only by the webpage, but by a managed account of what the agent has already tried.
The second job is model interaction. Browser Use asks the configured LLM for structured output rather than treating free-form text as executable instructions. The output can carry an evaluation of the previous goal, memory, a next goal, optional plan updates, and one or more typed actions. The agent applies retry logic and a model-call timeout around this request. For selected provider failures, a configured fallback model can take over for the remainder of the run. These mechanisms address availability and formatting problems; they do not guarantee that the model’s interpretation of the task is correct.
The third job is action execution. The agent passes the model’s action objects to the tool layer, which validates their parameters and supplies runtime context such as the current browser session or file system. It can execute more than one action in a step, but the action queue is not treated as a blind batch. Browser Use can stop remaining actions when a registered action is marked as page-changing, or when the current URL or focused browser target changes after an action. That protects later actions from using assumptions about a page that no longer exists. The detailed tool contract belongs to the next chapter; here the important point is that execution is a separate phase with its own checks and result channel.
The fourth job is post-processing. After tools return, the agent tracks downloads, updates plan state from the model output, records action and page fingerprints for loop detection, changes the consecutive-failure count, and logs completion when a result marks the task as done. A single action error can contribute to the failure count; a multi-action step has additional page-change and loop behavior. If the failure budget is exceeded, the run can stop, possibly after a final response attempt if that option is enabled. If the done action reports success, the agent can invoke an optional judge and a completion callback before cleanup.
This separation makes failures more diagnosable. A task can fail because the browser state was incomplete, because the model produced invalid or misguided output, because a tool could not execute, or because the run reached a timeout or failure boundary. These causes may look identical from the outside—“the task did not finish”—but they imply different remedies. Waiting for a dynamic page addresses an observation problem. A narrower tool schema addresses an action problem. A new plan addresses a reasoning problem. Requiring confirmation addresses a side-effect problem. The lifecycle gives the application a place to make those distinctions.
A simple appointment search shows why the distinctions matter. The initial action may open the service’s page, producing a history item before any model-selected step. The next observation might show a location field and a date control. The model proposes inputs; the tools execute them; and the result may say that suggestions appeared or that a new page loaded. If the page changes after the first input, a queued click should not be treated as valid merely because it was valid in the model’s original view. The agent records the partial result, captures the new state on the next step, and gives the model a chance to select the newly exposed control. If the site instead returns an error, the same result channel tells the next step whether to wait, go back, search differently, or stop.
The transition therefore has both a time dimension and an authority dimension. BrowserSession reports what exists now. AgentState decides what the run believes it has tried. The tool registry decides whether a proposed operation has a valid shape and can receive the required runtime objects. Post-processing decides what becomes evidence for the next iteration. Keeping those authorities distinct is what makes a trace interpretable; collapsing them into one model transcript would make it difficult to tell whether a failure came from the site, the model, or the runtime boundary.
What persists between steps
The agent’s continuity comes from several forms of state, not from a single undifferentiated memory. AgentState holds control information such as the current step number, consecutive failures, the last model output, the last action results, plan items, the current plan position, pause and stop flags, message-manager state, file-system state, and loop-detector state. This is the state needed to decide what the agent should do next.
The browser session holds a different kind of truth: the current pages, tabs, URLs, browser profile, focused target, cookies or storage state, downloads, and connection status. It can persist across follow-up tasks when configured to stay alive. That persistence is useful because a second task can continue from an authenticated or already navigated browser, but it also means that a follow-up task inherits environmental state. A design that treats each task as independent while reusing a browser is likely to miss that coupling.
The message manager holds the evolving conversation representation. It can preserve prior action results and memories while controlling how much history enters the next model call. An extracted page can be shown for one step, summarized into longer-term memory, or written to a file when it is too large for comfortable context. Sensitive values can be filtered during message and history handling. This is not the same as a perfect memory of the browser: it is a managed context designed to support the next decision.
History is the durable trace of what the agent did. Each history item can contain the model output, the results of its actions, a browser-state record, screenshots or screenshot paths, and step metadata such as timing. The resulting AgentHistoryList exposes visited URLs, screenshots, action names, extracted content, errors, model outputs, action results, step counts, durations, final results, and usage information. In other words, the final answer is only one view of the run. The history can show whether the agent reached the intended page, whether a tool returned an error, how many steps were used, and what evidence preceded completion.
That trace changes the meaning of “success.” The done action can mark a task as complete, and an optional judge can assess the trace, but the history does not independently verify every external consequence. A form submission may have returned without the expected confirmation. A site may have accepted a value different from the one the user intended. A cloud or provider failure may have stopped progress before the task’s real objective was reached. A history is therefore an operational instrument for inspection and recovery, not a substitute for a task-specific definition of correctness.
Planning and recovery also live in this persistent layer. The agent can keep a plan with pending, current, completed, or skipped items; inject a re-planning nudge after stalled progress; compact messages; switch to a fallback model; and stop after configured failure or step limits. Loop detection records repeated action patterns and stagnant page fingerprints, then warns the model when repetition becomes concerning. The detector is deliberately advisory: it adds context but does not block an action merely because it resembles an earlier one. A repeated action may be legitimate progress, so the runtime leaves the final interpretation to the model or application policy.
The lifecycle consequently ends in one of several meaningful states: a verified-enough completion according to the task’s own criteria, a model-declared completion that still requires external verification, a bounded stop caused by failures or maximum steps, or an interrupted run whose history is available for inspection. Those are not interchangeable outcomes. Browser Use provides the transitions and records the evidence; the application must decide which terminal state is acceptable for the work at hand.
The next dependency is now clear. Every model decision in this loop rests on the browser-state packet prepared at the start of the step. To understand why an index, screenshot, loading warning, or tab identifier appears in that packet—and why each can still be incomplete—we need to examine how Browser Use makes a live page legible to the model.