Making a Page Legible to a Model
Browser Use does not hand a language model a raw web page and hope that the model will reconstruct the interface. This chapter examines how its browser session turns DOM structure, accessibility information, layout, screenshots, tabs, and loading signals into a selective state packet, then turns part of that packet into the numeric indexes an action can target. Understanding that representation lets the reader distinguish what the agent currently knows from what it is inferring—and see why page representation is a reliability constraint, not just a prompt-formatting choice.
The state packet is a selective model of the page
Raw HTML is a poor description of what a person can act on. It may contain hidden templates, duplicated mobile and desktop controls, elements covered by another layer, values maintained by JavaScript rather than by an HTML attribute, or a custom component whose visible meaning is expressed through an accessibility tree. It also says little about where the user currently is in a long page, which tab has focus, whether a request is still loading, or whether the page is a PDF viewer that the browser has downloaded separately.
Browser Use’s BrowserStateSummary is designed for the model rather than for a browser engineer inspecting every byte. It can carry the current URL and title, open tabs, a serialized DOM state and selector map, a screenshot, viewport and page-scroll information, pending network requests, recent browser events, pagination cues, PDF status, closed-popup messages, and a safe explanation when the state could not be captured. Each field answers a different question about the environment.
Page identity prevents the model from confusing an old result with the current one. Tab information matters when a click opens a new page or when a task moves between a search result and a detail view. Page geometry tells the agent whether content lies above or below the current viewport. Pending network requests and page statistics can signal that a visually incomplete page is still loading rather than empty. A screenshot can show a layout, overlay, icon, or visual relationship that a text representation cannot express. These signals do not replace the interactive tree; they give that tree context.
The serialized DOM is itself selective. Browser Use gathers several browser signals before producing it. The DOM tree supplies element names, attributes, parent-child relationships, frames, and shadow roots. The accessibility tree contributes roles, names, and properties such as selected, expanded, disabled, or checked. The snapshot and layout data contribute visibility, bounds, paint order, live input values, scroll rectangles, and other information needed to decide whether an element can actually be targeted. JavaScript click-listener information can help identify controls that do not look interactive from their tag alone.
This combination is easier to understand through a form. An <input> may have no useful value attribute because a framework filled its live value after rendering. A button may have a short or empty text node but a meaningful accessible name. A menu item may be visible only after a previous action changes the page. A screenshot may reveal that a seemingly valid control is covered by a modal. A useful state packet brings those clues together without asking the model to parse every implementation detail of the page.
The signals are complementary because each one loses a different kind of information. The DOM is good at structure but can include controls that are hidden or covered. Accessibility data is good at naming the control as a person or assistive technology would encounter it, but it does not by itself describe the full visual arrangement. Layout data can tell the runtime whether an element is on screen and where it sits, but coordinates do not explain whether two similar-looking controls have different meanings. A screenshot exposes visual context, yet it can be hard for a model to use a screenshot alone as a stable selector. Browser Use’s observation layer is therefore a negotiated middle ground: enough structure to target, enough semantics to interpret, and enough visual context to resolve ambiguity.
The page summary also carries negative evidence. A model can be told that there are no usable interactive elements yet, that more content lies below the viewport, or that a network request is still in flight. Those signals do not say what the page will eventually become, but they prevent a premature conclusion that a missing control does not exist. In a dynamic site, knowing that the current observation is incomplete is itself part of knowing the state.
The model-facing prompt then adds a readable account of those structures. It reports page statistics such as links, interactive elements, iframes, images, scroll containers, total elements, and text density. It marks the current tab when identity is unambiguous, describes content above and below the viewport, and places interactive elements in a tree-like representation. If the page is a PDF viewer that cannot be rendered as ordinary page content, the prompt tells the agent to use the downloaded file instead of treating the viewer as extractable text. A file path and a browser state are different forms of evidence, so the runtime keeps them distinct.
This design has a useful consequence: “the page” is not a single object in the agent’s context. There is a live browser environment, a collected state snapshot, a compact textual or multimodal presentation of that snapshot, and the model’s interpretation of it. The snapshot can be stale by the time the model acts. The presentation can be truncated. The model can misunderstand a label. Keeping those layers conceptually separate makes it possible to ask which one failed instead of attributing every mistake to the language model.
The selective design is also a practical necessity. Passing every node, attribute, style, and script into a model would waste context on information that does not help choose the next action. Browser Use simplifies and optimizes the tree, filters elements according to visibility and paint order, and limits the length of the interactive representation. The result favors action-relevant evidence over completeness. That is a trade-off, not a claim that the agent receives a perfect replica of the page.
From browser signals to targetable indexes
The key interface between the state packet and the action layer is the selector map. During serialization, Browser Use first builds a simplified tree, optionally applies paint-order filtering and bounding-box filtering, optimizes away unnecessary parents, and then assigns indexes to elements judged interactive and targetable. An index is therefore not a line number in the HTML and not a universal identity for an element. It is a handle into the current serialized state.
An element generally needs to be recognized as interactive and visible before it receives a model-visible index. There are deliberate exceptions. File inputs can be hidden by styling while remaining functional. Form controls inside shadow DOM may lack ordinary snapshot layout data even though they can be used. A scrollable container can be represented when the agent needs to scroll it, and dropdown-like containers can remain targetable even when they contain other interactive descendants. These rules reflect a practical question: which object can the agent safely name in the next action, given what the browser currently knows?
The serialized tree makes those choices visible. Interactive elements appear with numeric markers such as [index], and newly appearing elements can be marked so the model notices that the page changed after its last action. Scrollable regions carry scroll information rather than looking like ordinary buttons. Iframes and shadow roots are labeled so the agent can reason about a boundary rather than assuming that every child belongs to the main document. The representation is compact enough for a model to scan, but it retains cues about why a control is available and what lies beyond the current viewport.
The index must be used in the context in which it was observed. Suppose a search form initially exposes a location field and a submit button. After the location is typed, an autocomplete menu appears and the old button or field ordering changes. Reusing the original numeric index without reading the next state could select the wrong element. Browser Use’s prompt rules therefore tell the model to use explicit current indexes, inspect new elements after an interaction, and wait when the page has not yet loaded enough structure. The index makes action selection concise; it does not make the page static.
The same distinction explains why Browser Use stores richer records for interacted elements. When an action is written to history, the runtime can retain the element’s frame identity, attributes, XPath, bounds, accessibility name, and hashes. The full hash captures structural and identifying information. A stable hash removes transient CSS-state classes such as focus, hover, animation, or selected states. If a historical action is replayed on a changed page, the runtime can try an exact match, then a stable match, XPath, accessible name, and unique attributes. This cascade gives replay several clues instead of betting everything on one selector.
Those clues solve different problems. An exact hash is useful when the page is effectively unchanged. A stable hash can survive a visual state change. XPath can help when the structure remains recognizable. An accessibility name can remain meaningful when a single-page application rebuilds a menu. An attribute match can support older history records. None of them proves semantic identity. Two “Continue” buttons can have different consequences, a structural path can point to a newly repurposed element, and a visually similar control can represent a different account or transaction. If matching fails, the runtime can report that it could not find a suitable element; it cannot prove that the intended business target still exists.
Loading and visibility are part of the same grounding problem. The state prompt can describe a page as apparently empty when it contains very few elements, or warn that network requests are still in flight while little text has rendered. Page information can show that more content lies below the viewport. A model can then choose to wait or scroll rather than infer that the page has no result. In replay paths, Browser Use can wait for a minimum number of interactive elements when the historical action implies that the page needs time to build its controls. That is a modest but important shift: waiting becomes a response to observed state rather than an arbitrary delay copied from a script.
Frames and shadow DOM show why browser state has boundaries. Same-origin and cross-origin content may require different processing paths, and the browser profile places limits on how many iframe documents and levels are processed. Open shadow roots can be represented, while closed roots remain a meaningful boundary. An iframe may expose a scrollable document that is not visible in the main page’s simple structure. Treating these regions as explicit relationships helps the model avoid assuming that a control is missing merely because it is not a direct child of the main document.
The representation still has hard limits. Dynamic content can arrive after capture. A screenshot may be unavailable, disabled, or insufficient to identify a control. The DOM can be incomplete while the page is loading. The interactive text can be truncated at a configured limit, and a closed component may remain opaque. Paint-order filtering can remove content that is technically present but not currently actionable. These are not defects that a more confident prompt can eliminate. They are conditions under which the application may need to wait, narrow the task, expose a custom tool, or escalate to deterministic code.
The result is a disciplined but incomplete contract. Browser Use tells the model, in a compact form, what the current page appears to contain and which elements are currently targetable. The model uses that account to choose an action. The runtime then checks the browser again and, when needed, remaps historical targets against the new state. The next chapter follows that target and that intention into the tool registry, where a model-selected action becomes a validated browser operation rather than a line of prose.