From Pages to Decisions: Accessibility Snapshots and Element Refs
Once camofox-browser has reached a page, its next problem is representation: what should an AI agent receive so it can make a good decision without being buried in raw HTML? This chapter explains how Camofox uses accessibility snapshots, element references, bounded continuation, and structured extraction to turn a changing page into actionable evidence. The important judgment is not whether the representation is small; it is whether the agent can tell what a control means, act on the current page, continue reading when necessary, and recognize when the available evidence is incomplete.
The page is represented by its accessible surface
Raw HTML is a record of everything in a document: layout wrappers, hidden templates, scripts, tracking elements, style hooks, and content that may not be visible or meaningful to a person using the page. It is also a poor default conversation format. Giving an agent the entire DOM increases the amount of text it must inspect and leaves it to infer which nodes are actionable. More data does not automatically produce a better next action.
Camofox's ordinary-page path starts with a Playwright accessibility snapshot. An accessibility tree describes the page through roles and names that correspond more closely to how a user or assistive technology encounters it: a search box, a button, a link, a heading, or a form control. The server reads that tree, identifies interactive roles, and assigns references such as e1, e2, and e3. The returned snapshot is then annotated with those references so the agent can point to a control it has actually observed.
The reference is not an arbitrary address into the document. It is a compact handle for a role, an accessible name, and the occurrence of that combination. If a page contains several links named “Details,” the server records which occurrence a reference represents. When the agent later asks Camofox to click e4, the server resolves the stored role and name and uses the occurrence index to disambiguate the locator. This is why a ref is usually safer than asking a generic CSS selector to match a page that contains several similar elements.
The distinction between semantic identity and physical identity is important. A ref does not mean “the element at this permanent coordinate.” It means “the control with this observed role and name, at this observed occurrence, in the current page state.” If the page is rebuilt by a single-page application, a modal appears, a list is resorted, or navigation occurs, that relationship can change. The ref is stable-looking within a useful window of observation, not stable forever.
The server also considers child frames. Some valuable controls, such as payment fields, can live inside an iframe rather than the main document. Camofox processes a bounded set of frames, records frame information with the ref, and can resolve a ref through the matching frame. It skips frames associated with analytics, tracking, pixels, and short-lived challenge systems. That filtering is not a claim that those frames are unimportant to the website; it is a reliability choice for the agent surface. Trying to snapshot every transient frame can make ref building hang or fail while adding little actionable information.
Google search results illustrate another boundary. Search-result pages can change their structure and can contain overlays that make a general accessibility pass inefficient. Camofox uses a specialized extraction path for Google SERPs, finding result headings and links directly and assigning refs to the extracted result surface. The broader lesson is that an agent-oriented interface can remain conceptually consistent while using site-specific mechanisms underneath. The agent asks for a snapshot and refs; the server does not need to pretend that every site exposes them through one identical internal path.
The relationship in the diagram is deliberately not a one-way conversion from page to text. The page produces an observation and a ref table; the agent uses those refs to request an action; a page change creates a refresh boundary. The structure summary and extraction schema are supplements to the observation, not replacements for it. Each object answers a different question: what is visible and actionable, which handle identifies it, what bounded form or table structure is present, and what values can be resolved from explicitly named refs.
Smaller than HTML does not mean less rigorous
The project describes accessibility snapshots as roughly 90 percent smaller than raw HTML. That number is a first-party estimate, and the repository does not provide a measurement method that would let a reader treat it as a general benchmark. The stronger, directly evidenced claim is about the shape of the representation: it uses an accessibility-oriented surface and gives the agent refs instead of requiring it to navigate every DOM node.
The smaller representation also has a practical limit. A large catalog, search result, or social feed can still produce more accessible text than one model request should carry. Camofox handles this with snapshot windowing. If the snapshot is within the configured limit, it is returned as a whole. When it exceeds 80,000 characters, the service returns a head portion, a truncation marker, and a tail reserved for navigation information. The marker includes the offset at which the next request should continue.
Preserving the tail is a subtle but important design choice. A naive truncation would keep the beginning of a product list and discard the controls that move to the next page. Camofox keeps roughly 5,000 characters from the end so links such as “Previous” and “Next” remain available even when the middle of the page is omitted from the first response. The agent can therefore see the beginning of the current result set and still discover how to continue.
An offset request is not a new page read. When the tab has a cached snapshot, Camofox retrieves a different window from that same stored representation and preserves the navigation tail again. This gives the agent a continuation operation rather than forcing it to choose between an oversized response and a second browser action. The total character count and next-offset metadata tell the caller whether it has reached the end.
Consider a page with hundreds of product cards. The first window may contain the page heading, search control, and early products, followed by a marker and the pagination controls. The agent can inspect the first results, decide that it needs later entries, and request the next offset. It does not need to click “Next” merely to read the omitted middle. If it does click a pagination link, however, that is a new page state; the cached snapshot must no longer be treated as current. Navigation and actions invalidate the old observation so the next snapshot reflects the new page.
This distinction turns context efficiency into a correctness issue. The point of windowing is not just to save tokens. It is to keep the agent's evidence legible while preserving a path to more evidence. A short response that silently omits the only useful control would be efficient but operationally poor. A bounded response that says what was omitted and how to continue is a more honest interface.
Structure summaries expose useful shape, not arbitrary content
Accessibility roles are excellent for finding controls, but an agent may also need to understand a form or a table. Camofox can attach a bounded structure summary to a snapshot. The summary describes forms, fields, submit controls, and tables within explicit limits on how many objects, rows, cells, options, and characters are returned. It resolves labels through accessible names, associated labels, titles, or names where available, and it can attach an existing ref to an unambiguous control.
That supplement is useful for a practical distinction. A snapshot may tell an agent that a page contains a textbox and a button. A form summary can show how those controls are grouped, whether a field is disabled, what options a select offers, and what table headers and visible rows are present. The agent can use that bounded shape to decide whether it has found the intended form rather than treating every textbox on the page as equivalent.
The summary also avoids returning values from fields whose type or labels look like passwords, secrets, tokens, API keys, authentication credentials, or payment details. This is a sensible reduction of accidental exposure, but it is pattern-based. A field with an innocent name may still contain sensitive information, and a sensitive field may not be named in an obvious way. The suppression rule is therefore a protective boundary around the convenience representation, not a complete data-loss-prevention system.
The same principle applies to tables. Returning a limited number of rows and cells can let an agent understand a table's shape and inspect representative data without moving an unbounded page into the model context. It does not prove that the omitted rows are identical, nor does it constitute a complete export. When the summary says that rows or forms were truncated, that limitation is part of the evidence the agent must consider.
Explicit refs make extraction deterministic
Camofox's structured extraction route adds another layer to the same model. An agent first observes the page and obtains refs. It then supplies a constrained object-shaped JSON Schema whose properties may point to those refs with x-ref. The server validates the schema, looks up the referenced control, obtains its accessible name, and can coerce that result to a simple type such as string, number, integer, or boolean.
This is intentionally narrower than a general scraping language. The schema says which properties matter and which observed refs should supply them. If a required property cannot be resolved, extraction fails with an explanation rather than returning a plausible-looking empty record. That behavior matters because an empty value can be mistaken for a real page value, while an explicit failure tells the agent that its observation, ref, or page state needs attention.
Suppose an agent has observed a result card with a link named “Acme Coffee” and a nearby price. It can define properties for the title and price and associate them with the refs it saw. The extraction result is deterministic relative to that ref table. If the page rerenders and the refs disappear, the required extraction should fail; the correct remedy is a fresh snapshot and a new mapping, not an invented value based on the previous card.
There are limits to what this route establishes. The implementation is anchored in accessible names and a bounded structural view. Type coercion can turn a text representation into a number or boolean, but it does not understand whether the price is the right product's price, whether a row is currently visible to the user, or whether a label carries hidden business meaning. Determinism of lookup is not the same as semantic correctness of the task.
This is the recurring design pattern in camofox-browser: make the agent's next step explicit and bounded, then expose failure when the evidence no longer supports it. Accessibility snapshots reduce irrelevant detail. Refs tie actions to observed controls. Windowing makes continuation visible. Structure summaries describe a limited shape. x-ref extraction makes the requested fields explicit. None of these mechanisms removes the need for the agent to inspect the returned state.
The representation has a freshness boundary
A Camofox snapshot should be treated as a current observation, not a permanent page model. That is especially important for dynamic websites. The server can rebuild refs after an action and refresh them when a caller's observation is no longer current. The agent's safest pattern remains simple: observe, choose, act, and observe again.
The boundary also clarifies what Camofox is and is not promising. It is not returning the entire truth of a web page in a compressed package. It is presenting a practical semantic surface from which an agent can make its next decision. Some content will be absent because it is not accessible, interactive, or within the selected bounds. Some values will be withheld because they look sensitive. Some refs will become invalid because the page changed. Those are not edge cases to hide; they define the conditions under which the representation is trustworthy.
The next challenge is consequently not another way to read a page. It is acting on one while the page is moving. A ref can become stale, and an action can change what should be observed next. The control loop must therefore bring the snapshot and the action back together without treating a bounded observation as a permanent instruction.