Keeping Perception Aligned with a Moving Page
RunAgent Super Browser offers a concrete answer to a difficult browser-agent question: how can a visual target remain useful when the page beneath it is changing? The focus here is the boundary between perception and page state. Understanding that boundary lets us decide when a cached target is safe to reuse, when the system must look again, and why a faster action is not necessarily a more reliable one.
A screenshot is a record of appearance at one moment. A browser action, by contrast, is a request to change the current interface. The system therefore needs more than a model that can point at a plausible region. It needs a way to connect that region to an interactive element, a current subgoal, and a state transition that can be checked afterward.
A screenshot proposes; the page decides
Super Browser treats vision as candidate generation rather than as a complete action instruction. On each screenshot, a vision model returns labeled bounding boxes for regions that might be interactive. The coordinates are normalized to the image and later translated into browser-space positions. This is useful because a model can identify a search field, a filter row, or a submit control without having to enumerate the entire page in language.
But a box around a visible object is not yet a box around a valid browser target. The visual model may include a label and its adjacent control in one region. It may miss the fact that a checkbox is already active. It may recognize an element that has moved, disappeared, or become covered since the screenshot was taken. The box expresses a hypothesis about what matters; the browser's current structure determines whether that hypothesis can be acted on.
Super Browser places a DOM-enrichment boundary between those two facts. The bridge associates a visual candidate with a selector entry and adds state that may not be visible in the screenshot: whether an element is expanded, whether a control is active, the resolved accessible label, and whether an enclosing row has a separate expansion control. The language model still chooses what to do, but it is not forced to infer every interaction property from pixels alone.
This matters most in compound controls. A filter row can display “United States” as ordinary text while a small chevron opens the list of regions. A single visual box spanning the row is a reasonable perceptual result, but it leaves the action layer with an ambiguity: should the target be the label or the arrow? The bridge can split the compound region, and the target resolver can prefer an interactive sub-element carrying expansion or toggle semantics. The intended action is thus determined by a combination of appearance, DOM meaning, and task relevance.
Task relevance is another boundary condition. A page may contain many controls that could serve the broad user request, but only one is useful for the immediate subgoal. Super Browser passes a short description of that subgoal to the vision stage and ranks candidates by local relevance. This prevents the operational worker from treating every visible affordance as equally important. A booking task may require opening a date control now, not selecting a filter that will matter several steps later. The same page can therefore yield different useful candidates as the plan advances.
The system also adapts perception to the page rather than assuming one vision setting fits all pages. Search results, dense filter panels, booking forms, and other control-heavy layouts can cause a small vision model to omit or truncate candidates. Once a page is classified as complex for the current session and URL, the system can use a larger fallback tier. If a validator says that an expected label was missing, it can request a coverage pass with broader candidate coverage and explicit label hints. The paper reports an internal estimate that such passes recover about 5% of otherwise-failed actions, but it does not provide a full independent ablation for that mechanism. The safe conclusion is that perception can be made responsive to observed misses, not that the reported percentage generalizes to all sites.
Freshness is handled similarly. Recent visual marks can be overlaid on a new screenshot to give stable controls a sense of continuity, much as a person remembers where a search field was while a page scrolls. Yet an old mark is not automatically authoritative. The overlay is suppressed when the prior perception is too old or the URL has changed, because a stale anchor can mislead more than it helps. This is a small but important design choice: continuity is useful only while it remains evidence about the current page.
The ownership chain is now clear. The vision model proposes candidate regions. The DOM bridge supplies interaction semantics and state metadata. The active subgoal determines which candidates deserve attention. The worker chooses an action. A post-action validator determines whether the page changed as intended. No individual component can stand in for the whole chain. A high-quality screenshot cannot compensate for stale DOM state, and a valid selector cannot compensate for a wrong subgoal.
The boundary is harder when the apparent page contains another document. Super Browser's snapper can descend into a same-origin iframe, recompute the target in the inner document, and translate the resulting position back into viewport coordinates. A cross-origin frame cannot be inspected in the same way, so the system must use a different frame-walking route or escalate. The distinction is not an implementation footnote: the screenshot may show one surface while the event receiver belongs to a nested browsing context. If the system collapses those contexts into one coordinate system, it can report a well-grounded target while dispatching to the wrong owner.
The same logic applies to custom widgets. A slider or filter may expose a visual surface, a shadow-DOM control, and a value that only becomes authoritative after a change event. A target resolver that stops at the visible rectangle has not yet shown that the intended value reached the widget's state. Super Browser's reported worked cases use DOM-aware handling for such boundaries. They illustrate the general rule without proving that the same fallback succeeds on every custom element: the component that receives the event and the component that owns the resulting value may be different.
A useful operational distinction follows. Perceptual confidence means that the vision model believes a region is relevant. Actionability means that the browser can resolve that region to the right interactive owner under the current page state. Effectiveness means that the resulting event produces the intended mutation or navigation. These are successive conditions, not synonyms. Treating them as one confidence score is how a system turns a plausible box into a silent loop.
Cache only while the page still agrees
The expensive alternative to stale perception is to call the vision model before every action. The expensive alternative to repeated perception is to let a cached target live forever. Super Browser separates these choices into declarative and procedural paths. A vision-grounded action uses the labeled region the model saw. A procedural action uses a previously resolved DOM selector when the system has evidence that the relevant page state has not materially changed.
That evidence comes from a composite fingerprint rather than from one signal. A DOM hash tracks changes in the interactive element set, including additions, removals, and reordering. A rendered-text and scroll hash catches in-place text changes and movement through the page. An iframe signature catches changes inside embedded content that may not alter the top-level DOM. The current URL identifies navigation, while a mutation delta records how much the page structure changed after an action. Together these signals form a practical cache key.
The rule is conditional. If the key still matches the last vision epoch and the mutation delta is within tolerance, a known selector can be reused without paying for a new visual call. If the key has drifted, the previous perception has lost authority. The worker must wait for a prefetched result or request a fresh screenshot and resolve the target again. This is not a claim that fingerprints perfectly describe page meaning. A page can change semantically without changing the signals, and a harmless change can force unnecessary re-grounding. The design accepts those costs to avoid treating a coordinate or selector as timeless knowledge.
Each signal catches a different kind of disagreement, which is why relying on only one would be brittle. A DOM-element count might remain stable while a label changes; a text hash might remain stable while an embedded calculator advances inside an iframe; a URL might remain unchanged while a filter toggles in place. Conversely, a harmless reordering or scroll can invalidate a cache even when the intended control remains usable. The gate is therefore a conservative policy boundary, not a perfect semantic model. It trades some redundant vision calls for a lower chance that the worker will silently apply yesterday's interpretation to today's interface.
This also clarifies why a no-op is valuable evidence. If a target is fired and the page produces no mutation, the system should not immediately conclude that the task requires more persistence. The absence of change may mean that the target was stale, covered, disabled, or semantically wrong. Super Browser's dead-action guard records repeated no-effect attempts and asks for a new perception, while allowing an exception when an active toggle legitimately changes without a large structural mutation. The response to failure is thus not simply “try harder”; it is “question the reference frame.”
Asynchronous prefetch makes the conservative rule less expensive. After a mutating action, the system schedules the next vision pass in the background while the language model reasons about what to do next. An ordinary action can use a short synchronization window; a critical action can wait longer because acting on a stale target would be more costly. If the result is not ready, the worker is told that vision is lagging rather than being given a false impression of freshness. The mechanism overlaps perception latency with reasoning latency, but it does not erase the underlying dependency.
The paper reports the effect on a representative twenty-step shopping task: a naive policy makes roughly one vision call per action, while the gating policy makes about five to seven calls without measurable loss of success rate in that trace. That is a meaningful engineering signal, especially when vision calls consume tokens and wall-clock time. It is not a broad causal proof. The authors do not provide a full component ablation showing that the gating predicate alone produced the result, nor do they guarantee that the chosen fingerprints detect every semantically important mutation.
The same principle becomes stricter after navigation. A successful URL change invalidates the old visual epoch because the previous page's candidate indices no longer describe the page that loaded. This rule protects against a subtle form of confidence error: the browser reached a page, so the agent assumes its previous interpretation still applies. In reality, a redirect, a new result list, or a different layout can make every prior target irrelevant. Re-grounding is not wasted repetition after a transition; it is the step that re-establishes the reference frame.
Direct navigation illustrates why this matters. Many sites encode a location or a set of filters in their URL, so a worker that knows the site's URL grammar can replace a long chain of interface operations with one deterministic hop. Super Browser describes a rabbit-adoption search whose target state could be reached with one structured URL rather than a dozen filter actions. This can remove many opportunities for visual targeting to fail, but it introduces a different risk: a plausible URL can be syntactically valid and semantically wrong.
The safeguards therefore operate at the page boundary. Navigation is pinned to the target domain and its allowed subdomains, blocked or challenged URLs are not blindly retried, regressions to already visited states are detected, and the new page is re-grounded. A direct route that lands in the wrong country or category is not a success merely because the server returned a page. The worker still has to verify that the loaded state matches the intended state. Speed is valuable only when scope survives the shortcut.
The resulting design is a controlled exchange between freshness and efficiency. Fresh perception reduces the chance of acting on yesterday's page, but costs time and model calls. Procedural reuse reduces that cost, but only under a page-state invariant. Direct navigation compresses interaction, but requires knowledge of URL semantics and a strong post-navigation check. The system does not solve the trade-off by choosing one side permanently; it makes the condition for switching paths explicit.
That condition is the essential lesson. A target is not reliable because it was once visible, and a page is not current merely because the browser session is still open. Perception becomes actionable only when its semantics, the page fingerprint, the active subgoal, and the observed transition agree. Even then, the resulting percept is temporary. The next continuity problem is what the agent keeps when raw screenshots and old observations are removed from the live context—and whether that memory preserves the thread without reintroducing stale state.