A Correct Coordinate Is Not a Correct Action

RunAgent Super Browser addresses the last and most deceptive boundary in browser automation: turning a visual intention into an action that reaches the right interactive owner and produces the intended state change. The focus is not how to move a pointer to a rectangle, but how to resolve semantic ambiguity, cross interface boundaries, and verify the result. That distinction gives us a way to tell a perception error from a dispatch error and a successful event from successful task progress.

A language model may say “open the region filter,” and a vision model may draw a box around the row that contains it. Between those two descriptions lie several decisions. Which element inside the box owns the behavior? Is it visible and enabled? Is it inside an embedded document? Will the page accept the event being sent? If the event is accepted, did the requested control actually open? A robust action layer has to answer each question at a different interface.

The target is a semantic object, not a rectangle

The target resolver in Super Browser begins with the center of a vision-generated box. It inspects the interactive elements at that point and looks for an element whose text or accessible label matches the expected target. Center probing is fast when the box is well aligned. It is not enough when a vision region contains empty space, several nested elements, or a visible label next to a separate control.

The recurring compound-row example makes the ambiguity visible. On a flight-search page, a row displays “United States” and a small downward chevron. The vision model returns one region spanning both. The visible words are the right description of the row, but the text itself may not respond to input. The chevron is the control that expands the regional choices. A center-based resolver can therefore hit the label, observe no meaningful change, and repeat the same mistake. In the reported trace, the chevron-aware resolver instead identifies the expandable candidate and the dropdown reveals 51 regions.

Super Browser handles that case by scanning additional points when the center does not yield a matching interactive element. Candidate scores combine how much of the vision box an element occupies, how well its text or accessible label matches the expected target, and whether its attributes or glyphs suggest expansion or toggling. A small semantic sub-element can consequently outrank a large, visually dominant label—but only under a bounded rule intended to prevent the semantic boost from winning when the candidates are not genuinely comparable.

This is a useful separation of responsibilities. The vision model says, in effect, “the relevant interaction is somewhere in this region.” The DOM layer supplies candidates that can actually receive an event. The expected label and the active subgoal decide which candidate means “the right one.” The resolver then returns a concrete owner and position. Each boundary can fail independently. A high-confidence box can contain the wrong owner; a perfect label match can describe a disabled control; a valid interactive element can still be irrelevant to the current subgoal.

The same problem appears when the page is nested.