Why Deep Search Exposes the Browser Agent's Weak Spot
This chapter examines BrowseComp-V3, a benchmark built to test whether multimodal browser agents can preserve an evidence chain across pages, images, and search hops. Its practical value is not a new leaderboard alone. By separating complete-task success from intermediate progress, the benchmark helps us ask where an agent lost continuity: in perception, in evidence identity, in planning, or in the transition from one page state to the next.
That distinction matters because a deep-search agent can be locally competent and globally unreliable. It may find a person in one image, identify an event in a text result, and still attach the final statistic to the wrong participant. A final answer gives little indication of where that chain broke. BrowseComp-V3 responds by making the chain itself part of the evaluation.
Designing a task that cannot be solved by a shortcut
A conventional visual question might show one image and ask for a directly visible fact. A browser agent can sometimes answer such a question from its internal knowledge, from a superficial text match, or from one well-chosen retrieval. BrowseComp-V3 is designed to make those shortcuts inadequate. Its 300 hand-crafted questions span 24 secondary domains and five primary categories—Science, Technology, Society, Culture, and Life—and the collection contains 383 images. Critical evidence is deliberately distributed across text and visual material, sometimes within one page and sometimes across several pages.
The benchmark treats cross-modal difficulty as more than the presence of an image. At the first level, the agent must align information within a visual region. At the next, it must integrate separate regions. At the highest, it must reason about relationships across images. The task distribution reported by the authors contains 89 Level 1 questions, 140 Level 2 questions, and 71 Level 3 questions. Search depth adds another dimension: a clue found on one page may identify the entity to look for on the next, while a later image or result supplies the fact needed to finish the answer. Each transition is a chance for the agent's remembered target and current page to stop referring to the same thing.
This is why the benchmark's unit of difficulty is an evidence chain rather than a page. The agent must preserve at least three relationships while browsing. It must keep identity stable—who or what a clue refers to. It must keep modality connected—what an image contributes that text does not, and vice versa. It must keep purpose current—whether the next search is still serving the active subgoal rather than a plausible but irrelevant interpretation. A system that gets the first two relationships right but loses the third can still look capable in isolated snapshots.
The authors also make searchability part of the design. Critical supporting evidence is required to be publicly accessible through standard search tools. The benchmark favors temporally stable information and concise, objective answers, so a model is not penalized merely because a live page changed after the task was authored or because two evaluators accept different formulations. These restrictions do not make the tasks artificial in every respect; they make failures more interpretable. If the evidence is public and the answer is objectively checkable, a failed trajectory is more plausibly a failure of searching, grounding, integration, or reasoning rather than an unavailable source.
The construction process follows the same logic. Experts first set the domain mix, task hierarchy, hop distribution, and initial examples containing the visual input, question, subgoals, answer, and metadata. Annotators then conduct exploratory searches with text, webpage, image, crop, and reverse-image tools, recording the route and the pivotal subgoals needed to obtain the evidence. Human verifiers reproduce those routes and check coherence, evidentiary support, and answer accuracy. State-of-the-art models are used to filter out trivial examples. The surviving records are converted into a common structured format and then audited by domain experts for safety, privacy, and factual accuracy.
That five-stage pipeline is more than dataset housekeeping. It makes the benchmark's answer traceable to a path that another evaluator can inspect. At the same time, the source does not provide annotator-agreement statistics here, nor does its filtering process prove that every possible shortcut has been eliminated. Public searchability improves reproducibility; it does not guarantee it. The important point is that BrowseComp-V3 tries to control the conditions under which a continuity failure can be observed.
The distinction between coverage and realism is important here. The five categories and three task levels are a deliberate sampling design, not a claim that the open web naturally divides into equal portions of Science, Technology, Society, Culture, and Life. Likewise, a question can be objectively answerable and still be difficult because the relevant clue is obscure, visually small, or separated from the final answer by several decisions. Searchability removes the excuse that the evidence is unavailable; it does not remove the work of discovering which page, image, or entity is relevant.
The annotated trajectory also changes the meaning of a “correct” run. A final answer can be right for the wrong reason, especially when a model's stored knowledge happens to supply a plausible fact. A trajectory that records the evidence-bearing steps gives evaluators a way to ask whether the route actually supported the answer. That does not expose every hidden inference inside a model, and the subgoals remain judgments made by annotators. It does, however, bring the observable interfaces—the searches, page visits, visual inspections, and intermediate findings—closer to the object being evaluated. For browser agents, those interfaces are where state can be checked and where a mismatch can become visible.
What the numbers reveal—and what they do not
BrowseComp-V3 reports two complementary outcomes. Success Rate asks whether the entire task was completed. Process Score is the proportion of expert-defined subgoals the system achieved: the number of achieved subgoals divided by the number required for that question. The second measure is not a softer version of the first. It answers a different question: how far did the agent progress along the evidence path before the run stopped being useful?
Imagine a task with several required links. The agent may acquire the first few clues correctly, then select the wrong image or lose track of which person a pronoun names. Success Rate records a failed task. Process Score preserves the fact that useful subgoals were completed. When Process Score is higher than Success Rate, the gap is evidence of partial competence followed by a breakdown in continuity. It does not, by itself, reveal whether the last error came from a bad visual match, a planning decision, or a stale page representation; it tells us that a final-answer score alone would hide the location of the breakdown.
The reported results show a large gap between human browsing and current model performance. PhD-level human participants using a standard browser reached an average Success Rate of 68.03% and Process Score of 82.93%. The tool-augmented services in the table stayed below 40% Success Rate: GPT-5.2-Thinking reached 39.13%, Gemini-3-Pro-Preview 22.90%, and Claude-Sonnet-4.5-Thinking 18.33%. Tool-free models were generally near or below the low-teens. These are results reported by the benchmark paper, which is an arXiv preprint under review, not an independent audit of every experimental condition.
The comparison with OmniSeeker clarifies the role of external interaction. When models were equipped with the benchmark authors' standardized multimodal browsing framework, GPT-5.2 reached 36.00% Success Rate and 57.70% Process Score; Doubao-Seed-1.8 reached 33.67% and 58.44%. The improvement over tool-free operation supports a limited but important conclusion: internal model knowledge is not enough for current, cross-modal evidence chains on the open web. The result does not support the stronger conclusion that providing search and image tools makes the agent dependable. Tools increase access; they do not automatically preserve the relationships among observations.
The failure analysis makes that distinction concrete. Visual grounding errors and image-perception failures dominate the reported errors across representative models. As task complexity rises from Level 1 to Levels 2 and 3, models struggle more with integrating regions and reasoning across images. The authors also report that closed-source systems reduce some perception and grounding errors, after which long-horizon planning becomes a more prominent bottleneck. Humans show a different limiting pattern in the reported analysis: TextSearch is their main weakness, associated with attention and cognitive load when processing large volumes of text.
The asymmetry is revealing. A person may be able to inspect a crowded page but lose time and attention while searching a large body of text. A model may process text rapidly yet fail to notice that the relevant visual region is a small control, or that two visually similar entities are not the same entity. Neither limitation is captured well by asking only whether the final string matches the reference. The important diagnostic question is what kind of representation failed. Was the evidence never acquired? Was it acquired but not grounded to the right region? Was it grounded correctly but joined to the wrong entity? Or did the agent preserve each local fact and still choose a globally inconsistent next step?
The benchmark's process framing also exposes a subtle evaluation trade-off. More intermediate annotations make a run easier to inspect, but they do not turn the evaluator into an observer of the model's private reasoning. A subgoal can tell us that a required fact was reached or a trajectory step was completed; it cannot, by itself, prove that the model understood every relation in the way a human would. Process Score is therefore best treated as structured behavioral evidence. It narrows the search for a failure and makes comparisons more informative, while leaving room for ambiguity about the internal cause.
These patterns should be read as a map of reported failure modes, not as a universal causal law for every model. Still, they establish an important systems distinction. Better access can solve the problem of not having an observation. Better perception can solve the problem of not seeing a relevant region. Neither guarantees that the observation is attached to the correct entity, that the agent will choose the right next hop, or that the final relation will remain consistent. The benchmark is valuable precisely because it makes those later failures visible instead of collapsing them into a single “wrong answer.”
This also explains why the benchmark is relevant to page-state synchronization even though it is framed as a deep-search evaluation. Every subgoal is a claim about what the agent has learned from a particular page or image. If the page changes, the target moves, or the agent's current screen no longer matches the evidence it remembers, the next subgoal is built on a false premise. A process metric can flag that the chain stopped making progress; a trajectory can then help locate whether the break happened at retrieval, visual grounding, cross-modal integration, or planning.
BrowseComp-V3 therefore supplies the measurement lens for the reliability problem. It asks not only, “Did the agent answer?” but also, “Which links in the evidence path did it actually secure?” That is a more useful starting point than a leaderboard when the goal is to improve the system. The remaining engineering question is how to keep a visual target meaningful as the DOM, the viewport, and the active subgoal change. That requires treating perception as a state-dependent service rather than as a timeless screenshot.