How e2e Sees an App

This chapter examines the semantic observation that sits between tester-army/e2e and the application under test. The key question is what the agent and a deterministic locator can actually identify when a UI is rendered as HTML, a shadow tree, an iframe, an iOS view, or an Android widget. Understanding that shared representation lets the reader choose a locator deliberately, interpret ambiguity as evidence, and know when e2e must use pixels or a platform-specific capability.

The previous chapter separated an adaptive path from an exact proof. Both depend on the same prior step: the framework must obtain a trustworthy description of the current screen. e2e calls that description a semantic observation. It is not a complete copy of the page and it is not merely a screenshot. It is a bounded model of the controls, text, state, and geometry that the runner can safely show, query, and act upon.

The semantic tree is the runner’s common language

A raw page contains far more information than a test needs. It includes layout wrappers, styling rules, scripts, hidden templates, framework-generated IDs, and implementation details that may change without changing what a person can do. A screenshot contains the opposite problem: it shows appearance but does not by itself say which rectangle is a button, which text labels a field, or whether a control is disabled. e2e’s semantic tree sits between those extremes.

A node can carry a role such as button, textbox, link, heading, status, or dialog. It can also carry an accessible name, visible text, a safe current value, a test ID, states such as checked, selected, expanded, disabled, focused, or hidden, and a rectangle in the viewport. The observation carries a viewport and a revision as well. The revision matters because a reference found on one observation is not automatically valid after the app has re-rendered; the runner must know which screen generation produced the reference.

This representation serves two consumers at once. The agent sees a compact, human-readable tree with node IDs and state markers. The deterministic screen layer turns a query such as getByRole('status') or getByLabel('Name') into a locator expression. The engine receives the expression and resolves it against the live surface. Because both consumers use the same semantic vocabulary, the name that helps a model choose a control is related to the name that an exact expectation later checks.

The tree is intentionally bounded. Names and text are limited, observations can be marked truncated, and a semantic capture can be unavailable if the engine cannot safely complete it. A bounded representation is a reliability feature: it prevents a dense or adversarial page from consuming unlimited model context. It also creates a meaningful failure condition. If the observation is incomplete, the agent is told that it is incomplete; the runner does not silently treat the missing nodes as proof that those controls do not exist.

The locator layer is semantic, but it is not vague. A role query asks for a particular role and, optionally, a name or state. A label query asks which form control is associated with a label. A text query looks for visible text, while a display-value query asks what a control currently shows. A test-ID query uses the application’s configured identifier attribute. These queries answer different questions, so choosing one is part of the test’s meaning.

Consider a billing form. getByRole('button', 'Save') identifies an actionable control by its role and accessible name. getByLabel('Card number') identifies an input by the label a user encounters. getByText('Pro plan') may identify a visible explanation or a status line, but it does not necessarily identify the control that changes the plan. getByDisplayValue('Germany') asks about the current value of an input or select, not the label beside it. getByTestId('billing-status') can be useful when the application has a stable test contract, but it says less about whether the interface is understandable to a person.

Matching is strict by default. A role name is not an approximate instruction to find something similar; exact text and whitespace rules apply unless the test explicitly requests a substring or regular expression. That strictness is what turns a query into evidence. If a query matches two controls, e2e reports LOCATOR_AMBIGUOUS rather than selecting whichever appears first. If it matches nothing, an action or assertion waits according to its rules and then reports LOCATOR_NOT_FOUND. A silent first-match policy would make a test appear stable while acting on the wrong row, dialog, or duplicate button.

Queries are lazy by design. Creating screen.getByRole('button', 'Save') does not immediately search the page; the action, read, or matcher resolves it when the test uses it. That matters because an app can change between the moment a locator is declared and the moment the test acts. The query is a description of what must be found now, not a frozen element handle that quietly points into an old screen. Scopes and filters narrow the question further: a row can be selected by the status it contains, or a repeated label can be resolved within a named dialog rather than across the whole page.

The same rule explains why visibility is an explicit part of the contract. A hidden duplicate left behind by a transition should not make a visible control ambiguous when the test explicitly asks for visible content. Conversely, a role query should not be widened to hidden accessibility nodes merely because a hidden copy exists. The runner’s strictness is not a preference for verbose tests; it is a way to make the boundary between “the UI offers this control” and “some implementation node happens to contain this text” observable.

classDiagram class Engine { +observe() +locate(expression) +perform(target, action) } class Observation { +location +viewport +revision +truncated } class SemanticNode { +role +name +text +states +rect +testId } class LocatorExpression { +query +scope +filter +index } class ScreenLocator { +getByRole() +getByLabel() +getByText() +getByTestId() } class AgentTarget { +nodeId +observationRevision } Observation "1" *-- "many" SemanticNode : contains ScreenLocator ..> LocatorExpression : builds LocatorExpression ..> Engine : resolves through Engine ..> Observation : captures Observation ..> AgentTarget : names AgentTarget ..> Engine : action target

The diagram shows why the framework can keep a shared test surface without pretending that a node ID is permanent. An observation supplies the current semantic nodes. A screen query becomes an expression that the engine resolves against that observation. An agent target refers to the node in the current revision, and the engine checks it before acting. The identity is semantic and contextual, not a promise that the same generated ID will survive every render.

Why the same query has different implementations

The shared vocabulary does not mean that browsers and native devices expose the same underlying structure. It means that each engine translates its surface into a common contract while preserving the places where the surfaces differ.

On the web, e2e’s reader derives accessible roles and names from the document. It handles native HTML controls, ARIA roles, associated labels, placeholder-based names where the platform defines them, contenteditable editors, and visible text. It also accounts for shadow roots and same-site child frames. A same-site iframe can contribute a subtree to the observation because it is part of the application’s own surface. A third-party frame remains a boundary; an advertisement, tracker, or unrelated embed is not automatically treated as application content for the agent to read and act on.

The web reader also distinguishes accessibility visibility from visual visibility. A node can be absent from the role tree while still painting, or it can be present in the DOM but not visible or actionable. This distinction matters to both queries and screenshots. A role query should not reach hidden accessibility content simply because the corresponding element exists in the document. A visual check may need pixels, but those pixels are subject to masking rules and are not a license to infer a control that semantic capture did not establish.

Web names are more complicated than the text in one element. An accessible name may come from a label, an aria-labelledby relationship, an aria-label, alternative text, generated content, or the control’s own content. A button with an icon and the word “Login” should be findable by the name a user encounters, not by an implementation-specific class or a private glyph. This is why the framework’s semantic reader is part of the engine rather than an incidental formatting step: the name affects what the agent sees and what a role locator matches.

Native mobile surfaces begin with accessibility snapshots rather than a browser DOM. The mobile engine projects platform element types into the same role vocabulary. A button remains a button; text fields become textboxes; switches, checkboxes, tabs, dialogs, lists, and navigation bars receive corresponding semantic roles. iOS and Android do not report every concept in the same way, so the projection also interprets platform-specific state encodings. React Native may describe a checkbox or radio state inside an accessibility value; Android may identify an editable view through its class and editability rather than through the same metadata a browser would expose.

Geometry matters more directly on a device. A native snapshot supplies rectangles in logical screen coordinates, and the engine preserves them so a tap or swipe can be associated with the same surface the observation described. The engine also filters system bars that would otherwise pollute a test with a changing clock, signal strength, or battery indicator. If a snapshot is sparse while the accessibility tree is still being published, the mobile engine retries instead of presenting a one-node screen as if it were complete. If a control appeared or moved after an action, a transition window gives it time to arrive before a deterministic action targets it.

These differences explain what portability means in e2e. A test that stays within app, screen, agent, and expect can express the same user-facing behavior on a web target and a device target. The engines still do different work underneath. A browser can expose a frame locator and CSS-backed browser fixture; a device may expose a device fixture and reject browser-only actions. The shared vocabulary reduces the amount of platform knowledge in the test, but it does not remove platform capability boundaries.

Portability also depends on what the application chooses to expose. A button with a stable accessible name can be found on both a browser and a native screen even though its underlying element types differ. A browser-only CSS selector, an Android resource ID used directly in a shared test, or a device action that has no web analogue cannot cross the same seam. The engine contract makes that difference explicit: capabilities are declared, and an unsupported operation should fail at the boundary rather than be quietly emulated. The result is less magical but more predictable.

This is why semantic design is not merely a testing convenience. Accessible names, roles, and states are the application’s public interaction language. They help a screen reader, a human using assistive technology, a deterministic test, and an agent identify the same task-relevant objects. A test ID can still be the right choice for a stable status region or a platform-specific surface, but it should be understood as a deliberate test contract rather than a substitute for every missing label. When the app’s public semantics are weak, the test author may be forced toward more fragile or less portable evidence.

When pixels are evidence—and when they are not

Semantic trees cannot describe every surface. A canvas, chart, map, game, image-based control, or system sheet may present an important fact primarily through pixels. e2e can request a masked screenshot when the engine supports it. The built-in agent normally starts with the tree and asks for pixels when the tree cannot supply enough information; a judgment can also request the tree, the tree plus a screenshot, or the screenshot alone when the question is specifically about presentation.

The conservative rule is as important as the fallback. Secure fields must be masked at the source, and the framework checks whether the screenshot’s masking covers the secure nodes it observed. If masking cannot be proven, pixels are withheld. If semantic capture is unavailable and no permitted, proven-masked screenshot can replace it, the observation fails rather than becoming an empty tree that the model might misread. After a secret has been filled, the attempt can become pixel-tainted and visual evidence is withheld for the rest of that attempt.

That behavior may feel restrictive when debugging, but it preserves the difference between “not observed” and “not present.” A model that sees an incomplete tree should not infer that a missing button does not exist. A screenshot that may contain a password should not be retained merely because it would be convenient. e2e’s evidence model accepts less information rather than converting uncertainty or privacy risk into false certainty.

The semantic surface is therefore part of application quality. An app with clear accessible names, stable roles, meaningful states, and useful test IDs gives both the agent and deterministic checks a better basis for action. An app with duplicate labels, unlabeled controls, volatile generated identifiers, or important state visible only in unsafe pixels narrows what the framework can prove. The next chapter follows what happens after this observation exists: how a bounded agent consumes it, how the runner waits for transitions, and how it handles a reference that has already gone stale.

References