The Test Has Two Kinds of Truth

This chapter examines how tester-army/e2e combines a flexible agent goal with an exact test oracle. The practical question is how to let an agent choose a path through a UI without letting its explanation decide whether the application reached the required state. By the end, the reader should be able to write a small hybrid test: one bounded goal, one immediate deterministic check, and a model judgment only where interpretation is genuinely necessary.

The distinction is easiest to see in the repository’s greeting example. The test asks the agent to get the app to greet Grace. The agent must identify the name field and the Greet button from the current screen. The test then checks that a status element contains the exact result, “Hello, Grace!” A separate judgment can ask whether the app greets Grace by name, but the exact status check does not depend on the model’s summary.

That is not two competing testing philosophies. It is a division of labor. The agent supplies adaptability at the path-selection boundary; the deterministic layer supplies a stable statement about the outcome.

A goal is a contract, not a chat prompt

An agent.act call is a test step with a bounded purpose. It is not an invitation to explore the whole application, explain what the model sees, or begin the next part of the scenario. The instruction should name one outcome in the vocabulary of the screen: “get the app to greet {name},” “open the Billing tab,” or “submit the form with the provided value.” The runner gives the configured agent that instruction, the step’s parameters, the current observation, and a finite budget. The agent chooses actions, but the test still owns the order of steps.

That one-goal rule has a direct diagnostic benefit. Suppose a test says, “sign in, create a project, invite a teammate, configure billing, and verify the dashboard.” If it fails, the failure could be a missing credential, an ambiguous control, a navigation problem, a delayed save, or an incorrect final assertion. The model may also spend its budget on a path the test author never intended to cover. Separate goals make the boundary visible: the sign-in step can be judged by a dashboard URL, the creation step by a project status, and the invitation step by a member row or confirmation. Each step has one question and one place to look for its answer.

The natural-language instruction is also separate from the values it names. In the greeting example, the instruction can say “get the app to greet {name},” while the test passes { name: 'Grace' } through params. This separation is useful for three reasons. First, it keeps the goal readable: the sentence describes the task, not a string assembled from test data. Second, it lets the runner validate and redact values independently of the instruction. Third, it gives the replay system a stable request shape even when a value is intentionally unique at run time.

The separation does not mean that parameters are invisible or magical. The agent receives the parameter in a distinct block and can use it to fill the named field. A test author still has to say what the value means and where the application should use it. “Get the app to greet {name}” gives the model a goal and a value; “do the greeting thing somehow” gives it neither a reliable target nor a useful failure explanation.

The runner also keeps the agent inside a controlled action vocabulary. The built-in agent can observe, tap, type, press, select, check, scroll, navigate, use point actions where the engine supports them, and invoke permitted project tools. It cannot return a selector string for the harness to execute as code. Each action is validated, resolved against an observation, dispatched through the engine, and accounted for. If the screen exposes two equally matching buttons, the runner reports ambiguity instead of allowing the model to guess. If the target engine cannot perform an action, the test records an unsupported capability rather than silently changing the test’s meaning.

The goal therefore acts as a contract among three parties. The test author defines the intended outcome and supplies any named values. The model chooses a route that the current screen makes possible. The runner decides whether each proposed operation is authorized and whether the step ended within its budget. A model summary can describe the route, but it cannot broaden the contract after the step begins.

The same boundary controls cost as well as diagnosis. A test that uses only app, screen, and expect does not need a model at all. A test that uses one small agent.act call spends model capacity only where the path is genuinely variable. A broad instruction, by contrast, can consume actions and model calls on incidental navigation before it reaches the state the test actually cares about. The one-goal rule is therefore a design constraint with an operational consequence: it limits both the surface area of uncertainty and the amount of external model work a run needs.

It also gives teams a way to migrate gradually. A deterministic test can establish the exact outcome first. An agent step can later replace only the brittle path while keeping the same expectation. If the agent proves unreliable, the test can return to explicit screen actions without changing the meaning of the assertion. Conversely, a stable agent step can be reused by the replay cache once a later check confirms it. The public API is hybrid by design, so adoption need not be a single decision to make every test adaptive.

Proof, judgment, and waiting have different jobs

The most important line in the hybrid test comes immediately after the agent step:

await agent.act('get the app to greet {name}', { params: { name: 'Grace' } });
await expect(screen.getByRole('status')).toHaveText('Hello, Grace!');

The first line is adaptive. It does not prescribe whether the model taps a field first, types into a visible textbox, or chooses among equivalent controls. The second line is exact. It states which semantic node must exist and what text it must contain. If the agent reached the wrong screen, failed to submit, or produced a different greeting, the expectation fails even if the model returned a confident explanation.

This pairing also creates a useful boundary for replay. A passing action can be recorded, but the framework’s cache design treats a later check as the evidence that the action achieved its intended effect. The test does not need to understand the internal trace format to benefit from that rule. It only needs to place a meaningful verification after the action rather than assuming that a completed model call is proof. The later chapter on replay will examine how the trace is relocated and rechecked; the important point here is that the verifier is part of the test’s design, not an optional comment about what the agent probably did.

Deterministic actions and deterministic expectations are not the same thing. A test may use screen.getByLabel('Name').fill('Ada') and then tap a Greet button when the exact interaction itself is the subject of the test. It may use expect(screen.getByRole('status')).toHaveText('Hello, Ada!') when the outcome must settle to a known string. The screen layer expresses an exact action or query; the expectation layer expresses a condition that can be polled until it holds. Both are independent of model prose.

The distinction matters when a result is delayed. A one-time read of a status element can catch the UI between a submit and the response. The framework’s locator matchers retry for the assertion window, while expect.poll can re-read a value until a value matcher passes. This is different from inserting a sleep. A sleep waits without checking whether the application has reached the state; a matcher repeatedly asks the relevant question. If it still fails, the report can distinguish a wrong expectation, a missing locator, or a state that never arrived.

Model judgments have a narrower but important role. agent.assert('the app greets Grace by name') asks the model to judge a visible fact. It is appropriate when the question involves meaning that is awkward to encode as one exact string—for example, whether an explanatory message makes clear why a payment was declined, or whether a confirmation screen communicates that an account is now active. The judgment sees the current observation, not an unlimited history of the test, and it can be inconclusive when the screen does not show enough evidence. That is a feature: a model should not invent an answer for a page it was never shown.

A judgment should not replace an exact check when the required fact is already precise. If the expected plan is “Pro,” assert that the relevant status contains “Pro.” If a list must have one item, assert the count. If the browser must be on a known route, use the browser URL matcher. Asking a model whether the page “looks like Pro” adds an interpretation step without adding evidence. It can still be useful as a second, broader question, but it should not be the only proof of a known value.

agent.waitFor occupies another position. It repeatedly observes a condition until it becomes true or the time limit expires, and it judges only when the screen has changed according to the runner’s observation policy. That makes it suitable for a condition whose arrival is uncertain but whose meaning is still semantic: an export finishes and a download link appears, or a background status changes to “Ready.” It is not a general polling loop for arbitrary application state. If the state is exposed through an exact locator or API response, a deterministic matcher is usually clearer. If the state is visible but its meaning requires interpretation, a bounded judgment may be justified.

agent.extract turns a visible screen into structured data under a schema. The schema checks the model’s result after extraction; it does not give the model permission to read information that the screen does not show. Asking for every todo title and the remaining count can be useful when the test needs to compare several visible values. But the extracted result should still be checked by the test, and missing information should be represented as uncertainty rather than silently becoming an empty string or zero.

These choices also make failure reports more useful. A failed expect can point to the exact value or role that did not settle. A missing or ambiguous locator says that the semantic surface did not support the intended action. An inconclusive judgment says that the visible evidence was insufficient, not that the application necessarily contradicted the claim. A model-provider failure points somewhere else again. The test author can then decide whether to change the app’s accessible naming, narrow the deterministic query, split the agent goal, extend a justified timeout, or remove model interpretation from the step. Without this separation, every red run would look like “the AI failed,” which is too vague to guide maintenance.

There is a further reason to keep exact checks close to agent actions: the check records the test author’s intended boundary while the screen is still in the state the agent produced. If later steps mutate the page, a delayed or distant judgment has to reconstruct which state it was meant to describe. An immediate matcher instead makes the causal sequence visible: the goal ran, the relevant state was checked, and only then did the test move on. That sequence is valuable to a human reader, to the runner’s report, and to the replay logic that may reuse the action later.

The same test can therefore contain several kinds of truth without becoming incoherent:

sequenceDiagram participant T as Test body participant R as e2e runner participant A as Agent participant S as App screen participant X as Expect T->>R: agent.act goal and params R->>S: Capture semantic observation R->>A: Send goal and redacted screen A->>R: Choose validated grammar action R->>S: Perform action S-->>R: Return settled state R-->>T: Return act result T->>X: Run exact expect matcher X->>S: Poll current semantic state S-->>X: Match or timeout X-->>T: Deterministic verdict opt A semantic question is needed T->>A: agent.assert or extract A->>S: Read fresh visible evidence S-->>A: Current observation A-->>T: Judgment or schema-checked data end

The agent’s truth is about how to pursue a goal. The screen and expectation layer’s truth is about whether a known condition holds. A wait or extraction step answers a bounded semantic question against fresh evidence. The runner’s truth is whether the entire step stayed within policy, capability, and time limits. Keeping these layers distinct makes the test easier to maintain and the failure easier to interpret.

A useful design rule follows: use the least interpretive mechanism that can answer the question. Let the agent navigate a variable path. Use screen for exact interactions and values. Use expect for known states that may take time to settle. Use agent.assert, waitFor, or extract when the visible question is semantic or multi-part and cannot be expressed more clearly as a deterministic matcher. This is not a rejection of AI. It is how e2e gives AI a precise place in a test whose final meaning remains inspectable.

The next question is what the agent and the locator layer actually see. Their shared vocabulary comes from a semantic representation of the app: roles, names, text, states, geometry, and platform-specific accessibility information. That representation determines both how a path can be chosen and how an exact proof can find the state it is meant to verify.

References