Why Natural-Language UI Tests Need a Harness

This series examines tester-army/e2e, an open framework for end-to-end testing on web and mobile apps. The focus here is not simply the use of a language model to click through an interface; it is the larger system that turns a natural-language goal into a controlled action, an observable state, and a test verdict. That distinction matters because the reader needs to judge where agent flexibility helps, where exact proof is still necessary, and what the framework can honestly promise.

The central idea is bounded semantic control. e2e lets an agent work with the meanings exposed by an app—roles, labels, visible text, states, and navigation—while a runner controls what the agent may do, how long it may act, and what counts as evidence. Deterministic assertions, replay traces, and privacy rules sit beside the model rather than disappearing behind it. The result is an attempt to make variable UI paths useful without making the test itself variable in every important sense.

The subject is bigger than the model

The repository describes e2e as an end-to-end framework for web and mobile applications: a test can state a goal in natural language, let an agent drive the app toward it, and then check the result with locators and assertions. That description names three participants that are easy to collapse into one. The app supplies the surface and its state. The agent proposes and carries out a path. The runner decides whether the path is authorized, observable, bounded, and verified.

The distinction becomes clearer in the project’s small examples. In a greeting test, an agent is asked to get the app to greet Grace. The agent must work out which field and button to use. The test then checks that a status element says exactly “Hello, Grace!” A second model judgment may ask whether the app greets Grace by name, but the exact status check does not depend on how the model describes what it did. In a conventional version of the same example, the test fills the field and taps the button directly. No model is needed at all.

That relationship defines the real subject: tester-army/e2e is a runner and engine system that can host both adaptive and deterministic interaction. Its core package provides the SDK, runner, and CLI. A Playwright-based web engine drives browsers, while an agent-device-based engine drives iOS simulators, Android emulators, and connected phones. Other packages add hosted resources, reporting, or decision-model execution. The model is important, but it is one participant in a system with its own contracts.

This framing avoids a common mistake. If e2e were only a prompt wrapped around browser automation, its main engineering question would be whether the model can find a button. In the actual repository, the harder questions are broader: what the model is allowed to see, how an action is tied to the current screen, how a stale control is handled, how a result is checked, how a successful action can be reused, and what information must never enter a screenshot or trace. Those are runner questions, not merely model questions.

That boundary also clarifies ownership. The app owns the behavior being tested: its labels, navigation, validation, loading, and state transitions. The engine owns the translation between that app and the runner—browser or device operations, semantic observations, platform capabilities, and artifacts. The core runner owns the rules that should not vary with the platform: budgets, locator strictness, secret policy, step accounting, and the meaning of a verdict. When these responsibilities are mixed, a failure becomes hard to interpret. A missing accessible name can look like a weak model; a delayed save can look like a bad assertion; an unsupported device action can look like an agent mistake. Separating the boundaries does not remove those failures, but it gives each one a place to be diagnosed.

The distinction matters for maintenance as well. An exact script can break when a button moves, even though the user-facing task remains the same. A natural-language step can survive that layout change, but it may still fail if the new screen no longer exposes a usable role or label, or if the expected result is not stated precisely. e2e’s promise is therefore not that one layer replaces the other. It is that a team can place flexibility at the path-selection boundary and keep the application’s important outcomes under a more explicit proof boundary.

Why goals need proof

Natural-language goals are useful because a UI flow often has more than one reasonable path. A user might open a menu, choose a tab, search for a record, or follow a changed layout. An exact script has to name each interaction in advance. An agent can instead receive the outcome that matters—upgrade the workspace, find the relevant setting, or submit the form—and choose among the controls currently visible.

That flexibility has a cost. A sentence such as “the agent upgraded the workspace” is not the same thing as evidence that the workspace is on the Pro plan. The model may have misunderstood a label, stopped after an intermediate screen, or described an intended action rather than a committed state change. Even a successful click can leave a save request pending or fail to change the application at all.

The framework’s answer is not to reject natural language. It is to separate semantic intent from exact proof. The agent is well suited to finding a path through a changing interface. screen and expect are better suited to checking a known fact: a status contains a plan name, a URL has changed, a list has a particular count, or an alert contains a validation message. Model judgments such as agent.assert remain useful when the question is genuinely interpretive, but they are not required for every test and should not silently replace an exact oracle.

The same principle applies to waiting. A current text read answers what the screen says at one moment. An expectation can poll until the UI reaches a condition. A test that needs structured data can ask an agent to extract it under a schema and then assert on the returned value. Each operation has a different evidence burden. Treating them as interchangeable would make it difficult to tell whether a failure came from the application, the locator, the model, or the timing of the observation.

The runner therefore acts as a boundary around agent freedom. An agent step has a goal, a finite action budget, a finite model-call budget, and a deadline. Its actions go through a known grammar rather than arbitrary code returned by the model. The runner can reject an unsupported capability, refuse a secret in the wrong field, or report that a target is ambiguous. A model may choose the path, but it does not redefine the test’s authority.

This boundedness changes what a failure means. If the model runs out of steps, the result is not silently converted into a pass because the final explanation sounds confident. If a locator matches two controls, the runner does not pick the first one and hide the ambiguity. If a device or browser cannot perform an action, the capability boundary is reported instead of being simulated with an unapproved shortcut. These choices can make a test fail more often in the short term, but they keep the failure attached to a fact that a maintainer can investigate. A flexible path is useful only if the system can also say when it did not complete the requested path or cannot prove that it did.

That is the difference between an exploratory assistant and a testing framework. Exploration values discovering what might work. Testing must preserve the distinction between an attempted action, an observed transition, and an established outcome. The repository’s design puts those distinctions in the runner, where they can be applied to both web and mobile engines and to tests that never call a model at all.

flowchart LR App["App under test"] --> Obs["Semantic observation"] Goal["Natural-language goal"] --> Agent["Bounded agent"] Obs --> Agent Agent --> Action["Validated action"] Action --> State["Settled app state"] State --> Proof["Deterministic proof"] State --> Judge["Model judgment"] Proof --> Verdict["Runner verdict"] Judge --> Verdict

This is why the word “harness” matters. It names the control surface between an intelligent action proposer and an application whose state must be tested. The harness supplies the semantic observation, validates the action, waits for the effect, records what happened, and applies the verdict rules. Without it, a natural-language test would be closer to an exploratory demo than to a repeatable end-to-end check.

What the system can and cannot establish

The repository supports a strong architectural claim: e2e is designed to combine adaptive UI driving with deterministic, evidence-oriented testing. It also documents concrete limits. The project is still on the way to 1.0, so APIs and configuration may change between minor releases. Agent steps require a configured AI SDK model and provider authentication unless a replay or custom execution path handles the step. Browser and device targets have different capabilities. Semantic interaction depends on the application exposing meaningful names, roles, states, or other identifiers.

The material does not establish that e2e finds more bugs than conventional frameworks, costs less, eliminates test maintenance, or succeeds on every UI. The repository contains examples and benchmarks, but those demonstrate intended usage rather than a broad comparative evaluation. Nor does a model’s ability to complete a flow prove that the flow is safe to replay or that the result is correct. Those conclusions require current observations and a postcondition.

There are also limits that belong to the application team, not only to the framework. A semantic test is only as clear as the interface it observes. If a form has no useful accessible name, if several controls expose the same description, or if the meaningful change appears only in pixels that cannot be safely captured, the runner has less evidence to work with. Likewise, a mobile target brings a device pool, app build, permissions, and platform-specific actions into the test’s operating conditions. These are not incidental setup details: they determine which claims the test can make and which failures are environmental rather than product behavior.

A sensible evaluation therefore asks more than whether an agent can complete a happy path once. It asks whether the important outcome is observable, whether a failure is classified clearly, whether the evidence can be retained safely, whether a second run can verify the same effect, and whether the team can maintain the app’s semantic surface. Those questions turn the framework’s architecture into an engineering decision rather than a demonstration of model fluency.

The practical judgment this series supports is therefore conditional. Use agent flexibility where the path is variable and the outcome can be stated clearly. Keep exact interactions and assertions where a value, count, permission, or irreversible action must be controlled precisely. Treat the app’s semantic surface as part of the test design. Then ask whether the runner’s budgets, replay safeguards, privacy policies, and target capabilities are sufficient for the risk of the flow.

The next step is to look closely at that division of labor. A single e2e test can let an agent choose how to reach a state without letting the model’s explanation decide whether the state exists. Understanding that two-part truth is the foundation for everything that follows.

References