When Evidence Can Become a Liability

This chapter examines tester-army/e2e’s security boundary around credentials, screenshots, semantic observations, traces, and sessions. The practical question is how a secret-bearing test can still prove that a sign-in or payment flow succeeded without turning the model, replay cache, or artifact directory into another place where the secret can appear. The reader should leave knowing which evidence is available after sensitive input and which evidence e2e intentionally withholds.

Replay makes evidence reusable, but reuse also makes leakage durable. A password typed once can appear in a model prompt, a screenshot, a trace, a debug transcript, a browser page, or a hosted recording if the system treats it like ordinary test data. e2e’s design responds by making sensitive values opaque and by separating semantic proof from visual evidence.

The secret is a handle until the sink is authorized

The test configuration declares credentials and generic secrets, but test code receives handles rather than plaintext values. A credential exposes a username and an opaque password handle. A generic secret can represent an API key, token, or other sensitive value. The model can be told that a secret named admin.password exists, but it does not receive the value that the runner will eventually fill.

That distinction is more than a logging convenience. A value that enters the model’s context is difficult to guarantee against later reproduction. A handle remains a reference that the trusted runner can resolve at the moment it is needed. The runner can then pass the plaintext directly to the engine’s sensitive fill path and keep it out of the agent’s returned text, action summary, cache payload, and report.

The fill is not authorized merely because the test declared a secret. The runner checks the sink first. The node must be an enabled editable input. A password-purpose secret must go into a field the semantic observation identifies as a password field. A generic secret has a wider editable-input policy, but it still cannot be sent to a disabled control or a non-editable element. Only after those checks pass does the runner resolve the secret value.

The distinction between password and generic secret protects against an easy category error. A credential password belongs in a password field whose purpose is to conceal it. An API key might be entered into a setup field or passed to a controlled tool, but treating every sensitive value as an interchangeable password would either block valid use or widen the password’s allowed sinks too far. The secret’s purpose is part of the policy, not a comment attached after the fill.

The agent step also has to declare which secret handles it may use. A model cannot invent a name and ask the runner to resolve an undeclared secret. This keeps the model’s authority narrower than the project’s complete secret registry. A test can grant one step access to one credential while leaving other credentials and environment values outside that step’s scope.

Once a value is resolved, the runner’s redactor covers it in instructions, parameters, observations, ledgers, action descriptions, traces, reports, and debug transcripts. A trace records a stable secret name rather than the plaintext and re-runs authorization when it reaches that sensitive action. This is what makes a secret-bearing step compatible with replay without writing the secret into the replay entry.

The sequence also explains what the policy does not solve. The app may echo the secret into its own page or server log.