The Evidence Ladder Behind a Routing Decision

This checkpoint connects the evidence chain already established for Experiential: declared traffic becomes immutable traces, real transitions ground retrieval, and bounded rollouts extend what can be compared without becoming new observations. The practical question now is what each kind of artifact is allowed to prove before a router is fitted. The reader should leave with a working test for provenance continuity and with enough separation between observation, simulation, judgment, and release measurement to read the next optimization stage without overclaiming.

Four kinds of evidence, one traceable chain

A later routing decision is traceable back to real traffic only when the chain of identity remains intact at every transition. The source must be declared rather than inferred. Normalization must preserve source identifiers, digests, model evidence, and explicit exclusions. The trace dataset must be immutable and content-addressed. A representative task set must name the exact dataset from which it was derived. Retrieval must point to verified real traces and preserve transition and lineage identity. If any link is replaced by an unrecorded “latest” file or an inferred provider, the final policy may still run, but its relationship to production traffic becomes an assertion rather than a checkable fact.

This is a stronger requirement than merely saving a prompt and response. A saved conversation may tell an engineer what happened in one instance, but it does not necessarily say which source convention interpreted the record, whether a malformed line was excluded, how a tool result was paired, or which task selection produced the workload used for evaluation. Experiential's artifact graph carries those distinctions forward. It allows a later reader to ask not only “What did the model answer?” but also “Which evidence qualified that answer for this decision, and what was deliberately left out?”

The chain can be read as a sequence of increasingly specific claims. A declared source says where an interpretation began. A normalized trace says which records survived that interpretation and which uncertainties were retained. A real-transition index says which observed action-to-subsequent-observation pairs are eligible for retrieval. A grounded rollout says what a simulator predicted after consulting some of those transitions. A judged evaluation says how a candidate response was scored under a rubric. A locked policy says which fit evidence was authorized to determine routing. A held-out report says how that already-locked policy performed on evidence opened afterward.

These claims depend on one another, but they are not interchangeable. A task set is not an observation. A retrieval neighbor is not a guarantee of representativeness. A judge's label is not a provider trace. A policy report is not a live service-level objective. The evidence ladder is useful precisely because it preserves both connection and difference: the artifacts form one lineage, while each artifact answers a different question.

The first visual summarizes that progression:

flowchart LR A["Declared real source"] --> B["Immutable normalized traces"] B --> C["Real-transition RAG"] C --> D["Grounded rollout"] D --> E["Judged fit evidence"] E --> F["Locked router policy"] F --> G["Held-out report"] D -.-> X["Forbidden: generated rollout becomes retrieval truth"] X -.-> C

The forbidden edge is not a claim that simulation is useless. It marks the point where a helpful counterfactual would become self-reinforcing evidence. A rollout can be preserved for comparison, but it cannot acquire the authority of an observed transition merely because it was stored in a durable artifact.