When a Trace Becomes Evidence
This chapter examines Experiential's first trust boundary: the transition from a declared trace export to evidence that later evaluation and serving can safely reference. The key question is not whether the project can parse a file. It is whether the project can preserve what the file claimed, record what it could not establish, and bind the resulting workload to a reproducible project graph. Understanding that lifecycle lets the reader distinguish a raw export, a normalized trace, a representative task set, and a completed build—and see why each transition is deliberately validated.
A source is a claim about provenance
Experiential does not begin by guessing which vendor produced a file. The caller declares a source kind, and an explicit loader handles that kind. The source table includes families such as OpenTelemetry exports, observability-vendor records, and plain chat JSON, but the important design choice is not the size of the table. It is that the table is authoritative. A name that is not declared is rejected rather than routed through a best-effort detector.
That may seem conservative for a system meant to make onboarding easier. In practice, format detection can answer only a syntactic question: “Can these bytes be interpreted as JSON with these fields?” It cannot reliably answer the semantic question: “What did this field mean in the system that produced it?” A vendor's model field, a tool result, a parent span, or a timestamp can be plausible in several formats while carrying different assumptions. By making the caller name the source, Experiential places the interpretation decision at the boundary where it can be recorded and reviewed.
The declaration also keeps a worker-local path from becoming the only identity of the evidence. A durable source label can stand for the imported corpus after the worker that read it has gone away, while the source-byte digest preserves which bytes were actually interpreted. Moving an export or rerunning a build does not, by itself, authorize a later stage to treat a different file as the same source. The project can compare the declared identity and digest rather than trusting a filename or an artifact directory to carry meaning.
The loader then normalizes different source shapes into one trace representation. Each normalized trace keeps an immutable source identity and the exact source-byte digest. Original source trace and span identifiers remain available as evidence; opaque vendor identifiers can receive deterministic W3C-shaped identities so internal relationships remain stable without erasing the source identifiers that a person may need to inspect. Parent relationships are retained when they are unambiguous rather than reconstructed by guesswork. The same principle applies to model identity: if an export declares both provider and model, the identity can be resolved; if it names only a model, that name remains model evidence instead of being paired with an inferred provider.
This is the first important state transition. The project is not claiming that normalization has discovered the one true meaning of every source record. It is claiming something narrower and more useful: this record was interpreted under a named convention, with the source identity and unresolved ambiguity kept visible. A later optimizer can make a decision about that evidence. It does not have to rediscover which assumptions were made while loading it.
Normalization preserves failures instead of hiding them
A useful evidence pipeline must distinguish “this record was invalid” from “this record was absent.” Experiential keeps explicit normalization issues beside the canonical trace records. A malformed JSONL line is not silently skipped. Records that cannot be normalized are excluded from the valid trace set, but the exclusion is retained with its source record and reason. The count of invalid records is checked against the issue list, so a summary cannot quietly drift away from the detailed explanation.
That behavior changes the meaning of a successful build. A project with valid traces and recorded exclusions is not the same as a project whose loader happened to return the same number of traces after discarding failures without a record. The former gives a later reviewer a denominator and a place to ask whether the exclusions could bias the workload. The latter makes the missing data look like it never existed. Explicit issues also make reruns interpretable: if the same source bytes and convention produce the same issue set, a replay can be checked; if they do not, the evidence identity should not be treated as unchanged.
The provider-free command adds another explicit boundary: public build preparation accepts 100 through 1,000 valid normalized traces after validation and source deduplication. This is a constraint on that build path, not a universal statement that a retrieval index with fewer or more traces is intrinsically invalid. The broader retrieval contract treats the trace count separately from whether the corpus contains valid observed transitions. Keeping these statements apart prevents a convenient onboarding range from being mistaken for a law of evidence quality.
The same discipline appears in the treatment of identifiers and tools. Tool results are paired with tool calls using an explicit call identifier when one exists; a tool name and source order are fallback evidence only when the export itself declares no identifier. The normalizer preserves declared relationships but does not manufacture a more confident relationship simply because one would be convenient for a downstream workflow. For an agent system, that matters: a wrongly paired tool result can make a model appear to have caused an outcome it did not cause, which would corrupt both evaluation and any later routing decision.
There is a similar restraint around missing model metadata. A model name can remain useful as evidence even when the provider is absent, but silently filling in the provider would turn an observation into an inference. That distinction later affects catalog resolution, pricing, and candidate eligibility. A trace may be valid enough to inspect while still being insufficient for a role that requires a verified provider identity or price. In other words, normalization does not force every record into a binary “good” or “bad” category; it preserves the evidence needed for later stages to make narrower eligibility decisions.
The lifecycle can be pictured as a series of state changes with different owners and failure meanings:
An issue-recorded state is not a successful trace, but it is still part of the evidence story. A normalized trace is usable input, but it is not yet a task or a model evaluation. The arrows therefore do more than describe chronology: they mark when information is filtered, when identity is added, and when a new artifact becomes dependent on an earlier one. The state diagram also clarifies ownership. The source loader owns interpretation of the declared format; dataset persistence owns canonical bytes and issue records; task mining owns representative selection; project selection owns the pointer graph. No later owner should quietly rewrite an earlier state.
From traces to a bound project graph
Once valid traces are persisted, Experiential can mine representative tasks. The task set is not an independent sample with a fresh provenance story; it is derived from the trace dataset and records that dependency. This distinction is important because later evaluation is intended to resemble the work represented by the imported traffic. It does not make the task set a complete model of that work. It makes its relationship to the source inspectable.
Representative selection is therefore a controlled reduction, not a replacement of the corpus. The project may need a manageable set of tasks for evaluation, but the task set should still point back to the trace evidence from which it was derived. If a selected task changes, the relationship to the source should be visible in the artifact graph. If the source dataset changes, a prior task set should not silently continue to look current merely because its human-readable task names are unchanged.
The project then treats the trace dataset and task set as immutable artifacts with manifests. The manifest binds file digests, artifact type, inputs, and the code revision that produced the artifact. When the task set is loaded again, the project checks that it names the exact trace manifest. A rebuild can therefore either reproduce the same content-addressed identity or produce a different artifact that must be treated as different evidence. A process restart does not turn “the current directory” into an implicit source of truth.
This is useful in both failure and success cases. If a run stops after the trace dataset is written but before later artifacts are selected, a later invocation can verify and reuse the completed lower-level artifact instead of reinterpreting the raw source. If an artifact exists under an expected identity but its content or manifest differs, replay must fail rather than adopt the mismatch. Reproducibility here means more than obtaining similar outputs; it means refusing to call a changed input graph the same graph.
The completed build extends that chain. The selected project graph can bind the source dataset and task set to the serving retrieval index, the fit retrieval index, and the grounded world-model artifact. Those later artifacts are not interchangeable: serving retrieval and fit retrieval have different roles, and a grounded world model must name the retrieval material it actually uses. The value of the graph is that the question “What was used to make this policy?” has a checkable answer rather than a collection of remembered settings.
This is also where the provider-free boundary becomes meaningful. The local preparation step can normalize traces, persist issues, mine tasks, and stop before model proposals, judge calls, or paid provider work. Later stages may need model or embedding clients, but they should not be smuggled into the evidence-preparation step. Separating the stages makes cost and interpretation visible: the project can establish what traffic it has before asking a model to evaluate candidates or generate counterfactual behavior. A dry run can show the preflight without making a provider call, and exact replay can reuse completed indexes without pretending that replay was a new experiment.
The result is not a guarantee that the dataset is representative. A complete manifest cannot reveal outcomes that were never captured, and a valid trace can still reflect a narrow operational slice. What the lifecycle provides is a disciplined starting point: the evidence has a declared origin, explicit exclusions, stable identities, and dependent artifacts that cannot silently change roles. That is enough to support the next question without answering it in advance. How can these real traces ground a world model that predicts what might happen after an agent action, while keeping those predictions outside the source of truth?
The distinction between these artifacts is especially important when a project is resumed. A trace dataset is the canonical record of normalized source evidence; a task set is a derived representation of the workload; a retrieval index is a selected projection of observed transitions; and a grounded world model is an executable layer that depends on that projection. Treating them all as interchangeable “training data” would erase the reason each manifest exists. A pointer to a task set must agree with the task-set manifest, the task set must name the exact trace dataset, and later retrieval and world-model artifacts must name the inputs they actually use. The graph is therefore a chain of specific claims, not a label attached to a folder.
This also gives operators a way to ask a more useful question than “Did the build finish?” They can ask which stage finished, which evidence was excluded, whether a later artifact was replayed or newly materialized, and whether its upstream digest still matches. A finished normalization step may be enough to inspect a corpus but not enough to run a grounded simulation. A finished task-mining step may define an evaluation workload but not prove that a candidate model is better. The project’s lifecycle keeps these stopping points visible, so a partial result does not acquire the authority of a completed graph merely because it has a plausible name.
There is a cost to this explicitness. Operators must carry source declarations, artifact identities, and catalog decisions instead of relying on inference. But that cost is paid at the boundary where ambiguity is cheapest to resolve. Once a router has been fit or a production decision has been made, recovering whether two similar-looking inputs were actually the same evidence is much harder. Experiential's choice is to make the early transition precise enough that later optimization can remain focused on model behavior rather than quietly repairing its own data history.