How Experiential Learns a Router Without Moving the Goalposts

This chapter examines the decision order that turns Experiential's mixed evidence into a router policy. The focus is not the mechanics of nearest-neighbor selection; it is the more consequential question of when evidence is allowed to influence the policy and when it is reserved for testing it. The reader should be able to distinguish production and simulated cells, understand how judgments and finite budgets qualify fit evidence, and see why the policy must be locked before held-out results are opened.

A plan can contain different kinds of cells without collapsing them

A routing evaluation begins with a combined plan, but “combined” does not mean undifferentiated. The plan can contain observed production cells and simulated cells, each tied to an evaluation protocol and a candidate model snapshot. An observed cell carries behavior that occurred in the real system. A simulated cell carries a rollout produced by the grounded world-model path. Bringing them into one plan lets the project compare candidates across a broader workload, while preserving the evidence source that produced each result.

That distinction matters because the two cell types answer related but different questions. An observed cell asks how a candidate or incumbent behaved on a real task under the recorded conditions. A simulated cell asks how an agent episode unfolds when the environment continuation is supplied by the bounded world model. The latter can expand coverage or make a controlled counterfactual possible, but it does not acquire the authority of the former. A router can use both according to a declared protocol; a reader should still be able to tell which kind of result moved the policy.

The plan also binds the surrounding assumptions that make a comparison interpretable. Candidate identities, pricing, an embedding set, a task set, a rubric or calibration, and the world-model settings are not loose context added after the result. They are inputs to the evaluation graph. If a candidate's model snapshot changes, or a pricing artifact no longer covers the plan's candidates, the workflow should not quietly compare the new state to old evidence. The point of binding these objects is not to freeze every future experiment. It is to keep one experiment from changing its terms while it is being read.

Judgment is another distinct layer. A judge applies a rubric or calibration to a candidate result and turns a response into evaluation evidence. That signal may be necessary for a router optimizer, but it is not the same thing as the underlying response and not the same thing as a real user or environment observation. A provisional judgment status and a human-calibrated status are also different forms of provenance. Making the status visible prevents a report from presenting all scores as equally direct measurements.

The workflow therefore has at least four relevant kinds of content in its fit plan: observed behavior, grounded simulation, judgment, and the immutable configuration that explains how the first three were interpreted. The project can use these together without claiming that they are interchangeable. A candidate that looks good under a judge may still be unsupported on a tool capability.