Checkpoint: A Working Mental Model of Placement

For JustVugg/colibri, the placement model is useful only if it explains both execution and results: how a sparse route becomes physical work, why one machine waits on storage while another waits on compute, and when an optimization is still serving the same model. This checkpoint connects those relationships into a practical test. A reader should now be able to look at a Colibri configuration and ask three separate questions: where are the selected weights, what resource is binding, and has the proposed speedup crossed a boundary that requires a new quality claim?

One model from route to result

A routed token begins as a logical request. The model’s router selects expert IDs and gate weights for a layer. Colibri then forms the union of distinct experts needed by the batch, so repeated selections do not cause repeated physical loads. The cache resolves each unique ID through the pinned store or the per-layer LRU. A miss becomes a bounded read from the indexed checkpoint, usually a slab containing the expert’s projection weights and associated scales. A format-specific kernel computes the expert outputs for the rows that requested it, and the engine scatters those outputs back with their gate weights.

Seen this way, the hierarchy is not a pile of independent features. Each boundary answers a different question. Routing owns which logical computation is requested. The union owns how many physical copies must be prepared for this batch. The cache owns where those copies currently live. The storage index owns which bytes represent them. The kernel owns how those bytes become numerical output. The residual update owns how the result continues the model’s state. Telemetry records the route and residency so later placement decisions can use evidence from the workload.

This division also explains what a valid placement substitution looks like. If the selected expert is in RAM rather than on disk, the logical route is unchanged and the difference is primarily waiting time. If the same expert is served from a resident accelerator copy, the execution site changes but the request still names the same expert. If the batch union loads one copy for three rows and applies three different gate weights during accumulation, the physical work is shared while the logical contributions remain separate.

The chain has failure boundaries as well. A cache miss is recoverable because the selected expert remains available on the backing store. A prefetch failure can be discarded because it was not required by the current route. A malformed range, an incompatible tensor shape, or a missing scale is different: allowing the computation to proceed would mean inventing the meaning of the weight. Likewise, dropping a selected expert or replacing it with a convenient resident expert may save I/O but changes the route. The fact that the process returns text cannot settle that distinction.