Prefetch, Mirrors, and the Cost of Being Wrong

JustVugg/colibri does not stop at reacting to a cache miss; it also tries to move likely future expert reads earlier and to use more than one storage path when the machine has them. This chapter examines those predictive choices as a diagnosis problem: what is being predicted, what resource the prediction consumes, and what happens when it is wrong. Readers will be able to distinguish a harmless prefetch hint from a real future load, understand why mirror routing and within-read striping are different mechanisms, and judge each one by end-to-end waiting time rather than by recall or raw drive bandwidth alone.

Prediction starts before the next miss

A normal cache learns after an expert has been requested. That is useful, but it means the first access still pays the full storage cost. Colibri’s router-lookahead path asks whether the current hidden state contains enough information about the next layer’s route to begin that read earlier. For the GLM-5.2 path, the project reports that applying the next layer’s router to the current post-attention state recalled 71.6% of the true top-eight experts. Reusing the previous token’s expert set recalled only 41.3%.

The comparison matters because it identifies a mechanism, not just a correlation. The next layer is about to consume a state that already exists, so its router can be evaluated before the main execution reaches that layer. The prediction is imperfect, but it is better aligned with the next request than simply assuming that the previous token will repeat itself. If the predicted experts are cold, Colibri can issue a readahead operation while the current layer computes.

The simplest form, PILOT, is a hint to the operating system or storage path. It asks for the future ranges to become more readily available, but does not necessarily place a completed expert into the future layer’s LRU. A real load, PILOT_REAL, goes further: it reserves a future cache slot, reads the expert into that slot, and publishes the result only when the load succeeds. The distinction is operational. A hint can be lost without changing cache ownership. A real load consumes memory and I/O before the current route has asked for the expert, so it needs reservation, eviction, and failure rules.

Colibri’s real pilot path keeps the prediction outside the current layer’s semantic critical path. A worker checks whether the future expert is already pinned, resident, or reserved. If not, it marks a slot as reserved before releasing the placement lock. The slow read happens outside that lock so other work can continue. When the read completes, the worker publishes the expert and refreshes its recency. If the read fails or the main computation has reached that layer first, the reservation is dropped and the normal demand path remains responsible for serving the expert.

That order protects two different resources. The reservation protects memory ownership: two prediction workers should not load the same expert into the same slot or evict one another’s in-flight work. The layer barrier protects correctness: the main path must not inspect a future slot while a worker is still writing it. A failed prediction is therefore allowed to lose time and bandwidth, but it is not allowed to expose a half-loaded slab as a valid expert.