The Cache That Learns the Workload

JustVugg/colibri learns from a session in one narrow but consequential way: it records which experts the workload actually routes to, then uses that evidence to decide which weight copies deserve scarce fast memory. This chapter focuses on the lifecycle of that evidence and of the expert copies it influences. Readers will be able to tell what .coli_usage represents, why Colibri combines a pinned hot store with a per-layer LRU, and how stale or mismatched history can make placement worse without being allowed to change the model’s logical identity.

From a route to a resident expert

When a token selects an expert, Colibri has two jobs. The immediate job is to serve the current request, using a resident copy if one exists or loading the expert from the backing store if it does not. The longer-term job is to remember that selection so later requests can make a better placement decision. The persistent history belongs to the second job. It is not a new set of model weights, and it is not a learned replacement for the router.

The history is organized around the model’s layer-and-expert coordinates. An expert that is selected often in one layer accumulates more usage there than an expert selected rarely. That distinction matters because the same numerical expert ID in a different layer is a different weight block. A placement policy needs to know not only “expert 37 is popular,” but “expert 37 in this layer is popular.” Colibri keeps that structure so a hot copy is placed where the model will actually look for it.

The first time a workload reaches a cold expert, the route creates a miss and the loader fills a reusable LRU slot. The current computation can finish, and the usage counter records the selection. If later tokens request that expert while it is still in the LRU, they get a hit and refresh its recency. If repeated requests keep selecting it, the accumulated heat can make it a candidate for the pinned hot store. The path from cold bytes to a resident copy is therefore also a path from observation to policy.

A pin and an LRU entry solve different locality problems. A pinned expert is a long-term bet: the engine reserves space for it because the measured workload has repeatedly made it valuable. An LRU entry is a short-term bet: it stays because recent access suggests that another request may use it soon, but it can be displaced when the cache needs room. Colibri’s automatic history pinning and adaptive LRU share the same RAM budget, so pinning more experts can leave fewer slots for short-lived locality. A good policy has to reserve enough reversible capacity for the workload to change.

Consider two workloads on the same model. A sequence of prompts about one domain may repeatedly route to a concentrated group of experts. Pins can keep those experts close and let the LRU handle occasional variation. A multilingual or mixed coding-and-chat workload may move among different groups. A pin set learned from the first workload can occupy memory while the second workload repeatedly misses on the experts it actually needs. The history has not become a semantic error; it has become a poor prediction of locality.

That is why Colibri describes history as a workload prior. It predicts where bytes should live. It does not authorize the engine to alter the route to match the prediction.