Why MoE Sparsity Changes the Memory Problem
In JustVugg/colibri, the important question is not simply how many parameters a model contains. It is how many of those parameters a token needs at one moment, which of them are needed repeatedly, and which can wait on storage until a router selects them. This explanation focuses on the reference GLM-5.2 geometry so readers can distinguish total parameters, active parameters, dense state, and routed experts—and use those distinctions to predict where Colibri will spend memory and time.
A router does not load the whole model
A mixture-of-experts model, or MoE, is built from many alternative feed-forward networks called experts. A router examines the representation of a token and assigns it to a small selection of those experts. Their outputs are combined, usually with weights that reflect the router’s scores, before the token continues through the layer. The other experts remain part of the model, but they do not contribute to that particular token at that particular layer.
That last distinction is easy to state and easy to lose. “Inactive” does not mean “unimportant,” and it does not mean “removed.” An expert that is not selected for one token may be essential for another token, another language, another subject, or another position later in the sequence. The model’s full capacity is distributed across possible routes; the work for one token is sparse within that capacity.
Colibri’s reference case makes the scale difference concrete. Its documentation describes GLM-5.2 as a 744-billion-parameter MoE with roughly 40 billion parameters active for a token. The complete model is therefore much larger than the computation associated with any one token, even though the token still passes through many layers and makes a new routing decision as it goes.
The routed part is especially large. Colibri’s description counts 19,456 routed experts: 75 main MoE layers with 256 experts each, plus the model’s additional draft-head row. At int4, an individual expert is described as roughly 19 MB. Keeping every routed expert in fast memory would therefore require a space measured in hundreds of gigabytes. The routed set is described as roughly 370 GB on disk, while the dense portion is much smaller in the reference container.
The arithmetic explains why sparsity changes the memory problem. If every token required every expert, the engine would need the full routed set close to the processor, and disk streaming would be irrelevant or impossibly slow. If each token needs only a few experts per layer, the engine can reuse a working slot after one expert has finished, then fill that slot with another expert selected later. The full set still has to exist somewhere, but it does not have to occupy the fastest tier simultaneously.
This is a temporal form of sparsity. At the beginning of a layer, the router selects a few expert IDs. During the layer, Colibri needs the corresponding matrices and their scales. After their contribution has been combined into the token’s hidden state, those particular working buffers can be reused. The next layer has its own selection, and the next token may select a different pattern. The model is not smaller; the set of bytes that must be immediately available keeps changing.
That changing set is why a parameter count alone is a poor predictor of whether a model can run. Total parameters describe the model’s possible capacity. Active parameters describe a portion of the work for one token. Neither number, by itself, tells us how much dense state must remain resident, how many routed experts fit in a cache, how quickly a cold expert can be fetched, or how often later tokens will reuse the same expert. Colibri is built around the gap between those descriptions.
Dense state stays, routed state moves
The reference GLM-5.2 layout divides the working problem into two broad kinds of state. The dense part includes attention, embeddings, and shared components. It participates repeatedly, so Colibri’s design keeps it resident in RAM; the README describes this portion as about 17 billion parameters and roughly 9.9 GB at int4. The routed experts are the large, selective part. They can be distributed among a fast tier, a warm cache, and a cold disk-backed set.
This is not an arbitrary division between “important” and “unimportant” weights. It follows from access frequency and control flow. Dense operations are on the path for every token, so repeatedly fetching them would make the entire model wait. Routed operations are conditional, and their expert identities are known only after the router runs. They are the natural candidates for staged movement because the engine can defer the cost until the route is known.
The word “resident” also needs care. A resident expert is not a second version of the model. It is a copy of the same expert weight data held where computation can reach it more quickly. A routed expert in RAM and the same routed expert on disk are alternatives in location, not alternatives in meaning. If the router selects expert 137 for a layer, placement should determine whether Colibri finds expert 137 nearby or has to read it—not whether it quietly replaces expert 137 with a different expert that happens to be cached.
The fast tier can be small because it only needs to cover the working set that matters for the current workload. Some experts may be selected repeatedly by related prompts and become good cache candidates. Others may be rare and remain on disk. A changing conversation can move the useful set again. This is why Colibri’s memory hierarchy is not a one-time partition of the model; it is a policy for deciding which physical copies earn scarce capacity while the logical model stays whole.
The diagram shows two different kinds of continuity. The dense foundation stays on the path from token to token. The routed branch changes its physical location according to the router and the cache policy, then rejoins the same hidden-state computation. The important unit is not “all weights in memory,” but “the current layer’s selected weights available before that layer must finish.”
Why expert count matters to cache capacity
An expert count is operational because Colibri’s cache is not just a general-purpose pile of bytes. The runtime must account for expert sizes, layer boundaries, dense memory, attention state, and working buffers. A cache that can hold one expert for each sparse layer behaves differently from a cache that can hold many experts in only a few layers. When a token moves through the network, the cache needs useful coverage at the layers where its routes are likely to miss.
The reference numbers make the pressure visible. A cold token can require around 11 GB of expert reads because it touches eight experts across the layers. That figure is not the model’s total disk size, and it is not the amount of RAM that must be permanently allocated to every token. It is the volume of routed data that may have to cross the storage boundary when the relevant experts are not already resident. The same logical token can therefore be cheap or expensive depending on the cache’s recent contents.
This also explains why replacing a single large cache with an abstract “percentage of the model” can mislead. Two placements with the same total bytes may cover different layers or different routing patterns. One may keep experts that the next requests repeatedly use; another may hold cold experts and leave the current route on disk. Colibri’s design attaches the cache to the model’s layer-and-expert structure because that is how the router addresses the weights.
The cache does not need to predict every future route perfectly to be useful. It needs to turn enough repeated accesses into hits that the storage path is no longer the dominant cost. Conversely, a cache can be too small to make a meaningful difference even when the model technically runs. This is the boundary between capacity and performance: sparse activation can make execution possible, but only locality and bandwidth determine how much of the possibility feels practical.
A small fast tier also changes the meaning of “active.” The roughly 40 billion active parameters describe the model’s computation at a broad parameter-count level. They do not say that all 40 billion must be loaded into one buffer before the token starts. Different layers expose different selected experts over time, and the engine can stream those choices through reusable working space. Active computation is therefore a statement about the logical route; residency is a statement about where the bytes serving that route happen to be.
The model is sparse, not pruned
This difference matters for quality. Pruning or routing substitution changes which computation the model performs. Colibri’s placement thesis instead keeps the cold expert set available and moves selected weights among tiers. In the intended quality-preserving path, limited RAM should produce more misses and longer waits, not a smaller expert vocabulary. A machine with less fast memory may be slower while still evaluating the same selected experts.
That is a stronger engineering requirement than merely returning an answer. If the engine silently drops a routed expert because the cache is full, then the token’s hidden state changes even if the process remains healthy. If it substitutes a resident expert for the selected one, it has turned a placement problem into a model-change problem. The fact that the output is fluent would not prove that the original route was preserved.
The same caution applies to the word “compression.” A compact representation may allow more expert bytes to fit in RAM or VRAM, but the representation still has to be interpreted with the correct shape and scale. A smaller file is useful only if the resulting computation remains within the quality and semantic boundary being claimed. Colibri’s core idea is not that every reduction is safe; it is that the model’s sparsity creates room to optimize movement without assuming that movement and meaning are the same thing.
There is an analogous distinction between running the full model and running a fast model. Colibri can place the complete routed set on disk and serve experts on demand, but a cold storage path may be painfully slow. The project’s own description treats that slow case as an honest floor rather than hiding it behind a claim of interactive performance. The hierarchy removes an all-or-nothing capacity barrier; it does not repeal storage latency, cache misses, or compute cost.
From geometry to execution
The useful mental model is now precise enough to carry forward. GLM-5.2’s total parameter count describes the complete space of possible expert behavior. Its active parameter count describes the sparse work associated with a token. Dense components stay close because every token needs them. Routed experts can move because the router identifies a smaller set at each layer. Expert count matters because it determines how many possible weights compete for cache capacity, while the route determines which ones are needed now.
From this model, three predictions follow. First, a machine may execute a model whose weights exceed its RAM and VRAM if it has enough disk capacity and a way to read selected ranges. Second, a larger cache can reduce storage waits only when it holds the experts the workload actually uses. Third, the same placement can be acceptable for one workload and unusable for another because the route pattern, cache temperature, and available bandwidth differ.
The remaining mechanical question is unavoidable: after a router selects expert IDs, how does Colibri turn those IDs into physical reads and then combine the results without paying for the same expert repeatedly? That is where the abstract distinction between dense and routed state becomes a concrete token path—one that begins with routing, forms a batch of unique experts, resolves hits and misses, and returns a weighted result to the hidden state.