The Model Does Not Have to Fit—It Has to Be Placed
JustVugg/colibri is an open-source inference engine built around a specific question: can a frontier-scale sparse model run on hardware that was never designed to hold all of its weights? This opening focuses on the project’s central answer—place different parts of the model across VRAM, RAM, and storage—and on the judgment that follows from it. Readers will gain a working distinction between making a model fit and making it run, along with a clear boundary: slower placement may be acceptable, but silently changing the model is not.
The problem is not model size alone
The usual way to think about running a large model is simple: load its parameters into fast memory, then compute. If the parameters do not fit, the machine cannot run it. That picture works for a dense model whose weights are broadly involved in every token. It is incomplete for a mixture-of-experts, or MoE, model.
An MoE still contains a very large collection of weights, but a router selects only some experts for each token at each layer. The untouched experts are not deleted or approximated away. They are part of the model, waiting for a later routing decision. The important fact is that the complete parameter set and the per-token working set are different things.
That difference is temporal as well as numerical. The router makes a new choice as the token moves through the network, so the engine does not need every expert at the same instant. It needs the dense state that carries the token forward, plus the routed matrices required by the current layer. Once that layer has contributed its result, a working slot can be reused for another expert. A later token may request the same expert and find it warm, or request a different one and force a new read. The model remains large; the fast working set is allowed to move.
This is not the same as pruning the model. Pruning would remove or permanently alter parameters. Colibri instead keeps the cold experts on disk and asks the router to identify the bytes that matter now. That distinction is what makes a storage hierarchy relevant to inference: a disk can be too slow to serve arbitrary dense computation, but it can still serve a sparse sequence of selected expert ranges if the engine manages locality and overlap carefully.
Colibri’s reference case makes the difference unusually visible. Its documentation describes GLM-5.2 as a 744-billion-parameter MoE with roughly 40 billion parameters active for a token. The dense portion—attention, embeddings, and shared components—is described as about 17 billion parameters and can be kept resident in roughly 9.9 GB at int4. The routed portion contains 19,456 experts across the main layers and the model’s draft head. At about 19 MB per int4 expert, that routed collection occupies roughly 370 GB on disk.
Those numbers do not mean that a 25-GB computer has somehow acquired 370 GB of RAM. They mean that the machine can keep the dense foundation and a changing selection of experts close to the processor, while treating the remainder as a read-only backing store. A cold token can still require around 11 GB of expert reads. But the relevant question has changed from “where can I hold every parameter?” to “which bytes must be available for this token, and where can they wait until they are needed?”
Placement instead of an all-or-nothing fit
Colibri turns that question into a three-tier view of memory. VRAM is the fastest optional home for hot experts. RAM holds the dense state and a warm cache of routed experts. NVMe or another local disk holds the cold expert set. The tiers contain different copies or representations of the same model weights; they are not three different models.
This is why the project compares its weight movement to a just-in-time compiler. A JIT does not compile every possible path before a program starts. It watches which paths are actually used and spends resources on the ones that become hot. Colibri applies a similar idea to expert weights: routing history and current demand help determine which experts deserve scarce fast capacity, while misses are fetched when the router proves that they are needed.
The placement decision changes the cost of a computation. An expert already in RAM can be computed immediately. An expert on disk introduces a read, and a slow or nearly full cache makes that wait visible. If a GPU tier is available, a selected resident expert may be executed there instead. In the project’s intended quality-preserving modes, these locations are alternatives for serving the same routed expert, not opportunities to substitute a more convenient one.
That promise does not make the system fast by declaration. The project’s own reference benchmark reports about 0.05–0.1 tokens per second for cold decoding on a roughly 1-GB/s development drive. Other documented machines reach around 1 token per second when most experts are resident in RAM, while a six-RTX-5090 experiment reached 5.8–6.8 tokens per second after the full expert set was resident across VRAM and RAM and disk waits disappeared. These figures are not points on a universal product curve. They show the consequence of moving the same model’s weights closer to the active computation.
The tiers also impose different kinds of pressure. RAM is not just a warehouse for experts: it must leave room for the operating system, the dense model state, the attention cache, and the program’s working buffers. VRAM is faster but limited by device capacity and, on multi-GPU systems, by movement between devices. Storage offers the capacity that makes the full model possible, but every cold read adds latency and can become the dominant part of a token’s time. A plan that fills one tier without reserving space for the others can make the whole run less reliable or less useful.
This is why a warm run should not be confused with a cold run. Repeated prompts can make the same experts stay in the cache, while a changing workload can evict them and expose the disk again. The project records this difference rather than hiding it: a cache-rich machine may be limited by CPU matmul, whereas a small-RAM machine may never get past storage wait. “Runs locally” is therefore a capacity statement. “Feels interactive” is a separate performance claim that depends on the machine and workload.
The diagram’s most important relationship is not the speed difference between the bottom boxes. It is the fact that all three are downstream of the same routing decision. The hierarchy is useful because sparse activation creates a reason to move only part of the full parameter set at a time.
What Colibri is actually promising
Colibri’s documentation is unusually direct about the boundary between performance and semantics. Limited fast memory may reduce speed; it should not quietly redefine the model. The default policy therefore treats router decisions and model precision as part of the behavior that must be preserved. An optimization is not accepted merely because a local kernel benchmark improves or because a smaller representation fits more experts.
That distinction matters because several tempting shortcuts would solve a different problem. Reducing the number of routed experts can cut disk traffic, but it can also change the computation. A more aggressive quantization can save bytes, but it can change logits and quality. A predictive cache can improve residency, but a wrong prediction can consume bandwidth without contributing to the current token. Colibri includes experimental controls for some of these ideas, but its own documentation marks them as experiments rather than silently turning them into the definition of the engine.
The rule is practical, not philosophical. If an expert is absent from a fast tier, the quality-preserving response is to wait for that expert or let the request run more slowly. If the system instead chooses a different expert, drops too much of a routed set, or interprets a quantized payload with the wrong scale layout, it may produce an answer that looks normal while no longer representing the same computation. A speed statistic cannot reveal that by itself. The project therefore treats model identity, router behavior, precision, and end-to-end output as things that need explicit checks whenever an optimization crosses one of those boundaries.
This also gives local users a better expectation of what “consumer hardware” means here. A small machine can be a valid host for the experiment without being a comfortable daily assistant. It may need a large disk, tolerate long cold waits, and improve only after its own prompts have warmed the cache. That is not a failure of the placement idea; it is the cost that the hierarchy exposes instead of hiding behind an unavailable capacity requirement.
The result is a deliberately asymmetric promise. Colibri is willing to be slow when hardware capacity is scarce. It is willing to test different schedules, cache policies, storage layouts, and kernels. It is not supposed to hide a semantic trade-off inside an apparently harmless speed setting. The project describes this as having no service-level guarantee on speed but a hard guarantee on semantics. That is a design aim and a set of validation obligations, not proof that every backend and every numerical path will produce identical text.
A different way to judge local inference
This perspective also explains why one headline speed number tells us very little. On one machine, the binding constraint may be disk bandwidth because most selected experts are cold. On another, more RAM may turn those reads into cache hits and expose CPU matmul or memory bandwidth. On a large heterogeneous host, the next limit may be GPU compute or the links between devices. A successful optimization moves the bottleneck; it does not abolish bottlenecks.
The project’s open questions follow directly from that reality. Does routing history generalize to new prompts, or merely memorize a narrow workload? Can multiple SSDs improve end-to-end decoding rather than only a storage microbenchmark? Can an automatic planner choose well across laptops, NUMA hosts, and multi-GPU systems? When does speculation repay the extra routed work? When can CPU and GPU activity overlap without moving the wait somewhere else?
So Colibri should not be understood as a claim that every consumer machine can deliver hosted-service latency from a frontier model. It is a demonstration that model scale, fast-memory capacity, and practical executability are not identical constraints. By making the weight hierarchy visible and measurable, the project turns an apparent impossibility into an engineering trade-off that can be tested.
The next question is therefore concrete: once a token arrives, how does Colibri know which experts to request, which copies to reuse, and which waits can be hidden? Answering that requires looking at the MoE geometry behind the placement idea rather than treating the model as one indivisible block.