Planning and Tuning for the Machine You Have

JustVugg/colibri’s plan and tune tools address a practical problem: the right placement policy depends on the checkpoint, the context size, the memory topology, the storage device, and the bottleneck currently limiting tokens. This chapter explains how Colibri turns those facts into a safe baseline and then tests changes against the real workload. The useful result is not an automatic collection of faster-looking flags; it is a way to decide what to measure, what a projected hit rate means, and when a speedup should be rejected.

A safe plan counts more than model weights

A naive planner would compare the model’s file size with installed RAM and report either “fits” or “does not fit.” That is not enough for Colibri. The file includes cold routed experts, but the running process also needs dense weights, runtime allocations, attention and KV state, working buffers, and room for the operating system. A meaningful plan must count the memory that is retained, the memory that can be reused, and the bytes that remain on disk.

Colibri’s resource planner analyzes the actual checkpoint. It scans shard metadata and separates dense resident contributions from per-layer expert contributions and other fixed allocations. It keeps the expert sizes associated with their layers because a cache slot is charged for the representation it may have to hold. It also accounts for the configured context and the state required to serve it. This produces a model geometry that answers “what will the engine allocate?” rather than merely “how large is the directory?”

The plan then divides the available resources into roles. The disk tier is cold backing for expert bytes that are not resident. The RAM tier includes the dense model, runtime reservations, sequence state, and warm expert cache. An optional VRAM tier holds hot experts on qualified accelerator devices. The three budgets are related: allocating more to one tier can leave less for another, and a plan that ignores that relationship can look generous while overcommitting the machine.

The context length is part of this arithmetic. A longer conversation requires more attention state, and multiple isolated serving contexts multiply that state again. Those bytes do not look like expert weights, but they compete for the same physical RAM on a host that is trying to cache experts. The planner therefore reserves sequence state and workspace before it turns the remaining memory into cache slots. This is why a machine can have enough total RAM for the model’s dense portion but still have little useful capacity for routed experts at a large context.

The plan also distinguishes a detected GPU from a GPU memory budget that the engine is allowed to spend. On a discrete card, free VRAM can be a placement resource after an explicit reserve. On a unified-memory system, accelerator and host allocations draw from one physical pool, so treating them as independent would double-count capacity. A GPU whose free-memory relationship has not been qualified is reported as hardware information but does not automatically steer placement. “The operating system sees a GPU” and “Colibri can safely fill this much VRAM” are different facts.

Already-used VRAM matters too. Planning while another engine instance occupies most of the card describes the remainder, not the machine’s normal capacity. Colibri surfaces that condition instead of quietly presenting a small hot tier as the hardware’s capability.