The Weight Store: Random Access, Quantization, and I/O
In JustVugg/colibri, the phrase “stream the experts from disk” describes a precise storage contract, not a metaphor for loading a model slowly. This explanation follows an expert from safetensors metadata to a bounded byte-range read and then into a quantized matrix operation. Readers will see why Colibri indexes shards instead of opening whole files, why adjacent matrices are coalesced into one working slab, and why format validation and quality measurement are part of inference correctness rather than optional housekeeping.
A checkpoint becomes an addressable store
A model checkpoint is usually presented as a collection of named tensors: a weight has a name, a shape, a data type, and a sequence of values. Colibri needs one more piece of information before it can stream that weight: where its bytes live. The storage layer indexes each tensor in the safetensors shards with its name, dtype, shape, file descriptor, byte offset, and byte length. Once that index exists, asking for an expert is no longer “open a 370-GB model.” It is “read the ranges that represent this expert’s matrices.”
That change in granularity is the foundation of the placement architecture. The router selects an expert at a layer; the cache decides whether a resident copy already exists; if it does not, the loader uses the index to locate the expert’s bytes. The file can remain cold except for the ranges that the current route needs. The engine uses positional reads so several workers can access different ranges without sharing a mutable file position, and it can advise the operating system to discard streaming pages after use. Those choices keep the process’s memory closer to dense state plus active cache instead of allowing every touched page from the entire checkpoint to accumulate indefinitely.
The index also has to work across a model split into many files. Colibri scans the shard headers, records the tensors they contain, and builds a name lookup after the scan. A directory can be extended across drives so different shards contribute capacity and concurrent expert loads can use more than one device. A mirror takes a different approach: it holds read-only copies and routes an expert deterministically to one copy, with a fallback to the primary when the chosen copy is missing or fails. The important commonality is that the logical tensor name still resolves to a specific, validated physical range.
That validation happens before the range becomes a computation. A safetensors header declares its own length, so Colibri bounds that length against the file and a maximum header size before allocating memory for it. Each tensor must provide a usable dtype, shape, and pair of data offsets. The offsets must be non-negative, ordered, and inside the shard. Shape products are checked for integer overflow. The loader refuses duplicate tensor names across indexed shards rather than silently picking whichever file happened to be scanned first.
The reason for these checks is easy to miss when a file format looks declarative. The header is not a trusted description supplied by the engine; it is input from a model file or mirror. If a shape overflows while the engine calculates an element count, a later size check can be fooled. If the declared byte span does not fit the file, a read can reach outside the shard. If the dtype is interpreted as the wrong kind of value, the resulting numbers may be plausible enough to escape a superficial smoke test. Refusing the container at the boundary is safer than allowing an apparently successful inference run over bytes whose meaning was never established.
The storage layer keeps float tensors and already-quantized tensors on separate paths.