Correctness Under Compression, Speculation, and Acceleration

JustVugg/colibri’s central performance idea is valuable only if readers can tell when an optimization still serves the same model. This chapter examines that question across compressed attention state, persistent prefix reuse, speculative decoding, quantized kernels, and CPU/GPU backends. The useful distinction is not simply “correct” versus “incorrect”: it is whether the evidence establishes identical bytes, identical tokens, acceptable numerical error, preserved task quality, or only a faster placement path.

Correctness has layers

The first layer is the input container. Colibri validates tensor names, shapes, offsets, dtypes, and byte ranges before a selected expert reaches a kernel. That establishes that the engine is reading the intended kind of data from the intended location. It does not establish that every later numerical operation matches another implementation, but without this foundation there is no meaningful parity comparison. A wrong scale layout or truncated range can produce an answer that looks fluent while invalidating every higher-level test.

The next layer is the numerical primitive. A quantized matrix kernel takes compact weight bytes and scale data and produces floating-point activations. Different kernels may unpack the same representation with different instruction sets, reduction trees, or accumulation orders. A primitive check can measure error against a reference calculation. That result is useful because it isolates the kernel from routing and language generation, but it is not yet a token-level claim.

Colibri’s default forward validation goes further by comparing teacher-forced outputs against a transformers oracle. In teacher forcing, the engine receives a known sequence and predicts the next token at each position rather than freely sampling a whole continuation. The project reports typical results around 30–32 matching positions out of 32, with two tiny near-tie positions described as sensitive to floating-point toolchains. This test asks whether the complete forward path—embedding, layers, attention, routed experts, normalization, and output head—produces the expected decisions on a controlled sequence.

A 32-position teacher-forced match is strong evidence for that path and that test. It is not proof that every context length, quantization format, accelerator, speculative mode, or serving state will remain bit-identical. A teacher-forced test may not exercise the same batch shape as a decode step. A backend may use a different accumulation order only during prefill. A persistent cache may contain state from a different model configuration. Correct interpretation requires naming what was compared and under which conditions.

The project applies the same layered logic to specialized state. GLM-style MLA attention does not keep the full expanded key/value data for every head. Colibri stores a compressed latent representation: the README describes 576 floats per token instead of 32,768, a 57-fold reduction. That is a memory property, not automatically a behavioral identity claim. The compressed state must be reconstructed in the way the attention core expects, and the resulting scores and values must remain within the validation boundary.

Persistence adds a lifecycle question. Colibri can save the compressed attention state in .coli_kv and reopen a conversation without re-prefilling it when the saved state matches the active model and configuration. The benefit is not just a smaller file. It removes repeated work when a client resends the same conversation and keeps the state alive across process restarts. The cost is that a persisted state can become stale, incompatible, or only superficially similar to the new request.

Prefix reuse therefore records the exact token IDs that were fed into the state. A later prompt can reuse a leading prefix only when those IDs match and there is new text to process.