The Expert Cache That Follows the Router
FreeToken's shared expert cache answers a precise question left open by the difference between prefill and decode: how can one GPU memory space serve full-layer movement in one phase and route-level residency in the other without confusing stale bytes for valid weights? This chapter follows the cache's ownership model and control loop. The goal is to understand what a slot represents, why the host pool remains authoritative, and how routing-driven LRU changes performance without changing the model's computation.
One logical expert, two physical levels of memory
The cache begins with a separation between authority and speed. The complete routed-expert pool stays in host memory. The GPU holds only a subset of those experts in a faster slot cache, and the non-expert weights remain resident on the GPU. If a slot is evicted, the model has not lost anything: the authoritative host bank can supply the expert again. GPU capacity therefore determines how often data moves and how quickly a route can be evaluated, while the host pool preserves the set of valid weights.
That separation is what lets FreeToken treat the machine as more than a small GPU. The host pool is not a cold archive used only at startup. It is a live source for two possible actions. A missing expert can be copied over PCIe into a GPU slot, where it may serve the current route and future routes. Or it can be evaluated directly by the CPU executor, leaving the source in place. The cache decides residency, but it does not become the only copy on which correctness depends.
A slot represents a complete layer-expert record, not an arbitrary fragment of one tensor. An expert normally has at least a gate/up projection and a down projection, and quantized formats add packed values, block scales, globals, or biases. FreeToken's bank machinery keeps the corresponding rows together as one logical object. When the cache admits an expert, every bank required by the selected kernel refers to the same layer and expert. The GPU operation can then consume a consistent record rather than combining a new gate projection with an old down projection simply because each happened to fit somewhere.
The physical representation can change without changing that logical contract. A BF16 bank stores ordinary matrix values. An NVFP4 bank stores packed four-bit values plus scale information. A block-FP8 bank keeps FP8 weights and block-scale data. MXFP4 and DS-FP4 use their own packed layouts, while native GGUF Q4_0 uses another. These formats do not share one byte shape or one dequantization routine. They share the more important identity: one row in every required bank belongs to one layer-expert, and the selected kernel knows how to read those rows.
Some format-specific values are small enough to remain GPU-resident, such as the per-expert global scales used by particular NVFP4 kernels. That does not contradict the complete-record rule. It means the record has a stable auxiliary component whose lookup follows the slot or layer, while the large movement-sensitive banks remain in the host pool and slot cache. “Unified” therefore describes the residency and identity model, not a promise that every physical tensor is stored in one array or one dtype.
The cache uses a flat logical identifier to make this agreement efficient. Conceptually, the identifier combines the MoE layer number and the expert number: layer * num_experts + expert. One map answers where that logical object lives (slot_for_id), while the reverse map answers which logical object currently owns a GPU slot (id_of_slot). The layer-specific source bank is still selected when bytes are copied, so a slot can be reused by a different layer without making the host layout global or uniform.
This arrangement solves several interface problems at once. The router speaks in logical layer-expert IDs. The cache translates those IDs to physical slot IDs for the GPU kernel. The copy path uses the same admission decision to select source rows. The CPU executor bypasses the slot translation but reads the same host banks by layer. A quantization method can change how the row is interpreted without changing who owns it. The interface is narrow: routing identifies an expert, the cache establishes residency or a CPU assignment, and the expert kernel consumes the result.
It also explains why the cache can be shared across all MoE layers. A fixed per-layer allocation would protect one layer from another but strand capacity when their working sets differ. In FreeToken's global space, an idle slot from one layer can be used for an active expert from another. The trade-off is interference: a burst of activity in one part of the network can evict entries that another layer might have reused. The global cache buys flexibility and makes the memory budget fungible, but it must maintain an exact owner map whenever bytes change hands.
The same principle connects the cache to prefill. Full-layer prefill movement uses the first two expert-layer regions as alternating buffers. Those regions are still physically part of the unified slot pool. When a new layer is copied into one, the previous id_of_slot owners of those slots are invalidated, and the corresponding slot_for_id entries are cleared. The buffer can then be addressed by expert number for the current layer. The decode cache is not allowed to retain a fiction that the overwritten bytes still belong to the old owners.
This is the first important systems judgment: a cache is not just a capacity number. It is a contract among the source of truth, the physical bytes, the logical identity map, and the consumer kernel. If those four elements agree, a small GPU cache can be an elastic performance layer. If they disagree, a hit can be worse than a miss because it silently returns the wrong expert.
How residency follows demand without leaving the device
During decode, the cache control loop begins after the router has produced top-k expert IDs for a layer. The IDs may contain duplicates because several token routes can select the same expert. FreeToken first treats the active set as unique layer-experts. It then checks each one against the layer's residency row. A resident expert is a hit. A missing expert becomes a candidate for admission, but it is not yet a valid GPU row.
The cache updates recency for the active hits and chooses victims for the candidates it will fetch. Its eviction policy is LRU: entries used more recently are protected relative to older entries, subject to the global capacity and the active set. The important behavior is not the label “LRU” by itself. It is that the policy runs against the routes generated by the current workload, so the contents can follow a changing working set instead of remaining frozen after load or prefill.
For each selected miss, the cache records two facts: which physical slot will be overwritten and which source row from the current layer's host bank must fill it. The copy path moves the matching rows for every required bank into those slots. Once the movement is scheduled, the logical route IDs are rewritten in place into the physical slot IDs. The routed GPU GEMM can then use one slot-indexed view of the bank cache. A hit and a newly admitted miss look the same to that final kernel; both are resident rows at the moment of computation.
That in-place rewrite is an important boundary between routing and execution. The router's original IDs identify model experts. The expert kernel needs positions in the cache's physical arrays. The translation cannot be performed once at startup because admissions and evictions change the map every step. It also cannot be delegated to a slow host round trip at every layer without undermining decode latency. FreeToken keeps the decision and its worklist on the device, using fixed-size buffers and a valid count so the dynamic number of misses does not force a dynamic graph shape.
A global LRU can outperform a static placement when routing has short-range locality. If adjacent tokens repeatedly select overlapping experts, a recent route leaves useful entries in the slot cache. A placement selected from a single prompt may capture the wrong working set once the agent moves to a different file, tool result, or reasoning phase. The paper reports this contrast on identical routing traces: at one stated capacity, FreeToken's LRU misses were lower than both a prefill-updated placement and a routing-blind static split. Those numbers are evaluation results for the paper's models and workloads, not a theorem that locality exists for every checkpoint.
The policy's success condition is therefore observable locality plus enough capacity to retain useful reuse. Its failure modes are equally concrete. A cold start has no resident history. An abrupt routing shift can make recently used experts irrelevant. A cache that is too small can thrash even when each step activates only a few experts. And the shared pool can experience cross-layer contention. In all of these cases the cache may still be correct—the host pool can refill any missing expert—but its miss rate and latency rise.
The prefill buffer creates a different kind of miss by design. It overwrites slots that may contain decode residents so that it can stream complete layers efficiently. Consider a slot that previously owned layer 0, expert 3. If prefill then uses that slot for layer 1, expert 3, the bytes have changed even though the expert number has not. The cache must clear the old flat ID and mark the old layer's lookup as absent. When a later decode route asks for layer 0, expert 3, it must be treated as a miss and reloaded from the layer 0 host bank.
This is why cache invalidation is part of the movement algorithm, not a cleanup operation. The test suite exercises the hostile sequence explicitly: one request loads and uses experts from one layer, another request materializes a different layer into every slot, and the first request routes back to its old experts. The correct result is a new copy of the original layer's rows. A stale mapping would produce a plausible tensor shape and a fast result while reading another layer's weights, which is precisely the kind of failure a performance benchmark may not catch.
The cache also has to coordinate two users of the same physical regions. During prefill overlap, one buffer is being read by the compute stream while the other is filled by the transfer stream. A release event makes reuse wait until the current layer's GEMMs finish. A beginning fence prevents a new prefetch from racing with earlier decode work that may still be reading the borrowed slots. When a buffer is invalidated, its old ownership and recency are cleared so decode will not treat volatile buffer contents as stable cache entries. The result is a deliberate, temporary residency class inside a shared pool.
The host source remains the fallback for every such transition. If an expert is evicted, its source row is still available. If prefill has overwritten a slot, the next decode admission can restore it. If a CPU path handles a miss, it reads the source without changing GPU residency. This is the second important judgment: the cache does not need to be perfect to preserve model semantics. It needs to be truthful about what it contains and what it does not.
There are costs to this flexibility. Every miss can require a copy and a cache update. LRU tracks recent demand but does not predict future routes. A larger cache improves the chance of a hit but consumes VRAM that could hold KV pages, graph buffers, or prefill overlap regions. A full-layer prefill may warm many experts for the next phase, yet it can also displace decode history. These are allocation and scheduling choices, not defects that one cache policy can eliminate.
The cache's device-side control also changes how one should interpret a “dynamic” implementation. Dynamic does not mean Python examines every route and decides synchronously. It means a captured kernel updates fixed-shape maps, selects valid work, and lets the GPU continue. The CPU is involved only when the selected execution mode assigns work there. The detailed question is then no longer whether the cache can find misses, but how the system should serve the misses that remain after locality has done its work.
That is the problem for the next chapter. A missing expert can be fetched into the GPU, where it may become a future hit, or computed directly from host memory, where it avoids a transfer but does not warm the cache. Which action is better depends on the actual bandwidth balance of the machine. FreeToken's q-star policy turns that residual miss set into a measured transfer-versus-compute decision.