q*: Turning PCIe and CPU Bandwidth into a Decode Policy
FreeToken's q-star policy starts where an expert cache stops: after routing locality has produced a residual set of missing experts. This chapter asks why a miss should be treated as a choice between moving an expert to the GPU and computing it on the CPU, how measured bandwidth determines that choice, and how the implementation performs both branches without surrendering graph replay. The practical result is a way to estimate which direction the hybrid split should move on a particular machine, while keeping clear that it is a measured policy rather than a universal optimum.
A cache miss has two useful destinations
A decode step begins with a small set of unique experts that the router selected for a layer. Some are already in the GPU cache and can be evaluated there. The remaining m misses are absent from the fast slot space, but they are not absent from the system: their authoritative rows still sit in the host-resident expert banks. FreeToken gives each miss two possible destinations.
The first is the GPU. The engine copies the expert's banks over PCIe into a cache slot, runs the GPU expert kernel, and leaves the new resident available for a later route. This action pays transfer time now but can turn a recurring miss into a future hit. The second is the CPU. The engine leaves the expert in host memory, sends the activation and routing information to a CPU worker, computes the expert in place, and returns the partial result to the GPU. This avoids the immediate weight transfer, but it does not warm the GPU cache for the next token.
Neither destination dominates by definition. The GPU's on-package memory is much faster than host memory, but PCIe is the bridge through which a large expert must travel. The CPU already has direct access to the host bank, but its expert GEMV is bounded by host-memory bandwidth and CPU execution efficiency. The two actions also compete for the same host-memory subsystem: a DMA transfer reads the host bank while a CPU worker may be reading another expert. The policy must therefore balance current latency against future residency under the actual bandwidth relationship of the machine.
Let BP denote the measured bandwidth for moving expert data from pinned host banks to the GPU, and BH denote the host-side bandwidth of the CPU expert kernel. If the PCIe path is saturated, the paper models the CPU's concurrently available bandwidth as the residual max(BH − BP, 0). For S bytes per expert, fetching q of m misses and computing the rest gives the approximate branch times
- GPU/cache-fill branch:
qS / BP - CPU branch:
(m − q)S / (BH − BP)
Balancing those times yields the ratio q / (m − q) ≈ BP / (BH − BP), or the simpler fill estimate q* ≈ m · BP / BH. In plain terms, a faster PCIe link calls for more fetches, while a stronger CPU expert path makes it worthwhile to compute more misses in place. When host and PCIe bandwidth are close, q-star approaches m: nearly every miss should be fetched because the CPU has little residual bandwidth left. When host bandwidth is much higher, the CPU can absorb more of the misses.
The paper's Figure 2 makes the arithmetic concrete with four misses and a profiled PCIe-to-host balance of roughly one to four. It shows one miss moving over PCIe and three computed on the CPU.