Why Prefill and Decode Pull the System Apart
FreeToken's most important phase distinction is between prefill and decode: the first processes an existing prompt, while the second extends it one generated token at a time. This article focuses on why those phases stress the same MoE model in opposite ways, and that distinction gives us a practical test for local serving: expect broad expert movement during prefill, but reason about a small, changing set of cache misses during decode.
Prefill turns sparse routes into a transfer problem
An MoE router is sparse at the level of an individual token. A prompt, however, is not one token. Prefill processes the prompt as a batch of many tokens, and each token may choose a different subset of experts in every MoE layer. The relevant quantity for movement is therefore not the route of one token but the union of routes across the whole prompt. As the prompt grows, that union can approach the layer's complete expert set.
This is the first reason a naive “fetch an expert when it is requested” strategy behaves poorly. Suppose the first few prompt tokens ask for experts 3 and 19, then later tokens ask for 44, 71, and most of the remaining set. If the engine waits for each route to reveal itself, it repeatedly exposes host-to-device transfer on the critical path. The arithmetic remains sparse per token, but the weights that must pass through the interconnect become close to dense at layer scale. The GPU can spend its time waiting for data rather than evaluating the model.
There is also a granularity mismatch. The transformer processes layers in a known order, while the router reveals token-level choices only as activations reach each layer. A layer's order is predictable before the prompt runs; its exact expert requests are not. FreeToken exploits the predictable part. It does not try to predict every token's route before prefill. Instead, it starts moving the complete expert set for the next layer, using the layer sequence itself as the schedule and leaving the GPU to apply the router once the data is ready.
FreeToken therefore changes the unit of prefill movement. Instead of making a separate demand decision for every routed expert, it streams the complete expert set for the next layer. The implementation reserves two full-layer regions in the shared GPU expert cache. While the compute stream evaluates layer l from one region, a dedicated transfer stream copies layer l + 1 into the other. When the current computation releases its region, the two roles alternate.
The apparent waste—copying experts that some prompt tokens may not use—is intentional. During long prefill, the route union makes fine-grained selection a weak saving, while the cost of waiting for a sequence of small transfers is high. Loading the full next layer also removes a dependency: the transfer can begin before the next layer's exact token routing is known. What matters at that point is the layer order, which the transformer already fixes. FreeToken uses that predictable order to turn movement into a pipeline rather than a series of stalls.
The pipeline has a simple timing objective. If the transfer of layer l + 1 finishes before the computation of layer l releases the current buffer, movement is hidden behind useful GPU work. The exposed prefill time then approaches the slower of the two activities instead of their sum. The paper reports this behavior for one Qwen3.6 study: with overlap enabled, an 8,192-token prefill chunk completed in roughly 1.19–1.22 seconds on an RTX 5090, close to the time required to stream the measured expert pool at the link's practical bandwidth. Disabling the second buffer serialized transfer and computation and reduced reported prefill throughput by 19% at 4,000 tokens, 25% at 8,000, and 26% at 16,000. These are setup-specific measurements, not guarantees for every checkpoint or card, but they show what the pipeline is trying to hide.
That overlap depends on the two timelines actually remaining independent. The transfer stream must not overwrite a buffer while the compute stream is still reading it. FreeToken places events at the beginning and end of the buffer's use: the copy stream waits until earlier GPU work is safe, the compute stream waits until the next layer's copy is ready, and release is recorded only after the current expert GEMMs finish. The same coordination protects the cache's ownership table. A buffer is not merely a reusable byte array; it temporarily changes which logical layer-experts its slots represent.
The two buffers are not free scratch space. FreeToken borrows them from the same unified slot pool that later serves decode. For a model with E experts per layer, prefill overlap requires at least 2E slots. Those slots cannot simultaneously represent ordinary decode residency while they are being overwritten with the current and next full layers. The cache machinery invalidates the old ownership mapping when a buffer is reused, marks the buffer ready only after the transfer stream's event fires, and releases it after the layer's GEMMs finish. Without those ownership transitions, a later decode lookup could believe that an old layer-expert still lived in a slot whose bytes now belong to a different layer.
This shared-pool choice has an important benefit as well as a cost. A separate prefill cache would avoid some interference, but it would duplicate scarce GPU memory and require a phase handoff. With one pool, experts that survive the prefill movement can become useful decode residents. The price is coupling: a long prefill can evict or overwrite decode entries, and the decode cache must be able to discover that its former owners are gone. FreeToken chooses a single residency space so that memory can flow between the phases instead of being stranded in a pool dedicated to only one of them.
Finite memory also determines how much prompt work can be admitted at once. FreeToken's scheduler reserves the page span needed for the prompt and requested output, then chunks a long prefill when the configured budget or a model-specific window pool cannot hold the whole extension. Chunking limits the working set and lets the engine free state that has moved outside a sliding window. It also introduces boundaries: some recurrent snapshots are useful only at aligned points, and the current multimodal path requires a multimodal prompt to fit in one prefill chunk. These are not incidental scheduler details. They are the conditions under which hiding movement remains safe.
Chunking does not change the phase's basic strategy. Each admitted chunk still advances through the layers, and each layer can use the alternating buffers when the capacity contract is met. What changes is the lifetime of the prompt's temporary resources. The scheduler must account for pages needed by the full prompt and output, while the window-aware paths must return out-of-window entries before allocating the next chunk. A larger chunk can reduce scheduling overhead, but it also consumes more KV capacity and may leave less room for the expert buffers. A smaller chunk improves admission flexibility but exposes more boundaries and more opportunities for the transfer pipeline to be interrupted.
The result is a useful way to read prefill performance. If a long prompt is slow, the cause may not be the model's dense compute alone. The engine may be exposing a large expert transfer, failing to establish asynchronous overlap, or chunking against a limited window pool. Conversely, a fast prefill result does not tell us that decode will be fast: the full-layer buffers may have done their job while leaving a different cache state for the token-by-token phase.
Decode replaces broad movement with a miss decision
Decode starts from the opposite workload shape. The engine has already processed the prompt and now advances the sequence one token at a time, or a small batch of sequences at once. A token still goes through multiple MoE layers, but it activates only a few experts in each layer. The immediate data movement is consequently proportional to the unique experts selected for that step, not to the complete layer.
That makes on-demand movement reasonable, provided the engine has a GPU cache. The router emits logical expert IDs. The cache checks which of them are resident, leaves hits available to the GPU expert kernel, and identifies the misses that need another path. A newly fetched expert can serve the current token and remain in the cache for a later token. The small-step working set is not guaranteed to be stable, but it is small enough that recency and capacity can matter.
The word “unique” matters here. Several tokens in a decode batch, or several top-k positions for one token, may select the same expert. The movement path should not pay for duplicate copies, and the cache should update one ownership record for one layer-expert. FreeToken's device-side admission path deduplicates the active IDs, compares them with the residency map, and schedules only the rows that are genuinely absent. That is a different optimization target from prefill's full-layer stream: decode gains from avoiding unnecessary rows, while prefill gains from avoiding a long chain of route-dependent stalls.
The contrast with prefill is important. During prefill, the question is “How can the engine move almost an entire layer without making the GPU wait?” During decode, the question is “Which few missing experts should be made available now, and which can be served another way?” Loading the whole layer for every generated token would waste transfer bandwidth and erase the advantage of sparse routing. Conversely, trying to discover and transfer every prefill route one by one would make a long prompt pay the movement cost in fragments.
The decode cache also has a different correctness pressure. Prefill's double buffers repeatedly overwrite a fixed portion of the GPU cache, while decode's logical-to-physical mapping changes as experts are admitted and evicted. The mapping must follow the bytes actually in a slot. FreeToken's tests include a deliberately hostile case: one request loads experts from one layer, a prefill for another layer overwrites all slots, and the original request later routes to its old experts again. The expected behavior is a miss and reload, not a stale hit. That test captures the boundary between a fast cache and a correct cache.
A static placement policy cannot resolve that boundary in advance. It can decide which experts live on the GPU at load time or after one observed prefill, but routing can shift with every token and workload. A cache that follows demand instead treats residency as a current observation: a hit refreshes an entry, a miss can replace an old entry, and the next step sees the new map. This does not guarantee a hit—an abrupt change can defeat any recency policy—but it lets the machine respond to the generation it is actually serving.
The two phases can be compared directly:
| Question | Prefill | Decode |
|---|---|---|
| What is being processed? | Many prompt tokens in a layer-ordered pass | One new token or a small active batch |
| What does routing imply for movement? | The union of routes can cover most of the layer | Only a small set of unique experts is active at once |
| Useful movement unit | A complete next layer | Individual missing expert rows |
| Main latency threat | Exposed host-to-device transfer and repeated prompt work | Cache misses, evictions, and the way residual misses are served |
| Shared resource tension | Two temporary full-layer buffers consume GPU slots | The same slots must retain useful residents for future tokens |
The table is not a claim that the phases are isolated. They share the host expert pool, GPU memory, model weights, page accounting, and scheduler. Prefill can seed the entries that decode later uses; decode can change the cache state that the next prefill sees. The point is that the same shared resources are consumed differently. A useful serving design must coordinate the interaction instead of optimizing each phase as if the other did not exist.
At this stage, it is useful to distinguish three kinds of performance. Time to first token includes prompt processing and therefore reveals the cost of broad prefill movement and context recomputation. Decode throughput measures the repeated extension loop after the prompt has been accepted. Per-token latency is affected by the number and size of residual misses, cache locality, GPU expert computation, and—when enabled—the CPU path. A single average token rate can hide which of these regimes is failing.
For example, an engine might have a respectable steady-state decode rate after its working set has warmed, yet deliver a poor first response because the initial prompt streamed a large expert pool serially. Another might prefill quickly by using aggressive GPU memory, then suffer decode stalls because its placement leaves too few useful slots for the routes that follow. FreeToken's design makes those trade-offs visible: the prefill buffers borrow from the unified cache, and the decode policy has to account for what remains resident after that borrowing.
The distinction becomes even more visible for an agent. A tool-using session may alternate a short generation with a long prefill of the updated conversation. If the agent removes an old thinking block or replaces a tool output, the next turn can pay the broad prefill cost again even though only a suffix changed. If the session is already decoding, however, the active route may remain small and partially cacheable. A system that optimizes only steady-state decode can therefore look impressive in a short benchmark and still feel slow at the start of every real turn.
The workload phase also changes what “warm” means. A warmed expert cache can reduce decode misses, but it does not automatically eliminate the transfer required by a new long prompt. A cached prompt prefix can reduce recomputation, but it does not automatically place every expert needed by the new suffix on the GPU. The relevant warm state is a combination of model weights, expert residency, attention state, and the current request phase. Those layers of state are why a service-level benchmark must report more than one number.
FreeToken treats the two phases as related but not interchangeable. They share the authoritative host expert pool and the GPU slot space, so a prefill can seed the cache that decode will use. But they use that space at different granularities: full-layer buffers for prefill, route-level residency for decode. The shared space creates coupling—overlap consumes slots and invalidates old owners—yet it also avoids maintaining two disconnected copies of expert state.
The practical diagnostic follows directly. For a local MoE deployment, ask first whether the workload is prefill-heavy, decode-heavy, or alternating between them. Long prompts and frequent agent turns put pressure on transfer hiding and reusable context state. Long generations with a stable route pattern put more value on adaptive expert residency and miss handling. A narrow PCIe link, a strong CPU memory subsystem, a limited VRAM budget, or a model with a special attention cache can change the answer again.
The next step is to open the shared cache itself. Its central question is not merely how many experts fit, but what a slot means, who owns the authoritative copy, how a route becomes a resident layer-expert, and how the system repairs that identity when prefill and decode compete for the same memory. Once those rules are clear, the remaining decode misses can be assigned deliberately rather than treated as an unavoidable transfer bill.