Memory State with Meaning: Prefill, Agents, and Recurrent Context

FreeToken's bandwidth policy decides where a missing expert is computed, but agentic latency often depends on an earlier question: which parts of the previous context are still valid after the agent edits it? This chapter focuses on the state that follows a conversation across turns—ordinary full-attention KV, recurrent state, and sliding-window or compressed state—and explains why a tool-call boundary can be a better reuse point than an arbitrary checkpoint. The reader should come away able to tell when a prefix can resume safely and when the engine must recompute it.

A recurrent prefix needs a checkpoint, not just a matching string

Full attention and recurrent attention remember the past in different physical forms. A full-attention layer stores key and value entries associated with individual tokens. If a later request begins with the same token prefix, those entries can be reused directly, subject to the cache's normal ownership and locking rules. The state is distributed across positions, so a matching prefix has a natural endpoint.

A recurrent layer compresses its history into an evolving state. The state after token 1,000 is not a collection of independent records that can be safely spliced at token 500. It represents the result of processing the entire preceding sequence. If the engine wants to resume a recurrent layer at a prefix boundary, it needs a snapshot of the complete recurrent state at that boundary. A full-attention KV hit without the corresponding recurrent state is therefore not enough to resume the whole model correctly.

FreeToken makes this a second cache currency. The ordinary prefix tree tracks reusable KV pages; a hybrid radix cache can attach a recurrent-state snapshot to selected tree nodes. On a match, the engine walks the shared token prefix but truncates the reusable length to the deepest boundary that still owns a live snapshot. The request receives a fresh live state slot, and the snapshot is copied into it before the next forward. This copy-on-write step matters: the cached snapshot must remain stable for other requests while the new request continues changing its own state.

The lifecycle has distinct ownership transitions. A new request receives a live recurrent slot. During prefill, the forward path can write a frozen copy at an aligned internal boundary. The cache may donate that frozen slot to a radix node, which then owns it as a reusable snapshot. If a later request matches the node, the snapshot is protected while it is restored into a new live slot. When no request protects it and the state pool needs room, the snapshot can be evicted independently of the full KV pages. At completion, the final live state can be donated when its endpoint is a valid boundary; otherwise the request's state slots are returned without pretending that an unaligned state describes a shorter prefix.

This is more than bookkeeping. The snapshot and the token prefix must describe the same endpoint. If a state encodes 1,024 tokens but the tree node is treated as ending at 1,000, a future hit would restore a state that has already processed twenty-four tokens the request does not contain. FreeToken consequently aligns snapshot boundaries with the model's state-update granularity and the page geometry used by the shared cache.