The memory behind the next token
For vLLM, the key question is not simply how much memory a model's weights occupy. It is how the engine stores the attention state created by every active request, shares the part that is reusable, and releases the part that is no longer needed. The KV cache is where that question becomes concrete, so understanding its lifecycle lets a reader judge when vLLM's block management and automatic prefix caching will actually help.
Why the next token needs yesterday's state
A decoder-style language model generates an answer one token at a time. To choose the next token, it needs to relate that token to the prompt and to the tokens already generated. During the model's attention computation, each processed token contributes key and value information that later tokens may consult. vLLM retains those key/value pairs in a KV cache rather than recalculating the entire preceding context at every step.
That retained state grows with the request. A short question followed by a short answer needs less of it than a long document followed by a long conversation. Meanwhile, a serving engine is processing many requests with different lengths and at different points in their lifecycles. If each request received one large, private, contiguous memory region, the engine would face two problems at once: unused gaps would appear when requests finished or grew unevenly, and a shared beginning would be stored repeatedly even when its computed state was identical.
vLLM's current V1 core treats KV state as a pooled resource divided into blocks. A block holds a fixed range of token state for a cache group. The request still has a logical sequence of tokens, but the physical blocks holding those tokens can be selected from a common pool. This is the practical idea behind page-oriented attention: separate the logical position in a context from the physical location where its state resides.
The separation matters because allocation and computation are no longer the same event. The scheduler can ask the KV-cache manager which blocks already represent a request's computed prefix, allocate additional blocks for tokens that still need to be processed, and remove blocks that a sliding attention window no longer needs. The model runner then uses the resulting block mapping when it executes attention. The exact layouts and block sizes depend on the model and attention backend; the stable idea is ownership and lifetime, not one universal kernel layout.
A block can be in one of several practical conditions. A newly allocated block is writable by its current request. Once it is full and its contents have been identified, it may become a prefix-cache entry. If another request uses the same entry, the block's reference count protects it from eviction while both requests depend on it. When no request needs a cached block, it can remain available as an eviction candidate until the pool needs its space. A partial tail may be useful to its current request but is not normally a reusable full prefix block yet.
The important transition is not “memory becomes fast.” It is “state changes ownership.” A cache hit lets a new request adopt already-computed blocks, but those blocks must remain intact while the old request or the new request uses them. A cache miss does not make the request impossible; it means vLLM must allocate space and perform the computation again. The pool therefore supports both reuse and recovery through eviction and recomputation.
Sharing requires explicit ownership
The V1 block pool pre-allocates its KV blocks when the cache manager starts. It also maintains a free-block queue and a mapping from cache keys to physical blocks. This arrangement avoids creating a new Python object for every token or request and gives the engine a complete view of which blocks are available. It is closer to managing a fixed inventory than to asking the operating system for an independent allocation whenever a prompt arrives.
Reference counting supplies the safety rule for sharing. A cache hit “touches” the matching blocks. If a block had no active references, it is removed from the free queue before its reference count increases; otherwise it could be selected for eviction while the new request was adopting it. The block can then be associated with more than one request without being mistaken for idle memory.
The same rule limits writing. A request cannot freely mutate a block merely because that block appears in its logical prefix. If the block is shared or still represented by a cache entry, changing its contents would silently corrupt the state observed by another request and would invalidate the meaning of the cache key. vLLM's allocation and cache-management logic consequently distinguishes blocks that are private and writable from blocks that are shared or cached. Extensions to a context are handled by allocating or copying the state needed for the new portion rather than assuming that an existing shared prefix can be edited in place.
Freeing is also ordered. When a request finishes, vLLM releases its blocks in reverse order so that the less reusable tail is placed earlier in the eviction path. A long prefix is more likely to help another request than the final few tokens of a particular answer. The block pool further separates blocks that have cache identities from blocks that never became reusable: cached blocks can be kept for later prefix hits, while ordinary blocks can be reclaimed more eagerly for locality.
When capacity is needed, the free-queue head is the least recently used cached candidate. Eviction removes the block's cache mapping and its hash metadata before the physical block is allocated again. This is an important property of a cache-backed serving engine: eviction is not a correctness failure. It only changes the future request from “reuse this state” to “recompute this state.” The correctness requirement is that a block must not be evicted while a live reference still protects it.
The distinction between physical blocks and logical context also explains why vLLM can support more than one cache behavior. A model may use a sliding window, a hybrid attention pattern, or recurrent state, and the KV manager can coordinate different cache groups. Some groups may retain longer prefixes than others. The reader should therefore resist the simplification that every token is stored forever or that every model has the same cache geometry. vLLM's promise is a resource-management framework whose exact retention policy follows the model and configuration.
How vLLM recognizes a reusable prefix
Automatic Prefix Caching, or APC, answers a different question from ordinary block allocation. Allocation asks, “Where can this request's next state fit?” APC asks, “Has another request already computed this beginning?” vLLM uses a hash chain for cacheable blocks. The key for a block incorporates the hash of its parent prefix, the token IDs in the block, and extra identity values when they matter.
The parent hash makes position part of identity. The same local token sequence should not match if it appears after a different earlier context. Including the block's exact token IDs reduces the chance that a hash collision makes different content look equivalent. Extra keys account for conditions that token IDs alone do not capture. A LoRA adapter can change the effective model computation, and a multimodal placeholder sequence may stand for different images or other media. Those identities must participate in the cache key as well.
By default, vLLM caches full blocks. That boundary is easy to overlook. Suppose a cache block holds four tokens and a new prompt shares ten tokens with an earlier prompt. The first eight tokens form two complete reusable blocks, while the remaining two do not fill the next block. The new request may still need to process that incomplete portion. Prefix caching does not turn an arbitrary character or token overlap into a free computation; it reuses the aligned, fully identified state that the cache can safely adopt.
There is also a subtle consequence of V1's append-only block tables. Imagine a first request with prompt tokens A B C D E F and generated tokens G H I, using blocks of four tokens. Its first block contains A B C D. As decoding proceeds, a second block becomes full with E F G H and is entered into the prefix cache. When the same prompt is submitted again, the second request can reuse the first block, but its own block table has already been allocated with a new physical block for the continuation. V1 keeps the table append-only instead of replacing that new block with the old one in the middle of execution. For a time, two physical blocks may represent the same hash. The duplicate is harmless because the request's current mapping remains stable; the redundant block is eliminated when the request is freed.
The cache manager also limits an apparent perfect hit. If every prompt token is already cached, the engine still needs the final token's logits to determine the next result. The lookup therefore stops short of treating the entire sequence as completed state, and block alignment can mean that more than one token is recomputed. This is a small but important reminder that cache reuse is constrained by the output that must be produced, not just by whether bytes happen to exist.
The workload decides whether reuse matters
APC is especially useful when many requests share a long beginning. A document-query service may repeatedly ask different questions about the same manual or report. A multi-turn assistant may send the same conversation history with a new user turn. In both cases, the first request pays the cost of prefilling the shared context, while later requests can avoid much of that repeated prompt computation.
The benefit stops at the boundary between prompt and generation. A cache hit can reduce the time spent processing the shared prefix, but it does not generate the new answer on the caller's behalf. If answers are long, decoding may dominate end-to-end latency. If prompts are mostly unrelated, the cache has little to reuse. A deployment should therefore measure cache queries and hits alongside TTFT, generation length, and decode latency instead of treating “APC enabled” as a performance result.
Cache identity is also a privacy decision. In a multi-tenant service, one tenant may try to learn whether another tenant has already submitted a guessed prefix. A cache hit can make prefill faster, creating a timing signal even though the cached text is never returned directly. vLLM supports an optional cache_salt that is mixed into the first block's identity. Requests with different salts cannot reuse one another's salted prefixes, so a tenant-scoped secret can trade some cache efficiency for isolation.
That salt should be an unpredictable secret, not a visible user name or account number. If an attacker can guess the other tenant's salt, the isolation goal is lost. Conversely, a single-tenant service may omit salting when cross-user prefix privacy is not a concern and maximum sharing is more valuable. The choice belongs to the trust boundary of the deployment, not to a universal performance recipe.
The result is a more precise mental model of vLLM's memory system. The KV cache is neither an untouchable model constant nor a disposable scratch buffer. It is a pool of evolving state with owners, identities, reuse opportunities, eviction rules, and recomputation fallbacks. Once that state is visible, the next question follows naturally: when several requests have different amounts of cached and uncached work, how does vLLM decide which tokens advance in the next execution step?