The first operating model: memory, time, and token progress

The clearest way to test vLLM's design is to place three workloads beside one another: repeated questions about the same long document, unrelated prompts followed by long answers, and a busy service whose KV pool is nearly full. The mechanisms already visible—block reuse, token budgets, decode priority, and recomputation—do not produce the same result in each case. Comparing them gives an operator a first useful judgment about whether a deployment is limited by prompt work, decode latency, cache capacity, queueing, or recomputation.

A shared document: the cache changes where work begins

Imagine a service that answers questions about an annual report. The first request includes a long system instruction and the report itself, followed by a question about revenue. vLLM must process the report and create the corresponding attention state before it can generate an answer. As the request proceeds, complete blocks in that prefix become eligible for automatic prefix caching.

A later request sends the same instruction and report but asks about a different section. The two requests do not have identical complete sequences, because their final questions differ. They can still share the aligned beginning. The prefix-cache lookup follows the chain of block identities until the requests diverge. The later request can adopt the blocks representing the shared report rather than recomputing that portion of the prompt.

That changes the later request's starting point in the scheduler. Its num_computed_tokens is advanced by the reusable prefix, so the gap between computed state and required state is smaller. The remaining question still needs to be processed, and the model still has to generate a new answer. Reuse removes repeated prompt computation; it does not turn the new response into a lookup.

The distinction matters for a multi-turn assistant as well. Each turn may repeat the conversation history and add one new user message. If the history is long and aligned with complete cache blocks, later turns can avoid much of the same prefill work. But the new turn and all generated tokens remain new work. A service may see a substantial improvement in time before the first token while seeing little change in the time between tokens during a long answer.

Chunking still matters in this scenario. The first request may be too long to process in one scheduler step, so vLLM can advance it in bounded pieces. Once the shared blocks exist, later requests can start farther along, leaving more of their token budget for the new question or for other requests. The cache and the scheduler are doing different jobs: the cache reduces the amount of work that must be performed, while the scheduler decides when the remaining work joins the batch.

There are two boundaries to keep in view. First, vLLM normally caches complete blocks, so a shared prefix that ends partway through a block does not make that partial state automatically reusable in the same way. Second, even a nearly perfect prompt hit does not remove the need to obtain logits for the next output. Cache efficiency is therefore governed by token alignment and by what the caller still asks the model to generate.

Unrelated prompts: the scheduler still matters when the cache is cold

Now consider a general assistant serving unrelated questions. One user asks for a travel itinerary, another asks for code, and a third submits a private document that no other request has seen. Prefix caching has little common material to reuse. Each request must perform most of its own prefill, so enabling APC does not by itself change the amount of prompt work very much.

This does not make vLLM's scheduler irrelevant. The requests can still have different prompt lengths and arrive at different times.