How V1 turns requests into a living batch
In vLLM V1, the scheduler's job is to decide which token work can advance now, not to assemble one permanent batch and wait for it to finish. That distinction matters when a long prompt, several ongoing answers, and a request with cached state compete for the same KV blocks and GPU time. The useful question is how V1 represents progress and turns it into a repeated decision that protects latency without admitting more state than the engine can hold.
One progress measure instead of two phases
The ordinary description of inference divides a request into prefill and decode. That distinction remains useful for understanding cost, but V1 does not make it the scheduler's fundamental state machine. Instead, each request tracks how many tokens have already been computed and how many tokens it currently needs the model to process, including speculative tokens when speculation is enabled.
The scheduler's central comparison is between num_computed_tokens and num_tokens_with_spec. If the latter is larger, the request has work that has been admitted conceptually but not yet computed. The scheduler assigns some number of new tokens so that the computed count catches up. For a new request, the missing work may be most of a prompt. For a running request, it may be one next output token. For a request with speculative candidates, it may include tokens that are proposed ahead of verification. The same comparison can cover all three without requiring separate queues and rules for “prefill mode” and “decode mode.”
This is a small abstraction with a large consequence. A request's position in the lifecycle is no longer defined only by the kind of phase it is in. It is defined by the gap between state already established and state needed for the next step. A prefix-cache hit reduces that gap at the beginning. A chunked prompt leaves part of the gap for a later step. A generated token increases the context and creates the next gap. Speculative decoding temporarily widens the work to be checked. The scheduler keeps translating those changes into token allocations.
Two limits keep that translation from becoming an unlimited promise. The first is a token budget for the step: the total amount of token work the scheduler is willing to send forward. The second is an input budget used to constrain the batch's input shape, including room needed by speculative work. These budgets are not model-quality settings. They are controls over how much computation and state the engine tries to make active at once.
The scheduler considers running requests before promoting waiting ones. That ordering reflects a practical obligation: a request that has already begun generating should not routinely stop producing every time a new long prompt arrives. When the request list contains an ongoing answer and a newly arrived document, the answer may need only one more token while the document may need thousands of prompt tokens. V1 can schedule the answer first, then spend whatever token budget remains on the document. If the document cannot fit in the remaining budget, it is split rather than rejected merely because its prompt is long.
This is why a vLLM batch is “living.” Its membership and per-request token counts can change at every engine step. A request can finish and release state; another can enter; a long prompt can contribute a bounded chunk; a running answer can advance by one or several verified tokens. The batch is a recurring allocation decision, not a container whose shape is fixed until all requests end.
Why chunked prefill protects a mixed workload
Prefill and decode create different pressures. Processing a large prompt is generally compute-heavy: many input tokens must pass through the model before the first answer token is available. Decode is often memory-sensitive and latency-sensitive: many requests may each need a small next-token step, and delaying one step is immediately visible as a larger gap in the stream.
Chunked prefill lets vLLM combine these pressures in one step. A long prompt is divided into smaller token ranges, and those ranges can be scheduled alongside the next-token work of requests already running. The purpose is not to make the prompt shorter. It is to prevent one prompt's entire burst from monopolizing the engine while still giving the accelerator useful work to do.
V1's documented policy prioritizes pending decodes before scheduling prefill when chunked prefill is available. This improves the chance that a response already in progress continues at a reasonable ITL. The remaining token budget can then be used for a prompt chunk, which improves utilization compared with leaving the GPU idle while the system protects every decode request one at a time. The policy is a compromise between two visible outcomes: how soon a new request gets its first token and how regularly existing requests receive their next tokens.
The main tuning lever is the maximum number of batched input tokens. A smaller limit leaves less room for prompt chunks, so it generally protects inter-token latency by reducing the amount of prefill work that can sit beside decodes. A larger limit lets vLLM process more prompt tokens together, which generally helps TTFT and can improve throughput, especially when the model is small relative to the accelerator. But a larger limit can also make a decode wait behind a more substantial mixed batch. There is no value that wins for every traffic pattern because the limit is expressing a workload preference.
Consider a service with two request types. One receives short questions and expects a steady stream of generated tokens. The other submits long reports and asks for a summary. If the scheduler gave each report its full prefill in one burst, the short answers could pause for the duration of that burst. If it gave every decode absolute priority, reports could wait indefinitely and GPU compute capacity could be wasted. Decode-first chunking gives the short answers a recurring place in the step while allowing each report to advance in bounded pieces.
The same model can behave differently when the workload changes. If nearly every request is a short prompt followed by a long answer, prefix reuse and prompt chunking may matter less than sustained decode capacity. If requests share a long system prompt or document, cache hits can make their initial gaps smaller, leaving more budget for other work. If prompts arrive in bursts, a larger budget may drain the backlog faster but may increase tail ITL during the burst. The scheduler's mechanism is stable; the useful setting depends on which delay the service is trying to control.
The sequence emphasizes ownership. The scheduler decides how much work to attempt. The KV-cache manager decides whether the required state can be allocated and which blocks can be released. GPU workers execute the admitted tokens and return results; they do not decide which waiting request wins the next slot. If capacity cannot be made available, the scheduler changes admission rather than asking the worker to exceed the cache it was given.
When the KV pool becomes the admission boundary
A token budget limits work in time, but it does not by itself guarantee enough KV memory. Every newly computed portion may require additional blocks, and a running request's context continues to grow as it generates. Eventually, the engine may have enough nominal token budget for another request but not enough free blocks to hold the state that request would create.
V1 handles this through admission checks and preemption. Before scheduling a request, the KV-cache manager estimates the blocks needed for the current work and, where required, checks whether the full sequence can fit under the relevant retention rules. It can release blocks that are outside a sliding attention window, but it cannot simply discard state that an in-flight operation still needs. The manager also supports reserved capacity for sequences already partway through prefilling or loading state. That reservation is a guard against a superficially efficient decision that consumes all free blocks and strands work already underway.
If a request still cannot fit, the scheduler can preempt another request to create room. Under V1's default recomputation strategy, the preempted request's already-generated output is not necessarily lost, but the state needed to continue may have to be rebuilt when the request is admitted again. This is a deliberate trade: recomputation uses model compute, while keeping every request resident would require more KV memory or less concurrency. The older idea of swapping a request's KV state to CPU is not the default V1 strategy described by the project.
The choice of a victim depends on scheduling policy. With ordinary first-come, first-served behavior, the scheduler can remove a request from the end of the running set. With priority scheduling, it can choose the lowest-priority running request, using arrival order to break ties. The precise policy matters less than the invariant: admission is conditional on both fairness policy and state capacity. A request is not “running” merely because it exists in the server; it is running because the engine has reserved enough resources for its next step.
This explains why several apparently unrelated settings affect preemption. Increasing GPU memory utilization can leave more room for KV blocks, but it narrows safety headroom for other allocations. Lowering the maximum number of concurrent sequences or batched tokens reduces the amount of state competing at once, but may leave throughput on the table. Increasing tensor or pipeline parallelism can reduce the weight burden on each GPU and leave more memory for KV state, while also introducing synchronization or latency costs. The right response to preemption is therefore not always “add a bigger batch.” It is to determine whether the service is constrained by weights, active context, prompt bursts, or communication.
Watermarks and reserved blocks make the same point in a less visible way. The scheduler may keep some headroom when admitting waiting or previously preempted requests, especially when other sequences are already scheduled. That headroom is not wasted capacity in the simplistic sense. It is insurance that a request already in flight can reach a safe transition instead of being repeatedly interrupted by newcomers. A system that admits work until every block is occupied may show high instantaneous utilization while producing worse latency and more recomputation.
Reading the symptoms of scheduling pressure
A scheduler problem often first appears as a latency problem, but the useful diagnosis comes from separating the latency components. TTFT is affected by queueing and prompt processing. ITL is affected by how often decode work is delayed by other scheduled work. Preemption is evidence that KV capacity, not merely the nominal token budget, has become a limiting resource.
If TTFT grows while existing responses remain smooth, the service may be spending its budget on ongoing decodes or waiting behind a prompt backlog. A larger batched-token limit can help drain prompt work, but it can also disturb ITL. If ITL grows when long prompts arrive, the mixed batch may be admitting too much prefill at once; a smaller limit or stronger decode protection is a plausible hypothesis. If both latency dimensions worsen and preemption counts rise, the first suspect is capacity pressure: requests are entering with more active state than the pool can sustain, and recomputation is adding work on top of the original workload.
These are hypotheses, not automatic diagnoses. A request can have a high TTFT because tokenization or media handling is slow before the scheduler sees it. A decode can have a high ITL because model execution, inter-device communication, or a selected attention path is slow even when scheduling is orderly. To separate those cases, compare queue and preemption signals with the actual prompt and generation timing, then inspect CPU, GPU, and network behavior rather than changing every scheduler setting at once.
The important habit is to preserve the causal chain. A larger token budget changes how much work the scheduler places in one step. That changes how much prompt work can coexist with decodes. The resulting batch changes GPU execution time and KV allocation. If allocation fails, preemption may trigger recomputation, which creates more work and can further affect latency. A single observed slowdown can therefore be the downstream result of an earlier admission decision.
The scheduler as a time-and-memory contract
V1's unified scheduler does not make prefill and decode identical. It makes their progress comparable enough to arbitrate them in one loop. That is the design payoff: cache hits, prompt chunks, ordinary next-token steps, and speculative work can all be expressed as gaps between computed and required token state, then bounded by the same step-level budgets.
The cost is that an operator must think in trade-offs rather than defaults. Chunked prefill is valuable because it shares compute-heavy prompt work with latency-sensitive decoding, but its chunk size changes whose delay is protected. Preemption is valuable because it keeps the engine from failing whenever demand exceeds resident capacity, but recomputation makes over-admission visible later. Reservations can lower instantaneous utilization while preventing a more expensive stall.
A scheduler is therefore not an invisible queue between an API and a GPU. In vLLM, it is the point where time, memory, and request policy meet. The next useful test is to apply that contract to contrasting workloads: a shared long prefix, unrelated prompts, and a cache pool under pressure. Those cases reveal when the same mechanism removes work, when it merely rearranges work, and when capacity turns a fast path into recomputation.