Choosing and operating vLLM with clear eyes

The central question for vLLM is not whether it has enough optimizations. It is whether its coordinated treatment of token state—reusable KV blocks, token-aware scheduling, hardware-specific execution, and layered request interfaces—matches the workload and trust boundary an operator actually has. The practical task is to turn that model into a deployment hypothesis, test it with the right measurements, and keep every performance or compatibility claim within the conditions that make it true.

Start with the workload, not the feature list

A useful vLLM decision begins before a flag is chosen. Describe the request stream in terms of prompt length, shared-prefix frequency, output length, concurrency, modality, model shape, and latency objective. A document-questioning service with repeated system context is a different workload from a chat service with unrelated prompts and long answers. A multimodal assistant adds media decoding and encoder work. A mixture-of-experts model adds token routing and synchronization. These differences determine which part of the token-state pipeline is likely to dominate.

Then name the objective. TTFT asks how quickly a request receives its first token. ITL asks how regularly a stream continues after that. Throughput asks how much useful work the service completes over time. Capacity asks how many active contexts the available memory can hold. Isolation asks which users, processes, and networks may share state. Those objectives can conflict. A larger prompt budget may drain a prefill backlog while delaying existing decodes. More graph capture may lower steady-state latency while reserving memory and lengthening startup. Cache salting may protect tenant privacy while reducing the number of reusable prefixes.

The token-state model supplies a common starting point. A request has work already computed and work still required. Its KV state can be private, shared, cached, evicted, transferred, or recomputed. The scheduler decides which gap advances next, subject to token and input budgets. The model runner and attention backend turn that decision into device work. The output layer turns numerical results into text, structured data, tool calls, or reasoning fields. A deployment choice is sound when it improves the limiting link without creating a larger cost at the next boundary.

This framing also prevents a misleading comparison between features that act at different levels. Automatic prefix caching removes repeated prompt computation when prefixes align; it does not shorten long generated answers. Chunked prefill changes the order and size of work in a mixed batch; it does not reduce the total prompt tokens. Quantization changes the memory footprint of weights; it may create room for more KV state but depends on hardware and kernel support. Data parallelism adds replicas; it does not give those replicas one shared local cache. Disaggregation separates objectives and transfers state; it does not automatically increase throughput.

Make the configuration question conditional

If the model or its desired KV capacity does not fit on one GPU, the first decision is about placement and representation. Quantization may reduce weight memory, but the selected format must be supported by the target hardware, activation type, layers, and kernels. Tensor parallelism can shard parameters within layers, while pipeline parallelism can split layers across devices or nodes. TP usually benefits from a fast local interconnect; PP can accommodate an uneven split or a model that spans nodes but adds stage-to-stage latency and pipeline-bubble concerns.

Once a model replica fits, the question may become request throughput. Data parallelism replicates the model and distributes independent requests across engine groups. Its KV caches are independent, so a router that knows queue depth but ignores cache locality can send a request to a cold replica even when another replica already holds a reusable prefix. Larger data-parallel deployments can also put pressure on API-server and coordination processes. More replicas are useful only if the frontend and network can keep them supplied.

Mixture-of-experts models add a routing criterion. Expert parallelism distributes expert networks across a larger group, while attention may use tensor or data parallelism within each group. All-to-all communication and synchronized forward work become part of the model's critical path. If expert popularity is skewed, an expert load balancer or redundant experts may help, but redundant weights consume memory that could otherwise hold KV blocks. The operator must compare routing imbalance with the memory and communication cost of correcting it.

Latency objectives can justify a different boundary. Chunked prefill inside one instance may be enough when prompt bursts occasionally disturb decoding. If strict TTFT and tail ITL control requires different resource pools, disaggregated prefilling can place prefill and decode in separate instances and transfer KV state between them.