Inside the hot path: from scheduled tokens to usable output

In vLLM, scheduling a token is only the beginning of execution. The engine still has to move that decision across process boundaries, choose an attention implementation that fits the model and hardware, prepare a shape that the compiled path can accept, and turn model logits into output a caller can use. This chapter follows that hot path so a reader can see why vLLM's speed depends on coordinated interfaces rather than on one universally fastest kernel.

The engine divides responsibility before it divides work

The first boundary is between the API-facing process and the engine's inner loop. The API server receives HTTP requests, tokenizes text, loads multimodal inputs when needed, and streams results back. It does not own the GPU model state or decide which waiting request should advance. That work belongs to the engine core, which runs the scheduler, manages KV-cache state, and coordinates execution.

GPU workers sit beyond the engine core. Each worker owns a GPU, loads the model weights, and executes forward passes. The engine core sends a scheduled batch to those workers and receives model results. Sampling can happen as part of the model-execution path or as a separate asynchronous operation, after which the scheduler updates request state and the frontend can continue processing the response.

This division gives vLLM a place to overlap work. The core can prepare or schedule another batch while workers execute the current one. With pipeline parallelism, a queue of in-flight batches helps keep successive pipeline stages occupied instead of waiting for one batch to travel through every stage before another begins. The benefit is not that more processes automatically make inference faster. The benefit comes when each boundary carries the right state at the right time without forcing the critical loop to wait unnecessarily.

The cost is easy to underestimate. Every process competes for CPU time, and the engine core's scheduling loop is especially sensitive to starvation. Tokenization, chat-template rendering, media loading, detokenization, and streaming all consume CPU resources even though the model's arithmetic runs on an accelerator. A deployment with many GPUs but too few physical CPU cores can therefore show disappointing GPU utilization: workers are available, but the process that prepares or dispatches their next work is late.

The flow below follows ownership rather than implementation names. Input processing creates the request representation. The engine core decides what is admitted. The workers run the model. Sampling and output processing return a usable result to the caller.

flowchart LR A["API and input processing"] --> B["Engine core schedule and batch queue"] B --> C["GPU model runner"] C --> D["Attention backend"] D --> E["Compiled or CUDA Graph path"] E --> F["Sampler and speculative verification"] F --> G["Structured, tool, or reasoning output processing"] G --> H["Streaming response"] H --> A

The arrows do not mean that every request takes the same physical route. They show the logical handoff. A batch may execute eagerly, through a piecewise graph, or through a full graph. A model may have no multimodal input, no speculative proposer, or no structured-output constraint. The important invariant is that the selected path must preserve the request's state and the attention backend's assumptions across each handoff.

Attention is a compatibility contract

Attention is often described as though a serving engine only needs to choose the fastest available implementation. vLLM's design makes the real choice more conditional.