Scaling the token-state engine across models and machines

For vLLM, scaling is not one decision called “use more GPUs.” The right choice depends on what must move: model weights, KV state, expert tokens, prompt work, decode work, or adapter-specific behavior. This chapter compares vLLM's model interfaces, memory extensions, parallelism strategies, and disaggregated paths by the bottleneck each addresses, so a reader can choose a plausible design without confusing capacity, throughput, and latency isolation.

Breadth comes from contracts

vLLM supports native model implementations as well as a Transformers modeling backend. The important point is not the size of the supported-model table; it is the contract that lets different architectures enter the same execution system. A compatible attention module must call the expected attention interface and expose a unique layer identity so vLLM can associate it with KV state. A mixture-of-experts model must expose its experts and the information needed to route tokens. Nonstandard layers may need explicit tensor- or pipeline-parallel plans.

These rules keep model-specific structure at the edge of the engine's common assumptions. The model can have its own layers and configuration, but the serving system still needs to know where attention state belongs, how expert computation is presented, and how weights are divided when the model spans devices. In other words, model breadth is not achieved by pretending architectures are identical. It is achieved by defining the few interfaces that make their differences manageable.

Plugins extend the same idea beyond the built-in tree. A plugin can register a model, platform, input/output processor, logger, endpoint, or other capability through Python entry points. Because vLLM may create multiple processes, code needed by a feature has to be available in the processes that use it. That is useful for hardware and model evolution, but it also enlarges the trusted code base and leaves version compatibility to the plugin author.

The support label needs calibration. The repository distinguishes strict output consistency with another framework, sensible output, successful loading and execution, and support maintained through community feedback. A model can therefore be usable in vLLM without carrying a guarantee of identical numerical behavior or complete feature coverage. That distinction matters when a deployment depends on a particular combination of multimodality, parallelism, quantization, or adapters.

Memory choices change the room for token state

Model weights and request state compete for accelerator memory. Quantization reduces the precision and footprint of weights, which can make a larger model fit or leave more room for the KV cache and concurrent requests. But a quantized checkpoint is not a universal compression switch. Each format has hardware, activation-dtype, kernel, and model-layer conditions. A format that fits in memory may still lack the fast implementation or feature compatibility needed by the target deployment.

The decision should begin with the constraint. If the uncompressed model cannot fit, quantization may be the least disruptive way to create capacity. If the model already fits but concurrency is low because KV state consumes the remaining memory, reducing weight footprint may increase useful headroom. If quality or numerical consistency is the priority, the memory gain must be tested against the model's outputs and the chosen backend rather than inferred from the format name.

vLLM also supports per-request LoRA adapters on a shared base model. This is a different kind of memory trade. Instead of replicating a whole model for every specialization, the server can apply an adapter to selected requests and process base-model and adapter traffic together when configured.