When a request is more than text

In vLLM, a multimodal or tool-using request does not simply add a larger prompt to the same path. Images, video, audio, chat templates, encoder outputs, cache identities, schemas, and reasoning fields introduce new state before and after language-model execution. Following that path shows how vLLM extends its token-state model—and where a deployment must add trust, resource, and compatibility controls.

Media creates a second front end

A text request can usually begin with tokenization. A multimodal request may begin with fetching an image, decoding video frames, loading audio, or accepting a precomputed embedding. That work belongs near the API boundary because it involves request formats, remote URLs, local files, media limits, and model-specific preprocessing before the language model can consume anything.

The user-facing message may contain text interleaved with an image or video item. vLLM's frontend renders the chat template and represents the media in the model's expected form. The language prompt may contain placeholder positions, but those placeholders are not the media itself. An image processor or another encoder turns the item into features that are inserted at the corresponding positions before language-model prefill.

This creates at least two different forms of state. The encoder state represents work done to interpret the media. The language-model KV state represents the attention history after those features and text have entered the decoder. They can be cached and transferred separately. A request may reuse a processed image or encoder output while still needing a new language prompt and a new answer.

sequenceDiagram participant C as Client participant A as API server participant M as Media and processor cache participant E as Encoder budget and cache participant V as Media encoder participant S as Engine core and KV scheduler participant L as Language model participant P as Output parser C->>A: Chat request with text and media A->>M: Fetch, decode, normalize, and identify media M-->>A: Processed item or cache result A->>E: Check encoder state and reserve encoder work alt Encoder cache miss E->>V: Encode media features V-->>E: Cached encoder output end A->>S: Rendered tokens plus feature references S->>L: Schedule prefill and decode with KV state L-->>S: Sampled tokens S->>P: Parse or constrain output P-->>A: Stream-ready response deltas A-->>C: Stream or return response

The diagram separates the caches without pretending they are independent systems. The API server owns input processing and streaming, the encoder path consumes a separate compute budget, and the engine core still decides when language-model work can enter the GPU. A cache hit at the media or encoder layer changes the work before language prefill; it does not make the decoder's generated tokens free.

Identity must follow content, not placeholders

Prefix caching already requires token and context identity. Multimodal inputs make that requirement more demanding. Two prompts can contain the same placeholder token sequence while pointing to different images. If vLLM used only the placeholders in the prefix key, it could reuse the attention state for the wrong media.

The cache therefore includes a media identity alongside the token-derived block identity. By default, vLLM can hash media content. A caller can also provide a stable UUID for an item so repeated requests can reuse processed data without hashing the raw bytes every time. That shortcut moves responsibility to the caller: two different media items must not be given the same identifier on a shared server.