The First Half in One View: Following State Through SGLang

SGLang becomes easier to judge when a request is followed as one continuous state transition rather than as a collection of frontend, scheduler, and cache features. This checkpoint connects those pieces around one question: where is work created, where is it stored, and what changes when a later request can reuse it? The practical result is a way to distinguish a genuinely useful cache path from a configuration that merely adds capacity, movement, or queueing.

Consider an application that answers questions over a long, stable document. The application may use SGLang’s frontend to establish a fixed instruction and document context, then issue a generation operation for each user question. The same scenario can also arrive through an OpenAI-compatible request or the native Engine. The interface changes how much structure the caller exposes, but the runtime still has to turn the input into tokens, admit it into a batch, compute or recover KV state, generate output, and decide what state remains afterward.

One request crosses four boundaries

The first boundary is between application intent and request state. A structured program adds roles and fixed text, names a generation operation, and may carry earlier values into later control flow. A compatible API supplies messages and sampling parameters instead. In both cases, SGLang’s tokenizer-facing path turns the input into a request the runtime can schedule. At this point the request has text or token IDs, output rules, and any identity information that affects whether cached state may be shared.

The second boundary is admission. The scheduler receives a request that wants resources, not just an answer. It must determine whether there is a request slot, enough KV capacity, and enough token budget to process the next portion of the input while protecting the future decode step. It also considers the work already in the running batch. A long document may be compute-heavy to prefill, while shorter conversations already in decode need regular progress. The request can wait even while other model work is active, because “the GPU is busy” and “the request has a safe place in the next batch” are different facts.

Before new prefill work is performed, SGLang can ask the local radix cache how much of the token sequence is already present. Imagine that the document and instruction occupy the first several pages, while the user’s question is new. The cache walks its tree and returns the longest compatible prefix. The scheduler can then account for the matched state and reserve capacity for the unmatched suffix. The request still needs computation for its new question and later decoding, but it does not need to recreate the KV state for the shared pages.

The match is only valid if the identities align. The same token IDs may be separated by an adapter namespace, cache salt, version, or retrieval context. A textually identical prefix with different semantic conditioning is not automatically reusable. This is why cache identity is part of the request state: a fast match that crosses an invalid boundary is not an optimization.

The third boundary is model execution. SGLang runs prefill for the missing input and then moves the request into decode. During this transition, the request’s KV state is protected because active generation depends on it. The scheduler may keep the request in a running batch with other decodes, or it may process input in chunks when a long prompt should not enter as one unbounded operation.