How to Think About SGLang as a Serving System
The central question for SGLang is not which feature produces the largest number in a benchmark. It is how a serving system can expose enough request and runtime state to reuse work, schedule competing phases, constrain output, distribute computation, and change weights without confusing old state for new state. This conclusion brings those relationships together and turns them into a practical judgment model: diagnose the bottleneck, identify the state that crosses it, choose the smallest mechanism that addresses it, and validate the complete path.
Imagine a service that receives long document questions, must return structured results, and runs on a fleet whose model weights are periodically updated by an RL system. The same deployment may need prefix reuse, grammar constraints, online batching, phase separation, quantized kernels, routing, and memory handoffs. A feature catalog makes that architecture look like a pile of unrelated optimizations. SGLang becomes more intelligible when the service is treated as a chain of state transitions with explicit owners and boundaries.
The common architecture is state with a lifecycle
At the application boundary, SGLang can accept a familiar OpenAI-compatible request, a native Engine call, or a frontend program. The frontend offers the richest expression of intent: roles and templates shape the conversation, generation operations create named values, Python control flow consumes those values, and forks describe branches that begin from shared context. The compatible API does not expose all of that structure, but it still produces a request with tokens, sampling rules, and a model-specific conversation format.
That first state transition matters because reuse begins before the GPU. A repeated system instruction or document prefix is not just repeated text; after prefill it can become reusable KV state. A branch in a prompt program is not automatically shared computation, but it identifies a relationship the runtime can preserve or schedule. A grammar request is not merely a post-processing preference; it creates a state machine that controls which tokens are admissible. Programmability is therefore useful when it carries information that would otherwise be hidden in a finished string.
The runtime then turns that intent into an admission problem. Tokenization and request management prepare the input. The scheduler decides whether the next prefill or decode work fits the current budgets. Model workers compute the missing context or advance existing sequences. Detokenization and streaming return the result while the system continues to manage model-side state. The process boundaries can vary between the Python and Rust paths, but the ownership question remains stable: which component owns request state, which owns the batch, and which owns the KV locations that an active request needs?
Prefill and decode make that ownership dynamic. Prefill creates or extends the state for an input context. Decode repeatedly reads and extends it. The scheduler must protect a request’s live state, reserve enough capacity for future tokens, and decide whether a new prefill should share a step with active decodes. A running batch is consequently not a static list. It is a changing agreement among request arrivals, token budgets, available KV and request slots, phase rules, priority policy, and backend compatibility.
The local radix cache turns one part of that agreement into a data structure. It matches token spans, returns KV indices for the longest compatible prefix, creates only unmatched suffixes, and splits nodes when a match ends inside a stored span. Lock references keep state needed by active requests out of eviction. Hit and access metadata help decide what can be removed when capacity is scarce. Namespaces and cache salts prevent equal token sequences from sharing state when adapter, version, sampling, or retrieval context makes the reuse invalid.
HiCache extends the same relationship across memory tiers. GPU memory is the fast local tier. Host memory provides a larger but instance-private tier. An optional storage backend can provide a wider sharing scope when configured for it. The distinction is fundamental: a larger host pool does not make several processes share one cache automatically, and a storage hit is not the same latency path as a local GPU hit. Matching, prefetch, transfer, synchronization, write-back, and eviction all become part of the request’s lifecycle.