SGLang and the New Unit of Inference
SGLang is a serving framework for large language and multimodal models, but that description is only useful if we ask what “serving” includes. This opening focuses on the boundary around an SGLang request: the application’s structured intent, the runtime decisions that turn it into GPU work, and the reusable state left behind. Understanding that boundary gives us a practical way to judge SGLang’s design, rather than treating its many performance features as unrelated switches.
A model can be considered fast in isolation and still deliver a poor service. A request may wait for tokenization, compete with other requests for a batch, rebuild a long shared prefix, move key-value state between memory tiers, or spend time transferring data between workers. The user experiences the sum. SGLang’s significance lies in making many of those hidden costs explicit enough to manage.
A serving system, not only a model runner
At its most basic, an LLM server accepts a prompt and returns generated tokens. The model processes the input sequence, then generates one token at a time, using a key-value, or KV, cache to avoid recomputing the entire history at every step. That description is enough for a single request. It breaks down under real traffic, where prompts arrive at different times, have different lengths, share different amounts of context, and require different output behavior.
SGLang presents itself as a framework for this larger problem. Its repository describes a fast runtime with prefix caching through RadixAttention, continuous batching, structured outputs, speculative decoding, several forms of parallelism, quantization, and prefill-decode disaggregation. It also supports multimodal models, familiar OpenAI-compatible APIs, a native generation interface, a programmable frontend, and integrations for reinforcement learning and post-training. These are not all the same kind of feature. Some change how a token is computed, some change which tokens may be produced, some change where state is stored, and some change which worker owns a phase of the request.
That distinction matters. The project’s README also reports very broad adoption and production volume. Those statements are useful signals about the project’s ambition, but they are not a single independent benchmark, and they do not establish that every mechanism helps every workload. A more durable claim is narrower: SGLang is designed to coordinate request structure, execution state, and placement so that serving work can be reused or scheduled deliberately.
This is the central idea for this series: reuse-aware, programmable execution. “Programmable” means that an application can express more than an opaque prompt string. “Reuse-aware” means that the runtime can recognize and preserve useful state—especially shared context—rather than treating each request as unrelated. The two ideas reinforce one another. A frontend that knows where branches or common prefixes occur can expose opportunities; a runtime that owns cache and scheduling state can decide whether those opportunities are worth exploiting.
Where a request actually goes
Consider a client sending a chat completion. At the outer boundary, SGLang can look familiar: the quickstart documents an OpenAI-compatible endpoint, a native /generate endpoint, streaming, and an in-process Engine. The choice of interface affects how much control the caller has, but the logical path is similar. Input becomes a request record; text or multimodal content is prepared and tokenized; a scheduler admits the request; model workers perform prefill and decode; and generated token IDs are detokenized and returned.
The process layout makes the boundary concrete. In the Python Engine path, SGLang documents three principal responsibilities. A tokenizer manager receives and prepares requests. Scheduler subprocesses manage batches and tensor-parallel GPU workers. A detokenizer manager turns generated tokens into the output representation and maintains streaming-related state. These components communicate through inter-process channels. A separate Rust-server path can own API serving, tokenization, and detokenization inside an embedded server, so the exact process graph depends on how SGLang is launched. The important point is not the number of processes. It is that the model kernel is one participant in a larger request lifecycle.
The lifecycle also has two different computational shapes. Prefill processes the input context and creates the KV state needed for generation. It is dominated by work over the prompt. Decode advances active requests, usually one or a small number of new tokens at a time, while repeatedly reading the accumulated KV state. Prefill and decode therefore put different pressure on computation, memory, batching, and latency. A scheduler that simply waits for a fixed batch misses the reason an online service needs to decide continuously which requests can enter, remain, or leave a step.
That is why SGLang’s runtime state includes more than a queue. It tracks active requests, available request slots, token and KV-cache capacity, phase, sampling behavior, and the information needed to stream results. A request can be delayed because the next prefill would consume too many tokens, because there is no request row or KV slot, because a specialized backend cannot mix it with the current batch, or because another phase should run first. The service’s behavior emerges from these decisions.
The diagram is intentionally logical rather than tied to one deployment. In a simple instance, placement may mean choosing a data-parallel worker. In a large deployment, it may involve separate prefill and decode pools and a gateway. Likewise, “KV-cache state” can remain in GPU memory, move to host memory, or be written to a storage backend. Those later distinctions are important, but they all answer the same question: what work has already been done, who owns its result, and can the next request use it safely?
The unit of inference is a request plus state
The usual mental model says that inference begins with a prompt and ends with an answer. SGLang’s architecture suggests a more useful unit: a request together with the execution state it creates and the constraints that govern its next step.
The frontend makes this visible from the application side. Its language can add system, user, and assistant roles, issue named generation operations, use earlier outputs in later control flow, fork parallel branches, stream results, and attach constraints such as a regular expression or JSON schema. Internally, those operations are represented as expressions executed by a stream executor. A branch can carry forward the text and variables already produced; when appropriate, branches can be joined and their generated material appended with KV-cache handling rather than being reduced to unrelated strings.
This does not mean every branch is free or that every backend shares all computation automatically. Parallel requests still consume model capacity, and the actual benefit depends on how the backend handles them. The architectural value is that the relationship is expressible. A program can say, in effect, “generate two candidates from this same context, then use both,” rather than forcing an application to hide that dependency inside manually assembled prompts.
The same principle appears without the frontend language. The native Engine accepts text or token IDs, multimodal inputs, sampling parameters, sessions, priorities, streaming controls, and cache-related fields. A RuntimeEndpoint can adapt a frontend program to a running server, discover the model’s chat-template behavior, and commit a common prefix through a zero-token generation request. The OpenAI-compatible route remains a low-friction adoption path; the native and frontend routes expose more of the structure that SGLang can exploit.
This perspective also explains why cache reuse is not a decorative optimization. If two requests share a system instruction, a tool description, or a long document prefix, the KV state for that prefix may be reusable. If the requests differ in tokenizer behavior, adapter namespace, cache salt, or other context that affects meaning, reusing the same bytes may be wrong. Reuse therefore requires identity, ownership, protection, and eviction rules. The runtime must know not only that two strings look similar, but whether they represent compatible state.
What this lens allows—and what it does not
Once SGLang is viewed as a system that exposes and manages state, its broad feature surface becomes easier to organize. Scheduling and continuous batching manage competition among requests. RadixAttention makes shared prefixes addressable in local memory. HiCache extends that idea into host memory and optional distributed storage. Structured generation changes the admissible next-token set. Speculative decoding proposes several future tokens and asks the target model to verify them. Parallelism, phase disaggregation, and the Model Gateway move computation or state across boundaries. RL integrations add a changing owner: training must release, refit, version, or invalidate what serving has cached.
These mechanisms are related, but they are not interchangeable. A larger cache cannot compensate for a workload with little prefix overlap. Speculation can add draft-model memory and lose its advantage when acceptance is low. Disaggregation can reduce interference between prefill and decode while adding network transfer and failure surfaces. A quantization format may be available on one hardware/backend combination and unsupported or differently optimized on another. The right question is therefore not “Which SGLang feature is fastest?” It is “Which bottleneck is limiting this workload, which state crosses the relevant boundary, and what new cost or correctness condition does the remedy introduce?”
That question keeps SGLang at the center while giving the reader a way to reason about it. The next step is to examine the frontend more closely: how a prompt becomes an execution plan, and how that plan can expose dependencies, branches, constraints, and common work without making the application operate the serving machinery itself.