The Runtime's Clock: Prefill, Decode, and Admission
SGLang’s runtime answers a specific systems question: once a structured request arrives, when should it consume GPU work, memory, and a place in a batch? This chapter follows a request through waiting, prefill, decode, and completion, focusing on the different pressures created by each phase. The result is a more accurate understanding of continuous batching: it is not one large tensor assembled once, but a sequence of admission decisions made under changing budgets.
A server can receive a short chat question while another request is processing a long document. A third request may already be generating an answer and should receive its next token quickly. If the server treats all three as the same kind of work, one request’s progress will interfere with another’s. SGLang’s runtime is organized around making those differences visible to the scheduler.
Prefill and decode are different jobs
The first phase, prefill, processes the input context. It reads the prompt, runs the model across that sequence, and produces the KV state that generation will need. A long prompt creates a large amount of input-side work, but it can also be processed with substantial parallelism across its tokens. The second phase, decode, extends an already active sequence. Each step typically adds one or a small number of tokens for each request while consulting the accumulated KV state.
This makes prefill primarily compute-heavy and decode primarily memory- and KV-state-heavy. Prefill wants enough work to keep the model’s arithmetic units busy. Decode wants repeated, low-delay access to the state of many active sequences. A new prefill can therefore be good for throughput and bad for the time between visible output tokens if it monopolizes the resources that a decode batch needs. Conversely, delaying every prefill to protect current decodes can leave capacity unused and make time to first token poor.
The distinction is not just a performance slogan. It determines which state must exist at each transition. Before prefill, a request has input tokens and sampling instructions but no request-specific KV history in the runtime. During prefill, the system must allocate request and KV slots, look for any reusable prefix, and compute the missing portion. During decode, the request holds a growing sequence and must retain the state needed for its next step. When it finishes or is aborted, those resources can be released, and some of the resulting prefix state may become reusable by another request.
SGLang separates the responsibilities that make these transitions possible. The tokenizer manager prepares input and maintains request-facing state, including incremental output and metadata. Scheduler processes manage the waiting queue, assemble batches, and coordinate tensor-parallel GPU workers. A detokenizer manager converts generated token IDs into output text and handles the incremental shape of streaming results. This process split means that a slow response can originate outside the model kernel: the request might be waiting for admission, for a cache operation, for a phase transition, or for output processing.
The scheduler’s job is to turn those possible transitions into a forward step. It maintains a running batch, incorporates completed prefill work when appropriate, asks whether waiting requests can be admitted, and otherwise advances active decode work. The running batch is therefore a stateful object. Its membership and mode change as requests finish, new requests enter, chunks are processed, or a request is requeued after preemption.
The following state model shows why “the request is running” is too vague to describe the runtime.
The transitions are conceptual, not a promise that every internal path uses these exact labels. Their value is causal. Admission comes before new GPU work; KV protection lasts while an active request depends on the state; completion changes what may be evicted or reused. A request can be waiting even when a GPU is doing work for someone else, and a completed request can leave behind state that affects a later request.
A batch is a budgeted decision
In offline inference, a batch can be assembled in advance. Online serving cannot assume that requests have the same prompt length, output limit, arrival time, or lifecycle. SGLang’s scheduler consequently treats admission as a constrained decision rather than a simple “add everything that is waiting” operation.
At the beginning of a scheduling turn, the runtime considers the current running batch and any prefill work that has just completed. It may merge the surviving work, remove finished requests, and seek a new prefill batch. If prefill is possible, a PrefillAdder examines waiting requests in policy order. It accounts for remaining input tokens, the configured chunk budget, the number of mixed decode tokens, available request slots, KV-cache capacity, page size, maximum running requests, and a prefill batch-size limit. The names of these controls matter less than their relationship: a candidate is admitted only if the resources needed for its next piece of work fit alongside what is already running.
This is why an apparently idle GPU does not imply that every request can start. The missing resource may be a row in a request-to-token table, a KV-cache slot, a memory headroom requirement for a future decode, or a compatibility condition imposed by a specialized path. A candidate can also be held back by a maximum prefill batch size even if raw memory remains. In a deployment with LoRA adapters, the scheduler may need to account for which adapters are already represented in the active batch. In a distributed or attention-specialized configuration, rank synchronization and topology can further constrain what can be mixed safely.
The budget is prospective. The scheduler cannot allocate only what the current input needs and ignore the future. An active request that has just completed prefill will need space for generated tokens. A long-running request may need to retain protected KV state while other requests come and go. The admission logic therefore estimates remaining work and reserves capacity rather than treating each forward step as independent.
When resources are insufficient, SGLang can mark a batch full, defer the candidate, or use priority-aware preemption where that path is enabled. A preempted request is not necessarily failed; it can return to the waiting queue while the scheduler protects progress for work that can run now. This introduces a trade-off. Preemption can improve utilization or preserve a priority policy, but it also discards or rearranges in-flight work. A policy that reduces queue time for one class of request can therefore increase work or delay for another.
The runtime also needs to decide which phase gets the next turn. SGLang’s scheduling loop attempts to run a newly admitted prefill when possible. If there is no prefill batch, it advances a nonempty running decode batch. In modes that support mixing, prefill and decode can share a forward step; in other modes, the scheduler deliberately keeps them apart because a backend or synchronization requirement would make the combination unsafe or inefficient. The source’s implementation exposes these branches explicitly, so “continuous batching” should not be read as “all request types always share one batch.”
This phase selection affects the metrics users care about. Time to first token includes the wait and prefill path before the first output becomes visible. Inter-token latency reflects the ability to keep active decode work moving after generation starts. Throughput depends on how much useful work is admitted and how efficiently it is executed. A configuration that improves raw prefill throughput can still harm an interactive workload if it creates long gaps between decode steps. A configuration that protects inter-token latency too aggressively can make long inputs wait.
That is also why the benchmark tool must match the question. SGLang documents an online serving benchmark that sends requests to a live server and captures latency and throughput under realistic scheduling. It distinguishes that from in-process throughput, which removes HTTP overhead, and from a fixed-batch kernel benchmark, which bypasses the scheduler. Those tools answer different questions. A kernel-level result cannot, by itself, tell us how a changing waiting queue behaves under mixed prefill and decode traffic.
Chunked prefill makes long input a sequence of decisions
A long prompt can be admitted as one large prefill or divided into chunks. Chunked prefill turns one expensive event into several scheduler-visible pieces. This can prevent a very long input from occupying the entire forward path at once and can let the runtime coordinate it with other work. It also makes the request’s lifecycle longer: the request may be in prefill across multiple turns before it becomes a decode participant.
The benefit is control. If an online service has active conversations generating tokens, the scheduler can limit how much new context enters in one turn. If a pipeline or overlap path is available, chunks can provide units of work that keep stages occupied. The cost is additional bookkeeping and more transitions. Every chunk must preserve the correct input position and the request’s KV state; every boundary must be compatible with page allocation and the backend’s execution shape. A chunk that is too small may underuse the hardware. A chunk that is too large recreates the interference that chunking was meant to reduce.
SGLang’s admission path keeps the chunk budget separate from the total remaining input budget. When a request is already in a chunked-prefill sequence, the scheduler calculates the next chunk size and continues it even though ordinary request admission might otherwise consider the batch full. This is a subtle but necessary rule: a partially processed request cannot be stranded simply because the normal “new request” limit is reached. The runtime must distinguish a new admission from the continuation of state it already owns.
Overlap introduces a similar distinction between logical progress and physical work. The scheduler can prepare one piece of work while another is being executed when the selected mode and backend support it. That can hide some coordination cost, but it does not remove the underlying computation or memory traffic. It can also make debugging harder because the event that triggered a delay and the operation that eventually exposes it may occur in different parts of the loop. Observability must therefore report phase and batch context, not only total request time.
Priority and preemption add a policy layer on top of these mechanics. A priority queue can change which waiting request is considered first, but priority does not create capacity. If a high-priority request needs a large prefill and no safe budget remains, the scheduler must either wait, preempt compatible work, or reject/abort according to the configured policy. Likewise, a low-priority request with a highly reusable prefix may be economically valuable even if it is not first in line. SGLang’s cache-aware behavior and scheduling policy can interact, but they are not the same objective; reuse benefit must be weighed against responsiveness and fairness.
The runtime’s design therefore makes a useful operating principle visible: admission is where performance objectives become resource commitments. Token limits protect compute budgets. Request slots protect per-request state. KV accounting protects future decode. Page alignment protects allocator and kernel assumptions. Priority determines who gets considered, while preemption determines whether currently running work can be displaced. Phase rules determine whether the next step favors first-token latency or ongoing generation.
Reading the runtime correctly
When a request is slow, the right diagnosis is not “the model is slow.” Ask which clock is slow. Did the request wait in the queue? Was its prefill delayed by a running decode batch? Did a long prompt consume its allowed chunk budget over several turns? Was the batch full because of request rows or KV memory rather than arithmetic capacity? Did a specialized backend prevent mixing? Did output processing or an inter-process boundary add delay?
SGLang’s runtime does not make those questions disappear. It gives them explicit places in the execution model. The frontend declares what work exists; the scheduler decides when that work can fit; the model workers compute it; and the KV state survives or is released according to the request lifecycle. The next step is to examine the state that makes those decisions valuable. If two requests share context, how can SGLang recognize that fact, turn it into a radix-tree match, and change the amount of work that admission must reserve?