Back
GitHub

SGLang LLM Serving: State, Reuse, and Scale

How request state turns serving features into conditional trade-offs

9 chapters87 min total
GitHub
Tech
Keywords
SGLangInferenceServingKV-cacheRadixAttentionHiCacheBatchingGrammarSpeculationParallelism
Overview

This series explains SGLang as a reuse-aware, programmable serving system: structured requests become scheduled prefill and decode work; radix and hierarchical caches preserve compatible KV state; grammars and speculation change decoding; parallelism, disaggregation, routing, quantization, and RL handoffs move or invalidate state. It shows how to choose mechanisms by workload bottleneck and validate end-to-end performance without treating feature availability or benchmark results as universal guarantees.

Learning outcomes
  • SGLang treats inference as a request plus execution state, not an isolated prompt-to-token call.
  • Prefix reuse helps when long, compatible tokenized prefixes recur and saved prefill outweighs lookup and movement costs.
  • Grammar constraints enforce formal output structure, while speculative decoding helps only when proposals are accepted and supported.
  • Parallelism, disaggregation, routing, and RL refit trade capacity or isolation for communication, synchronization, ownership, and invalidation costs.
Keywords
SGLangInferenceServingKV-cacheRadixAttentionHiCacheBatchingGrammarSpeculationParallelism

Chapters

1
SGLang Inference: Requests Plus State
How structure, scheduling, and KV reuse shape service cost
8 min · How structure, scheduling, and KV reuse shape service cost
8 min
2
SGLang Frontend: From Prompt to Program
Roles, branches, streams, and prefixes become runtime work
9 min · Roles, branches, streams, and prefixes become runtime work
9 min
3
SGLang Runtime: Prefill, Decode, Admission
Why continuous batching is a changing budget decision
9 min · Why continuous batching is a changing budget decision
9 min
SGLang RadixAttention and HiCache
When prefix reuse beats recomputation—and when movement wins
10 min · When prefix reuse beats recomputation—and when movement wins
10 min
SGLang State Flow: Request to Reuse
Compare hot-prefix and diverse-prompt service paths
9 min · Compare hot-prefix and diverse-prompt service paths
9 min
SGLang Structured Generation and Speculation
When legal tokens or draft tokens change decoding
11 min · When legal tokens or draft tokens change decoding
11 min
SGLang Scaling: Parallelism, Disaggregation, Routing
Match communication and ownership to the bottleneck
11 min · Match communication and ownership to the bottleneck
11 min
SGLang RL Serving: Sleep, Wake, Refit
Keep process state while ownership and weights change
10 min · Keep process state while ownership and weights change
10 min
SGLang Serving: A Bottleneck-First Model
Choose the smallest state-aware intervention and measure the path
10 min · Choose the smallest state-aware intervention and measure the path
10 min

Preview every chapter · Unlock full text, audio, and video from chapter 4

Plans
Ch.1-3 FreeCh.4-9 Premium

FAQ

What is SGLang designed to optimize?

SGLang is designed to coordinate structured requests, runtime scheduling, reusable state, and distributed placement for model serving. Its mechanisms target different bottlenecks rather than guaranteeing one universal speedup.

When does SGLang prefix caching help?

SGLang prefix caching helps when requests repeatedly share long, compatible tokenized prefixes and the saved prefill outweighs lookup, transfer, synchronization, and storage costs. Cache scope and identity determine whether reuse is valid.

How do structured generation and speculative decoding differ in SGLang?

Structured generation restricts legal next tokens, while speculative decoding proposes future tokens for the target model to verify. The first targets formal output structure; the second targets decode advancement and depends on acceptance and compatibility.

How should SGLang performance be evaluated?

SGLang performance should be evaluated with an online serving benchmark that matches the workload, hardware, topology, cache state, and claimed latency or throughput metric. In-process and kernel benchmarks isolate layers but do not replace live-service measurement.