From One GPU to a Cluster: Parallelism, Disaggregation, and Routing

Scaling SGLang is not a matter of turning on more GPUs and expecting the same runtime to move faster. The central question is what must be partitioned, what must communicate, and who owns the KV state after each boundary is crossed. This chapter compares tensor, data, expert, and pipeline parallelism, then follows the move from a unified instance to separate prefill/decode workers and a fleet-level gateway. The goal is to choose a topology from a bottleneck rather than from a list of available switches.

A useful starting point is the request already traced through SGLang. It has input tokens, a growing KV state, a decode schedule, and possibly draft or constrained-generation state. Scaling changes the location and ownership of those objects. One design splits the model computation inside a step; another splits layers across stages; another sends different requests to replicated workers. Prefill-decode disaggregation goes further by assigning different phases to different worker pools and transferring the state that connects them.

Parallelism chooses what crosses the boundary

Tensor parallelism divides model tensors or layer computation across devices that participate in the same request. The benefit is that a model or a layer whose memory footprint exceeds one device can be executed collectively. The cost is frequent communication inside the forward pass: partial results, activations, or reductions must be exchanged before the next computation can proceed. Tensor parallelism is therefore especially sensitive to the interconnect and to the shape of the operation. Adding devices can make the model fit while reducing the fraction of time spent in useful arithmetic.

Data parallelism keeps multiple execution groups available for different requests or attention work. Instead of splitting every operation for one request across all devices, it provides more independent capacity for traffic. That can improve throughput and isolate request groups, but it introduces a placement question: which group receives a request whose prefix may already be cached? In SGLang, data-parallel attention can also alter how attention ranks are arranged relative to the model’s tensor-parallel layout. The scheduler and cache must agree on those rank relationships before a request can run.

Expert parallelism addresses a different kind of model. In a mixture-of-experts model, a routing decision selects a small set of experts for each token. SGLang can distribute expert weights across devices, then dispatch tokens to the devices that own the selected experts. The forward path becomes a coordinated sequence: choose experts, dispatch tokens through an all-to-all or alternative communication layer, run grouped matrix multiplications, and combine the results. The memory benefit comes from not replicating every expert everywhere. The communication cost comes from moving token activations to the right owners and bringing their results back.

This makes expert parallelism a coordination problem, not merely a sharding option. A prefill batch may contain enough tokens to make high-throughput dispatch worthwhile. Decode may contain fewer tokens and require a low-latency path that can work with CUDA graphs. SGLang’s documentation distinguishes normal mode for throughput-oriented prefill from low-latency mode for decode for some communication backends, and recommends automatic mode selection where appropriate. The same expert layout can therefore need different communication behavior across the request lifecycle.

The framework separates the stages around the expert computation. A dispatcher handles token movement; permutation methods adapt layouts; a runner performs the grouped expert computation; and a combine step returns results to the original token order. That separation is significant because communication and computation evolve at different speeds.