When Sparse Routing Meets the Network

DeepSeek-V3 makes only part of its expert capacity active for each token, but that choice creates a physical problem: the selected experts may live on different GPUs and different machines. This chapter examines how the model's training runtime and serving layout handle that traffic. The key judgment is not whether DeepSeek-V3 uses parallel hardware—it plainly does—but whether scheduling, topology-aware communication, and deployment-specific placement keep the arithmetic saved by sparse routing from becoming network waiting time.

The network becomes part of the forward pass

DeepSeek-V3 was trained on 2,048 NVIDIA H800 GPUs. Within a node, the GPUs communicate through NVLink and NVSwitch; across nodes, they use InfiniBand. This creates two communication environments with different characteristics. A token that stays within one node can use a fast local path. A token sent to an expert on another node must use a wider system whose bandwidth and latency are different. The model's four-node routing limit, introduced as part of its expert design, is therefore also a way to bound the physical journey of a token.

The training framework divides responsibility across several kinds of parallelism. Pipeline parallelism places different layers of the Transformer on different stages. Expert parallelism distributes the routed experts across GPUs and nodes. Data parallelism keeps multiple copies of the training work moving over different batches. The report describes 16-way pipeline parallelism, 64-way expert parallelism across eight nodes, and ZeRO-1 data parallelism. It says that memory optimizations make it possible to train without tensor parallelism, which would otherwise split individual layer operations across more devices.

This arrangement exposes a characteristic MoE tension. The arithmetic inside an expert can be performed where that expert resides, but the token representation has to get there first. After the expert finishes, the result has to return to the computation that continues the Transformer layer. In the described run, the cross-node computation-to-communication ratio is approximately one to one. If computation and communication occur one after the other, a large fraction of the cluster can spend its time waiting for the other half of the operation.

The communication is not a single message attached to the end of training. It appears repeatedly inside the forward and backward passes. A forward chunk includes attention, dispatching tokens to experts, the expert MLP work, and combining the expert outputs. Backward work has its own communication and can be divided into the gradient for the input and the gradient for the weights. Pipeline communication between stages adds another stream. The runtime must coordinate these activities without letting one interface become a barrier for all the others.

That is the problem DualPipe addresses. Rather than treating a pipeline stage as a simple sequence of “compute, then communicate,” it rearranges parts of forward and backward chunks so that a pair can execute concurrently. The scheduling is bidirectional: micro-batches enter from both ends of the pipeline. This reduces the idle gaps, or pipeline bubbles, that appear when a stage has no useful work while earlier or later stages catch up.