How UCCL-Tran Turns Multipath into a Software Capability

UCCL-Tran's multipath design addresses the flow-collision problem by turning path choice into software state. This chapter follows one message chunk as it moves through queue pairs, ECMP routes, GPU memory, and acknowledgements, then explains why reordering and CPU scheduling are the price of that flexibility. The reader should come away able to distinguish a real path-selection gain from the superficial claim that more queue pairs automatically produce more bandwidth.

The starting point is the mismatch between a logical connection and the physical network beneath it. An application may see one collective exchange, while the fabric has several equal-cost routes and the NIC exposes a set of queue pairs. UCCL-Tran uses the latter to create choices for the former, but choices are useful only if the software can observe their condition and make the data path use them efficiently.

Path entropy is more than a larger QP count

Datacenter switches commonly use ECMP to spread traffic across equal-cost routes. A hash of flow attributes—including identifiers associated with the source and destination queue pairs—selects a route. If a connection uses one stable identity, its packets usually stay on one path. That stability is helpful for ordinary flow management, but it is exactly what allows a coordinated collective to collide with another collective or with itself on an unfavorable route.

UCCL-Tran creates more possible identities by using multiple queue pairs. For UC and RC, the design uses 256 QPs by default for each pair of NICs. For connectionless UD, it can combine 16 source QPs with 16 destination QPs, yielding as many as 256 source/destination combinations while consuming fewer QPs on each side. Several logical connections between the same pair of NICs share that QP set, which matters when multiple GPUs use one NIC. The shared set supplies path entropy without allocating a separate full collection of queue pairs to every connection.

“Up to 256 paths” needs a careful interpretation. It means the identifiers can create that many possible ECMP hash choices under a suitable fabric. It does not guarantee 256 physically separate, uncongested routes, nor does it guarantee that the switch hashes every choice differently. A topology with few routes, a biased hash, or a bottleneck after paths reconverge can limit the benefit. Multipathing exposes opportunity; it does not manufacture capacity.

The software then chooses among those opportunities using per-path state. UCCL-Tran records round-trip time for each path and uses power-of-two sampling: it examines a small sample of candidates and selects the one with the lowest observed RTT. This approach avoids scanning every path for every packet while still favoring paths that appear less congested. The transport can maintain congestion-control state per path, or maintain a global state that governs all paths; the paper reports similar collective performance for the two choices in its testbeds and uses global control by default in the evaluation.