The GPU Networking Stack UCCL-Tran Has to Work With

UCCL-Tran changes transport control, but it does not start with a blank sheet. It has to fit between ML collective libraries, host CPUs, RDMA NICs, network switches, and GPU memory. This chapter traces that existing path and compares the three RDMA queue-pair choices—RC, UC, and UD—so the reader can see what UCCL-Tran is able to take over, what it must leave to hardware, and why GPUDirect keeps the software layer from becoming a bulk-data bottleneck.

The distinction is practical. A GPU collective is not one operation performed by one device. It is a coordinated sequence in which the GPU prepares or reduces data, the CPU submits network work, the NIC moves bytes, and the receiver reports completion. The transport choice determines who owns reliability, congestion control, segmentation, and ordering along the way.

A collective is a conversation between GPU, CPU, and NIC

Consider a collective such as allreduce. Each participating GPU has a portion of a tensor and must exchange intermediate results with other participants. The collective library launches a GPU kernel that performs reduction or copying and uses transport buffers in GPU memory to receive network data. On the sending side, the host CPU issues RDMA operations—often writes—describing the source address, destination address, and length of a data chunk. The NIC turns that request into network traffic. On the receiving side, the NIC places the incoming data in the specified transport buffer, and software polls completion information while the GPU kernel eventually moves or reduces the data into the application tensor.

The important point is that the CPU participates without necessarily touching the tensor bytes. It prepares work requests and observes completions; the NIC handles the transfer; the GPU performs the computation that combines or copies the received data. This division is why it is misleading to contrast “software transport” with “hardware data movement” as if only one could exist. The software may own the decision about how to send while the NIC and GPU still own the high-volume path that carries and processes the payload.

At a lower level, the CPU submits a work-queue entry to a queue pair. Issuing that work commonly means building a description of the operation and writing it to NIC registers through a memory-mapped I/O operation. When the NIC finishes a two-sided operation, it can create a completion-queue entry for software to poll. The exact bookkeeping is not the point of the article, but it reveals where the control boundary begins: the CPU decides what work to post and the NIC decides how its queue-pair transport semantics execute that work. A receiver may therefore learn that a write arrived through a completion event even though the received bytes have gone directly to a GPU transport buffer.

The collective library then coordinates two different kinds of completion. Network completion says that a chunk has reached the destination described by the operation. GPU-kernel completion says that the chunk has been copied, reduced, or otherwise incorporated into the tensor the application uses. Those events need not happen at the same time. A transport buffer acts as the handoff point: it lets the NIC deposit data while the GPU kernel consumes it according to the collective's synchronization rules. This is why queue-pair semantics and memory placement affect application behavior even when a reader never sees the individual packets on the wire.

The arrangement also explains why the host CPU can be both important and absent from the data path. It is responsible for submitting verbs, polling or processing completion information, and coordinating transport state. It does not have to load each tensor word into CPU memory and copy it back out. GPUDirect is valuable precisely because the high-rate movement follows the PCIe-connected NIC/GPU route. The CPU remains a control-plane participant whose cost is measured in work requests, state updates, and decisions rather than in the full bandwidth of every payload.

That hidden division is what ordinary RDMA abstractions tend to conceal from an application. An RC queue pair may acknowledge, retransmit, and regulate traffic inside the NIC. The collective library sees a useful reliable operation, but not necessarily the policy choices that produced it. A UC or UD operation exposes less built-in transport behavior, so a software layer can add its own reliability or scheduling—but it must then receive enough metadata to know which chunk arrived and where it belongs. UCCL-Tran's design starts from this exact question of ownership: which repetitive movement should remain offloaded, and which state must become visible if a new policy is to be implemented?

GPUDirect matters because the data path can remain direct even when the control path becomes programmable. NVIDIA's documentation describes GPUDirect RDMA as direct data exchange between GPU memory and a peer device, such as a ConnectX SmartNIC or BlueField DPU, over PCI Express. The purpose is to avoid a bounce buffer in ordinary CPU memory. In the paper's description of a GPU server, GPU-to-NIC traffic crosses the relevant PCIe switch rather than competing for the PCIe root complex attached to the CPU. That distinction becomes important when UCCL-Tran adds CPU work: the CPU can issue verbs and process transport metadata without automatically becoming the route through which every tensor must pass.

Direct access is nevertheless a capability with prerequisites, not a property that appears merely because a GPU and NIC share a server. The versioned NVIDIA GPU Operator documentation identifies user-space CUDA APIs and kernel-mode drivers as necessary for GPUDirect RDMA. It also describes nvidia-peermem, included in standard NVIDIA driver installers beginning with CUDA 11.4 and R470, as the module that allows Mellanox InfiniBand-based HCAs to read and write GPU memory directly. The page places the GPU Operator alongside the Network Operator: together they set up drivers and device plugins that let workloads use GPUDirect RDMA. This explains an operational boundary beneath UCCL-Tran. If peer access is not correctly enabled, a software transport may still make decisions, but it cannot claim the same direct GPU/NIC data path.

GPUDirect Storage is related but not interchangeable. NVIDIA describes it as a direct path between local or remote storage and GPU memory that avoids a CPU bounce buffer. That confirms a broader design pattern—keep bulk data close to the device that consumes it—but the storage path does not supply UCCL-Tran's congestion-control, multipath, or loss-recovery policy. The transport question remains specific to the network path used by collective communication.

Queue-pair semantics decide where transport responsibility lives

RDMA does not expose just one kind of connection. It exposes queue pairs, or QPs, whose semantics determine how much transport behavior is built into the NIC. The three choices in the UCCL-Tran paper can be understood as three different placements of responsibility:

The table describes ownership, not a simple quality ranking. RC supports the broadest set of RDMA operations and gives an application a familiar reliable message abstraction, but its convenience comes from putting important decisions inside the NIC. UC retains one-to-one message semantics and the NIC's useful bulk-transfer offload while leaving more of the transport contract to the caller. UD is more like sending independent datagrams: it can address different destinations and avoid a connection's fixed behavior, but a larger message must be broken into packets and reconstructed somewhere else.

The distinction between one-sided and two-sided operations matters here. A sender can issue a one-sided write that names where the remote payload should land, while a send/receive exchange requires the receiver to have posted a matching receive buffer. The paper relies on the former for UC/RC data movement and on scatter-gather send/receive entries for the UD path. In UC/RC, an immediate-data field can accompany a write and surface as a small control value in the receiver's completion entry. In UD, where that particular write form is not available, a fixed-size control header and the GPU payload travel together in the datagram and are split at the receiver. These are different mechanisms implementing the same goal: software must learn about the payload without reading the payload through the CPU.

AWS EFA makes the portability issue visible. The paper describes EFA's SRD as a datagram-oriented reliable transport with its own multipathing, reliability, and congestion behavior, rather than assuming that every provider exposes NVIDIA-style RC and UC. UCCL-Tran can use EFA's available datagram interfaces, but it cannot assume the same segmentation, timestamping, or control-header path. A software transport therefore needs an adapter at the primitive boundary and a way to degrade gracefully when a hardware feature is missing. Portability means preserving the control/data contract across those adapters, not pretending that their internal costs are identical.

The names can obscure the design choice. “Unreliable” does not mean the application must accept corrupted or incomplete collective results. It means the queue-pair transport does not promise to recover lost packets for the application. UCCL-Tran can build reliability above UC or UD in software. Conversely, RC's reliability is convenient, but the NIC's decision about how to detect, buffer, acknowledge, and retransmit packets is largely baked into the hardware. That makes RC a good high-performance baseline and a less flexible substrate.

UC is attractive precisely because it leaves a useful middle ground. The NIC can still segment a larger data chunk into MTU-sized packets and reassemble those packets at the destination, while UCCL-Tran supplies the control logic that UC omits. The CPU can therefore reason about a chunk, path, acknowledgement, or loss event without manually issuing a separate operation for every wire packet. This is the shape of hardware/software cooperation the system wants: hardware performs repetitive movement; software owns policy.

When UC is unavailable, UCCL-Tran can use RC with hardware congestion control disabled where the NIC permits that configuration. RC still offers useful hardware behavior, including faster acknowledgements and precise RTT estimates, but software cannot freely rewrite all of its reliability machinery. AWS EFA is a more restrictive case in the paper. Its NICs are described as using SRD rather than ordinary RC, and the relevant hardware does not expose the same UC-style escape hatch. UCCL-Tran therefore uses UD/SRD techniques to bypass more of the fixed transport logic.

UD changes the engineering balance. Because a UD operation carries only one MTU-sized datagram and does not offload segmentation or reassembly, UCCL-Tran must split large messages on the sender and reconstruct them on the receiver. The paper uses scatter-gather entries to combine a fixed control header from CPU memory with a GPU payload for transmission, then split them into separate CPU and GPU destinations on reception. That preserves a crucial invariant: the control header and its payload share the same fate in the network. But it also means packets can land out of order in GPU memory, requiring extra reassembly and a scattered-copy operation before the collective's normal reduction work.

This is why queue-pair semantics are more than an implementation detail. They determine which state is visible to software, how much work the NIC performs automatically, and where correctness must be restored when packets are lost or reordered. UCCL-Tran's architecture is therefore not “CPU replaces NIC.” It is a selection rule: use UC when it provides the right combination of offload and openness, use configurable RC when that is the available compromise, and use UD when avoiding fixed control logic is more valuable than the extra CPU and GPU work.

That rule also clarifies the meaning of transport control. Congestion control decides how quickly or under what credit a sender may transmit. Path selection decides where a chunk or packet should go. Reliability decides what counts as missing and what should be retransmitted. Segmentation, reassembly, and direct memory access move the bytes efficiently. These responsibilities interact, but they do not have to be owned by the same device. The central opportunity UCCL-Tran exploits is to move the first three into software while retaining as much of the last group as the available NIC can provide.

The resulting model gives us a way to diagnose the failures that motivate UCCL-Tran. If the NIC is responsible for a policy, ask whether that policy can see the collective's traffic shape, whether it can use all the paths in the fabric, and whether its recovery behavior fits GPU-sized transfers. If the data path is direct but the control signals are hidden, ask whether software receives enough information to make a good decision. The next problem is not the existence of RC, UC, or UD; it is what happens when their fixed behaviors meet low-entropy all-to-all traffic, receiver hot spots, packet loss, and mixed generations of hardware.

References

• UCCL-Tran paper
• GPUDirect RDMA and GPUDirect Storage — GPU Operator 22.9.1 documentation