Why GPU Memory Allocation Stalls on the Host
MoonBright targets a specific weakness in the ordinary GPU virtual-memory path: the host may control so many translation steps that a flexible allocation becomes a serialized operation. This chapter traces those steps in CUDA and HIP, then separates physical memory work from page-table construction and coherence. The result is a practical diagnosis: the reader can tell which part of an allocation is slow, which part is constrained by granularity, and why a cache can hide the problem without removing it.
An allocation request is a chain, not a single event
A high-level call such as cudaMalloc looks like one action because the application wants one result: a pointer it can pass to a kernel. Internally, that pointer has to acquire several kinds of state. The system must choose a range in the GPU's virtual address space, obtain physical memory, connect the two in page tables, grant access, and coordinate the device's cached translations. Each step answers a different question and can have a different owner.
CUDA's low-level Virtual Memory Management interface makes that decomposition explicit. cuMemAddressReserve reserves a virtual address range. It does not provide physical bytes; it establishes where a future mapping may live. cuMemCreate creates a memory handle for a physical allocation, but the handle initially has no device or host mapping. cuMemMap binds the handle to an address range that was reserved earlier. cuMemSetAccess then grants the requested locations permission to use the mapped range. The reverse path is also divided: cuMemUnmap removes the backing from the range, cuMemRelease releases the allocation handle when its references and mappings permit it, and cuMemAddressFree returns the reservation.
That separation is useful because it lets software choose a virtual layout independently of the physical layout. It also exposes the work that a simple allocator normally hides. A mapping is not usable merely because an address has been reserved or because physical memory exists. The mapping must fit the allocation's required granularity, and the access state must be set for the relevant device or location. CUDA documents that mapping does not itself make an address accessible and that the size, address, and offset must meet the applicable granularity rules.
HIP presents the same conceptual sequence on AMD hardware. An application first checks whether the target device supports virtual-memory management, obtains a physical allocation with hipMemCreate, reserves an address range with hipMemAddressReserve, maps the allocation with hipMemMap, and sets permissions with hipMemSetAccess. HIP emphasizes that the required granularity depends on the device and allocation properties. If a request is not aligned to that granularity, physical creation can report an out-of-memory error even when the device has enough bytes in total. “Virtual” therefore does not mean “arbitrary size”: it separates address placement from backing, but hardware still sets the units at which backing can be created and mapped.
The distinction matters when a buffer grows. HIP documents a pattern in which a program reserves a further range adjacent to an existing one and maps new backing there, preserving a continuous virtual address space while changing the underlying physical allocation. The example also warns that a requested address is not guaranteed to be available and should be checked. A contiguous-looking pointer is thus a property of the virtual layout, not proof that its physical pages are contiguous.
The sequence also has different synchronization implications. Address bookkeeping can be a host data-structure operation. Physical allocation can call a device-specific resource manager. Mapping and access control change translation state. Unmapping can invalidate a range that a running kernel may still reference. Treating all of these as “allocation overhead” hides which transition is responsible for a stall and which one is required for correctness.
The sequence diagram shows why a pointer-returning call can be a scheduling event. The application waits for a chain that crosses logical state, physical resources, translation structures, and the TLB. The details differ between vendor stacks, but the conceptual boundaries are documented in CUDA and HIP, while MoonBright's diagnosis concerns how the current implementation crosses them through the host.
Consider a request for a large region at the minimum page granularity. The allocator does not need one enormous physical object simply because the program wants one continuous pointer. It can reserve a virtual interval, obtain many physical pages, and connect each page-sized portion to the corresponding virtual portion. That is the opportunity created by VMM, but it is also the source of the scaling problem: the mapping has many translation relationships even though the application sees one range. If the host handles each relationship through a sequential path, the visible simplicity of the pointer hides work that grows with the range.
The same separation explains why cleanup cannot be reduced to “return the pointer.” A range may still be referenced by a running kernel, and a cached translation may outlive the instruction that first used it. Unmapping changes the page-table record, but releasing the physical handle too early could allow later work to use memory that has already been reassigned. The vendor documentation's separate unmap, release, and address-free operations reflect these distinct lifetimes. MoonBright's diagnosis does not depend on every implementation taking exactly the same route; it depends on recognizing that translation state has a lifecycle of its own.
The long pole is translation state
MoonBright's diagnosis begins with an empirical breakdown rather than with the assumption that every part of the chain costs the same. On an NVIDIA A100, the paper separates logical bookkeeping, physical allocation, page-table construction and transmission, and TLB flushing for cudaMalloc. At 2 MB, the reported total is about 59 microseconds, of which page-table construction accounts for 47. At 16 MB, the corresponding figures are about 437 and 422 microseconds. At 128 MB, they are about 966 and 949 microseconds. Across those tested sizes, the paper summarizes the page-table component as 80% to 99% of the total.
The pattern changes the diagnosis. If physical allocation were the main cause, optimizing the address-to-page translation would leave most of the delay intact. The measured breakdown instead points to the work required to represent a mapping: validate the range, build missing translation levels, encode leaf entries, transfer the resulting state, and preserve a coherent view for the device. The paper describes thousands of independent leaf-entry encodings as inherently data-parallel, yet the host-driven model processes them through a sequential driver path.
The interconnect is part of the cost, but it is not the entire explanation. Sending translation information between host and GPU adds transfer time, and the driver must coordinate shared address-space state. But even a faster link would not automatically change the fact that the host is being asked to perform a large number of small, ordered updates. The paper therefore treats the problem as a compound control-plane penalty: sequential PTE construction, state propagation, and conservative coherence enforcement reinforce one another.
TLB behavior adds a second kind of delay. The page table is the underlying record of a virtual-to-physical relationship; the TLB is a cache of translations used by the GPU during execution. When an existing virtual address is unmapped, remapped, or assigned different permissions, an old cached answer may no longer be valid. A conservative implementation can respond with a device-wide TLB shootdown. The isolated hardware flush component reported in the paper is much smaller than the largest page-table construction component, but a flush can also require synchronization with active work. The important cost is therefore not just the time spent invalidating cache entries. It is the boundary that may have to stop or drain execution before the update is safe.
This is where stream-ordered execution exposes the design mismatch. A stream is a sequence in which GPU operations can be ordered without making unrelated work wait. A host-side mapping operation that forces a global synchronization can turn a local memory decision into a device-wide pause. A kernel in one stream may be ready to run, yet a remapping request in another stream cannot safely complete until earlier accesses have drained. The result is a loss of overlap precisely in workloads that depend on asynchronous allocation and dynamic data structures.
The diagnosis has to distinguish this root cause from contributing conditions. Larger mappings create more page entries, so their cost grows with the amount of translation state. Fine-grained or frequently changing workloads expose the path more often. Physical allocation policy can add its own latency, and alignment can make a request larger than the application originally asked for. None of those facts makes them the dominant cause in every environment. The A100 breakdown supports a narrower conclusion: in the tested cudaMalloc path, page-table construction and propagation are the component that grows into the long pole.
A useful way to verify that diagnosis is to ask what happens as the mapping range changes while the logical request stays simple. If bookkeeping were dominant, latency would be comparatively insensitive to the number of pages. If physical allocation were dominant, page-table work would be a small residual. The reported breakdown instead shows the page-table component rising from 47 microseconds at 2 MB to 949 microseconds at 128 MB, while the other listed components remain much smaller. That does not prove that page-table construction is the only source of delay, but it rules out a story in which the allocator is primarily slow because it cannot find free bytes.
The distinction between a measured cause and a general explanation is important. A different driver might batch entries, use a different page size, or expose an asynchronous operation with another balance of costs. A different workload might spend little time mapping and most of its time computing. MoonBright's argument is strongest where the request creates substantial translation state on a critical path; it should not be read as evidence that every memory-management delay has the same remedy.
Pool allocators are a rational response to that long pole. Instead of asking the driver to build mappings for every small request, a framework can obtain a large region once and subdivide it in user space. A later allocation then finds a free block in an already mapped region. The hot path is fast because it avoids the host-driver sequence. cudaMallocAsync follows this pool-based direction, and framework allocators such as PyTorch's caching allocator use related reuse strategies.
But a pool changes the question from “can the system map these pages?” to “does the existing layout contain a suitable hole?” Suppose long-lived tensors occupy alternating portions of a large region. The pool may have enough free bytes in aggregate, but those bytes are split into pieces. A new tensor that needs one contiguous virtual block cannot necessarily use them. The allocator can reserve more memory, wait for a block to be freed, or accept lower utilization. It cannot cheaply make the free pieces appear contiguous if its only practical operation is suballocation inside a fixed region.
Low-level VMM offers the missing layout freedom: physical pages can be associated with virtual addresses independently, so scattered backing could support a contiguous software-visible range. Yet under the diagnosed cost model, using that freedom dynamically means re-entering the host-controlled page-table path and potentially taking a global coherence barrier. A pool therefore hides the slow foundation by giving up some layout flexibility, while VMM preserves flexibility but exposes the slow foundation. The two approaches are not merely different APIs; they choose different places to pay.
This also clarifies the role of fragmentation. External fragmentation means usable physical memory is divided into pieces that do not fit a requested layout. Internal fragmentation is space lost because the allocation unit is larger than the requested object. Virtual remapping can address the first problem by changing which physical pages sit behind a virtual range. It cannot make a hardware page smaller, and it cannot by itself make physical resource acquisition instantaneous. Keeping those forms of waste separate prevents a diagnosis of host-side mapping latency from becoming an exaggerated claim about all memory inefficiency.
The practical consequence depends on lifetimes, not just sizes. A short-lived temporary buffer can leave a hole that later requests cannot use because a long-lived tensor sits beside it. Repeating that pattern can make a pool reserve more memory than it can effectively deploy. A policy that coalesces neighboring free blocks helps when the holes are adjacent, but it cannot join nonadjacent physical pieces into one software-visible range unless the underlying mapping layer can change page relationships. That is why the paper treats fragmentation as a consequence of the same foundation: the allocator has layout information, but the affordable operation set does not let it act on that information.
The evidence does not say that every GPU runtime, driver version, or workload has the same bottleneck. CUDA's documentation describes the VMM contract, and HIP's documentation describes a parallel conceptual interface with different hardware-dependent constraints. MoonBright's measured breakdown is tied to the paper's A100 experiment. The reliable systems-level lesson is more specific: when the operation that changes translation state is serialized through a host control path, a workload with many mappings can spend more time preparing access to memory than performing the computation that uses it.
That is the point at which MoonBright's architectural question becomes unavoidable. If the expensive component is the construction of many independent translation entries, can the device perform that data work while the host retains validation and resource ownership? The next step is not to assume the answer; it is to examine how MoonBright exposes page-table state and divides the mapping pipeline.