Where MoonBright Pays Off: Fragmentation and Dynamic Workloads
MoonBright becomes useful at the point where a fast mapping is more than a microbenchmark: an allocator can use it to serve changing requests and repair a virtual layout without copying live data. This chapter follows that translation layer into synchronous allocation, asynchronous pools, virtual stitching, and dynamic inference. The goal is to connect the reported results to the workload patterns that create them, while keeping separate the things MoonBright improves—mapping latency and external fragmentation—from the physical, hardware, and application costs it does not erase.
From low-level primitives to an allocator a framework can use
A practical allocator has to serve objects that do not all have the same size or lifetime. MoonBright's synchronous, cudaMalloc-like design combines its low-level primitives into a familiar request path. Large requests obtain a virtual range and physical pages, create any missing page-table structure, and populate the leaf entries. Small requests are handled through a slab layer because the hardware page is much coarser than many application objects. The slab avoids paying a page-table operation for every sub-page object, while the virtual-memory layer remains responsible for the larger mappings underneath.
That division is important because it gives each layer the job it can perform efficiently. The slab chooses a free slot in an already backed region. The lower layer obtains new backing and changes translations when the current region cannot satisfy the request. A cache is still useful; MoonBright does not reject reuse. Its difference is that the cache is no longer the only way to avoid a slow mapping path, and the underlying virtual-memory layer can create a layout that does not match one physically contiguous block.
The asynchronous design makes this division explicit. Its fast tier operates in user space and uses a best-fit-with-coalescing policy to satisfy frequent requests from a pool. Finding a free block and merging adjacent holes can happen without a driver transition. When the cache is exhausted or a request exceeds its threshold, the slow tier reserves a fresh virtual range, obtains physical page frames, creates missing page-table levels, and launches the GPU-side PTE population operation. The slow path still has work to do, but the expensive translation construction is expressed as a stream-ordered device operation rather than as a sequence of host-side entries.
Stream ordering changes how that slow path interacts with computation. A mapping kernel can be enqueued before the kernel that consumes the new pointer, creating a dependency between producer and consumer. Work in another stream can potentially overlap with the page-table population if it does not depend on that mapping. The allocator therefore does not need to make every stream wait simply because one request is acquiring new backing. This is not the same as making the operation instantaneous: the dependent work still waits for the mapping, and host-side physical allocation or structural setup can remain exposed.
Freeing has the opposite ordering requirement. MoonBright's deallocation path first ensures that active GPU work using the target memory has completed. It then invalidates the relevant entries, flushes stale translations, and only afterward returns the physical pages to the system allocator.