The Hidden Bottleneck: Three MMU Conflicts
M3U's central diagnosis is that high-end post-copy does not slow down for one reason called “the kernel.” It slows down because several kinds of migration work meet inside the host memory-management system and are forced to share locks, page sizes, and consistency rules. This chapter separates the symptoms into three related conflicts—switching-time memory registration, post-copy page installation, and pass-through I/O faults—so the reader can tell which cause each later M3U mechanism is meant to address.
The first bottleneck is the stopped VM
The first cost appears before the destination is allowed to run. During the VM switching phase, the source is stopped and the destination must mark the remaining dirty memory as missing. This tells the destination that a guest access must not use an old or incomplete copy. It is a correctness step, but in a conventional path it is implemented through a large number of memory-management operations.
The relevant structure is the Host Page Table, or HPT. In the virtualization setting discussed by M3U, the HPT is the host-side structure that lets the user-space virtual machine monitor allocate, map, and unmap the memory used by the VM. An unmap is not just a bit flip in an isolated table. It can release a physical page, change the host mapping, coordinate with other translation structures, and trigger work needed to keep processors from using stale translations.
That sequence is manageable for a small number of pages. A high-end VM turns it into a count problem. In the paper's tested configurations, dirty-page registration accounts for 57–66% of total downtime. A 128 GB working set can trigger up to 32 million unmap operations, and the measured downtime exceeds three seconds under the reported random-write workload. The guest is suspended during this period, so every additional operation is directly visible as interruption rather than being hidden behind concurrent execution.
The important diagnosis is not simply “there are too many pages.” It is that the operation used to register those pages performs more work than post-copy needs at that moment. The destination will eventually need a way to make a missing page fault, but it does not necessarily need to destroy the physical placement of every page and rebuild it from scratch before the guest resumes. Conventional memory management protects a broad set of ordinary allocation and mapping cases. Post-copy has a narrower situation: the VM is stopped, its physical memory has already been allocated, and the destination deliberately wants some entries to look unavailable until their contents arrive.
That distinction separates physical ownership from guest visibility. The destination can retain the physical memory associated with a VM while temporarily telling the guest translation path that a page is not present. The page's location need not change merely because its contents have not yet arrived from the source. If the implementation instead treats “not currently usable by the guest” as “free this host page and later allocate another one,” it pays for allocation and deallocation that do not advance the transfer itself. The paper's diagnosis is that post-copy exposes this mismatch at high-end scale.
The stopped window also removes one source of concurrency: the guest is not executing ordinary memory accesses while registration runs. That does not make all updates independent, because workers still share page-table structures and the host must preserve translation correctness. It does mean that some invalidation and notification work designed for a running system can be reconsidered in this particular phase. The useful question is not whether an operation is generally necessary, but whether it is necessary before this VM resumes and whether its synchronization scope matches the region being updated.
Locking makes the operation-count problem worse. A lock that protects page-table updates or memory allocation keeps concurrent workers from corrupting shared state, but it also makes independent regions wait if they use the same protected path. The paper describes HPT updates as lock-serialized in the baseline. Thus, adding registration workers does not automatically turn 32 million operations into parallel progress. Workers can spend their time queued behind the same critical section, while the stopped VM continues to accumulate downtime.
This gives us a useful symptom-to-cause distinction. If switching time rises with the number of pages and the cost tracks the number of unmaps, the first suspect is not the post-copy network stream or the guest's page-fault pattern. It is HPT manageability: the way the host creates, removes, and marks VM memory during the suspended window. The paper's measurement that dirty registration occupies most of downtime supports that diagnosis for its platform and workload; it does not establish the same percentage for every hypervisor.
Page installation and device access create two more conflicts
The second cost appears after the destination resumes. A page that arrives from the source is not useful merely because its bytes have reached the target host. The destination must place those bytes into the memory associated with the guest and make the relevant translations agree about the page. M3U names three structures whose relationship matters: the HPT, the Extended Page Table (EPT), which supports guest execution by translating guest physical addresses toward host physical memory, and the I/O Page Table (IOPT), which supports address translation for pass-through device DMA.
These structures have different jobs but share a consistency relationship. If a page is allocated or remapped, the host must not let the CPU use one interpretation while the guest's execution translation or a device's DMA translation uses another. In the baseline path described by the paper, the page-in operation allocates a host page, copies the contents from user space into the kernel-managed memory, and updates the related tables under memory locks. Demand paging, active pushing, and vCPU fault handling can all reach this path at the same time.
The three-table relationship makes the critical section wider than the byte copy. HPT work determines where the host keeps the page. EPT work determines whether a guest vCPU can translate the guest physical address to that location. IOPT work determines whether a pass-through device can issue DMA to the same guest address. These views must agree before the page is exposed to all of its consumers. A page may therefore spend time waiting for table and notifier work even after its network transfer has completed.
This is also why a partial optimization can have a disappointing result. If one path updates EPT in parallel but page allocation, HPT updates, or IOPT-related work still takes the same exclusive lock, the other paths continue to queue. The paper uses TDP MMU as an example of a design that improves part of the cross-table problem but leaves important userfault-handler operations and the page-size conflict unresolved. The lesson is diagnostic: the visible bottleneck is the whole page-in path, not whichever table is easiest to parallelize in isolation.
That is why the observed post-copy problem is not solved by a fast link alone. The paper's baseline reaches only 9.2% of the available physical network bandwidth in the cited high-end experiment. The remaining capacity does not help if received pages wait behind lock-protected copying and cross-table updates. In one Redis test, the guest experiences a network interruption lasting more than 15 seconds after it has resumed. The paper compares a 24-second measured post-copy completion time for a 24 GB remaining set with a 3.84-second idealized transfer time at the stated bandwidth. The ideal number describes bytes on a link; the larger number includes the work required to make those bytes usable.
The same path also creates a page-size dilemma. A faulting vCPU wants a small unit. If it touches one missing 4 KB page, fetching a 2 MB region may transfer data the guest does not need and may delay the access while the larger unit is assembled. Active pushing has the opposite preference. It is a bulk path, so larger pages reduce the number of userfault notifications, copies, and page-table installations needed to transfer a large dirty region.
The baseline memory-virtualization path enforces one fixed page size across the relevant structures. Choosing 4 KB favors the urgent path but increases management overhead for bulk transfer. Choosing 2 MB improves bulk efficiency but can overfetch and can force the destination to wait for a larger unit before it can install a faulted page. This is not a matter of choosing the universally correct page size; it is a conflict between two legitimate timing goals that the same address-space organization cannot express independently.
The conflict is especially sharp because the two paths can touch related memory at the same time. A background stream may be reconstructing a large region while a vCPU asks for one page inside it. If the system treats the region as one indivisible unit, the urgent request waits for bulk work or forces the bulk path to change its schedule. If it treats every page as a separate 4 KB unit, the number of fault notifications, copies, and table installations grows, and the bulk path spends more time managing transfers than filling the link. High-end post-copy is where this choice becomes a system-level limit rather than a small implementation preference.
Page size also changes what “complete” means. A 2 MB region represented as 512 smaller pages cannot be declared fully restored until the individual pieces have arrived and the relevant mappings agree. Conversely, a 2 MB transfer can deliver data the guest will never touch during migration. The baseline's single-granularity rule therefore couples transfer efficiency, fault latency, and consistency work. A diagnosis that reports only network throughput misses the cost of deciding when a page-sized result is ready to expose.
QEMU's huge-page documentation illustrates the broader boundary. Post-copy requires compatible page sizes on source and destination, and a 1 GB page can take about one second to transfer across a 10 Gbps link, leaving the destination thread blocked until the full page arrives. The example is documentation guidance rather than an M3U measurement, but it makes the principle concrete: a page is also a scheduling unit, and a large unit can turn one access into a long wait.
The third conflict comes from pass-through I/O. A pass-through device can perform DMA directly toward guest memory. If the target page is not present, the device triggers an I/O page fault, or IOPF, through a path that includes a PCIe transaction and a host interrupt before the host can service it. The device's I/O queue may remain suspended during that process. Under a bursty network or storage workload, one unresolved page can therefore delay more than the guest thread that first exposed the missing data.
The paper reports roughly twice the average latency for IOPF demand paging compared with vCPU faults and up to 3.21 times the maximum in its comparison. It also finds that most IOPFs happen in the first few seconds after the VM resumes. Descriptor faults dominate the tested pattern: VirtIO queue metadata lives in fixed, reused locations, while data buffers are dynamically allocated and are usually touched by the CPU first, causing ordinary vCPU faults instead. This timing and structure make the IOPF problem different from general page-in contention. The system is not only slow at resolving a fault; it is repeatedly exposing a predictable device path to a high-latency event.
The descriptor and buffer distinction matters for diagnosis. Descriptor pages contain the metadata that tells the device where buffers are, how large they are, and which queue entries are active. They occupy fixed structures that the driver revisits, so an early missing descriptor can interrupt the device before ordinary guest execution has recovered much memory. A data buffer has a different history: the guest CPU often touches or initializes it before the device consumes it, so a normal vCPU fault may bring it in first. M3U's evidence does not say that every device follows this exact pattern; it says that the pattern is strong enough in the tested VirtIO setting to make pre-installation a targeted alternative to handling every IOPF reactively.
The three diagnoses also operate on different time scales. HPT registration is concentrated in the short period when the VM is stopped. Cross-table page installation continues throughout the post-copy phase and competes with guest execution. IOPFs are concentrated around device resumption but can amplify their cost through queue blocking and packet loss. Treating these as one undifferentiated “migration latency” would hide the reason a single optimization cannot address all of them.
These three conflicts are connected but not interchangeable. Dirty registration dominates the stopped window because HPT operations are repeated across many pages. Page-in degradation appears after resumption because HPT, EPT, and IOPT updates contend while the guest is executing. IOPFs are a device-specific version of the missing-memory problem whose hardware path can stall an entire queue. A useful diagnosis must keep the symptoms distinct even though all three expose the same broader condition: post-copy asks a general memory-management system to serve many concurrent, differently urgent operations under rules designed for stronger serialization.
That diagnosis also sets the design test for M3U. A successful response cannot simply remove locks, choose one larger page size, or add more migration threads. It must identify which work is redundant during switching, which copies can occur independently of consistency updates, which page granularities each path needs, and which device accesses can be made non-faulting in advance. The next chapter examines how M3U makes those changes while retaining the mappings and state relationships that the three conflicts reveal.