When Moving a Big VM Becomes a Memory Problem
M3U is a proposed memory-management design for post-copy live migration of high-end virtual machines. The question here is not simply how to move a large amount of memory faster. It is why a migration method that can escape pre-copy's convergence problem still develops long pauses, slow recovery, and guest-visible disruption when the VM has many processors, a large memory footprint, and an intense workload. This opening establishes that problem and the judgment this series supports: whether M3U changes the host memory path in a way that addresses the real bottleneck.
The scale changes the migration problem
Live migration is supposed to move a running virtual machine from one physical host to another without making the guest application care. Cloud operators use it for maintenance, workload rebalancing, and other changes that would otherwise require stopping the machine. The promise becomes harder to keep as the VM grows. A high-end VM is not merely a small VM with more memory. It can have dozens of vCPUs writing memory at the same time, a large working set that must be transferred, and network or storage devices generating bursts of I/O while migration is in progress.
The M3U paper uses a concrete operating definition: at least 64 vCPUs and 256 GB of memory, with configurations that can also include very high network and disk performance. Those thresholds describe the paper's subject, not a universal definition of a high-end VM. They are useful because they expose a change in regime. A page-management operation that is harmless when it is performed thousands of times can become the dominant cost when it must be repeated across a very large address space. A lock that protects a short operation can become a queue when many vCPUs, migration workers, and fault handlers reach it together.
That distinction matters because migration has more than one clock. There is the time during which the source VM is stopped, the time needed to finish transferring memory after the destination resumes, and the time during which the guest serves requests below its normal rate. A migration can finish in the sense that all pages eventually arrive while still violating the practical promise of continuity. M3U keeps these outcomes separate by focusing on downtime, post-copy completion time, and guest performance rather than treating “migration completed” as the only success condition.
The paper's motivation comes from production observations as well as controlled tests. In an analysis of more than 50,000 high-end-VM pre-copy samples from its cloud infrastructure, the reported migration success rate was 81%, with convergence failure identified as the cause of the unsuccessful cases. That is evidence about one infrastructure and workload population, not a general industry rate. It does show why the conventional method deserves scrutiny at this scale: the VM can keep changing its memory faster than the migration system can make progress.
The choice is not speed versus slowness
Pre-copy tries to minimize the final interruption by copying memory while the VM continues to run. The migration system sends an initial copy, tracks pages dirtied afterward, and sends those pages again. Each round should reduce the remaining set until the source can be stopped briefly and the final changes can be sent. The method depends on a convergence condition: the rate at which the guest creates dirty pages must eventually fall below the rate at which the system transfers them.
A write-intensive high-end VM can violate that condition. If its vCPUs dirty pages as quickly as the network removes them, each round leaves another large set to copy. More bandwidth can help, but it does not change the fact that the source keeps producing new work. A migration system can spend time and network capacity repeating copies without reaching a safe stopping point.
Post-copy changes the order of events. The source VM is suspended for a switching phase, the destination restores enough CPU and device state to run, and the destination resumes before all of the VM's memory has arrived. Memory that is still missing is not copied repeatedly while the source continues to modify it. Instead, the destination obtains the remaining pages through a combination of demand paging—fetching a page when the guest touches it—and active background transfer. Because the source is no longer executing and dirtying the VM, the remaining memory has a path toward eventual completion.
That is the central reason post-copy is attractive for high-end VMs: it replaces an open-ended dirtying race with a finite missing-memory problem. It is not a claim that post-copy is automatically gentle on the guest. A guest thread that touches a page still in transit can wait for that page. Many such accesses can turn the destination into a machine that is technically running but repeatedly interrupted by remote memory fetches. The migration can also spend much longer in its post-copy phase than the raw network transfer time would suggest, because receiving a page is only part of making it usable.
Post-copy also changes the failure boundary. During the post-copy phase, part of the VM's memory state is at the source and part is at the destination. QEMU's documentation therefore warns that a failure of either side can cause the guest to be lost. Network recovery is possible in documented circumstances: both sides pause, an administrator establishes a recovery channel, and the source resumes the interrupted migration after the two sides synchronize. But a destination thread that needs a page still held at the source can remain blocked until recovery finishes. Avoiding convergence failure is therefore a feasibility advantage, not a guarantee of uninterrupted service or failure tolerance.
The important contrast is this:
- Pre-copy keeps the guest running longer, but the guest can create more memory work than the migration can drain.
- Post-copy stops the source's production of new dirty state, but makes the running destination depend on timely remote page recovery.
M3U studies what happens after the second choice. Its subject is not a replacement for post-copy, and it is not a general argument that every large VM should migrate this way. It asks whether the host memory-management work required by post-copy is itself organized in a way that high-end machines can sustain.
The hidden question is how memory is managed
At first glance, the problem seems like a network problem. If a 100 Gbps link is available, why should a migration with enough bandwidth still pause or recover slowly? The M3U paper's answer is that every page must pass through a host memory-management path before the guest can use it. The path must decide which memory is missing, receive or copy the contents, install the appropriate mappings, and keep several translation structures consistent. Those operations can be serialized even when the network has capacity and even when many pages could be transferred independently.
The cost appears first during the switching window. Before the destination resumes, the remaining dirty memory has to be marked as unavailable there. In the baseline described by the paper, that means many page unmap operations. For a 128 GB working set, the paper reports up to 32 million unmaps, with dirty-page registration accounting for 57–66% of total downtime in the tested configurations. The operation is correct, but its allocation, deallocation, mapping, and lock-protected bookkeeping do not scale gracefully with the number of pages.
The cost appears again after resumption. A missing page fetched from the source must be integrated into the destination's address space. The host's memory-management subsystem coordinates multiple tables used for ordinary host memory, guest execution, and pass-through device access. Locks intended to preserve cross-table consistency can make demand paging, background pushing, and guest faults wait for one another. In the paper's baseline experiment, those paths used only 9.2% of the available physical network bandwidth. The unused bandwidth was not enough to solve the control-plane queue in front of it.
There is also a granularity conflict. A demand fault is urgent and usually needs only a 4 KB page: fetching a larger unit can make the guest wait for data it did not request. Active pushing is a bulk-transfer operation and benefits from a larger unit, such as a 2 MB page, because fewer faults, copies, and page-table installations are needed. A single fixed page size forces the urgent and bulk paths to compromise. Smaller pages reduce fault delay but add management overhead; larger pages improve bulk efficiency but can overfetch or make a fault wait for a larger transfer.
Pass-through devices add a third pressure point. A device performing DMA can touch a page that is not present at the destination and trigger an I/O page fault. That path includes hardware transactions and host interrupt handling, and a fault can stall an I/O queue rather than just one guest thread. The paper finds that these faults are concentrated early after resumption, especially around fixed VirtIO descriptor structures. If the system can identify and transmit that state before the guest runs, it may avoid a repeated critical-path cost instead of trying to make every device fault cheap.
M3U's response follows this map of the problem. It uses lightweight page flagging and parallel region ownership to reduce the work in the switching window. It separates page-data copying from the later updates that restore cross-table consistency, allowing background transfer and urgent demand paging to use different paths and page sizes. For pass-through devices, it identifies and transmits predictable device state before resumption. The design does not remove the need for correctness; it changes which work must be serialized, which work can be avoided, and which state can be made ready in advance.
The question for the rest of this series is therefore narrower and more useful than “Can a very large VM be moved?” It is: can post-copy's memory, fault, and device paths be reorganized so that high-end scale no longer turns correct work into a serialized interruption? To answer it, the next chapter follows a missing page through QEMU and Linux userfaultfd, from the guest access that exposes the gap to the transfer that fills it.