What Post-Copy Actually Does When the VM Resumes
M3U keeps the post-copy model but changes the host memory path underneath it. To understand why that change matters, we first need to follow an ordinary QEMU post-copy migration: what the destination prepares, what changes when the VM starts, and how a missing page travels between the two hosts. This chapter traces those transitions and the ownership of state at each one, so later performance problems can be located in the actual path rather than described as “migration overhead.”
From preparation to a running destination
QEMU describes post-copy as a sequence of states: ADVISE, DISCARD, LISTEN, RUNNING, and END. These names are more useful than a simple timeline because each state changes who can act on guest memory and what kind of work the migration stream must support.
The sequence also marks a gradual transfer of responsibility. Before post-copy begins, the source is the authoritative owner of a complete, running VM. During switching, the source pauses execution and the destination reconstructs enough state to become the place where execution will continue. Once the destination is running, neither host has the whole job in isolation: the destination owns the guest's next access, while the source still owns some of the bytes that access may require. The migration channel is the coordination boundary between those owners.
That distinction explains why the state names cannot be reduced to “copy, then start.” A normal copy can assume that the consumer waits until the producer has finished. Post-copy permits the consumer—the running destination—to ask the producer for data on demand. It must therefore prepare a memory representation in which “not present yet” is a deliberate state, provide a listener that can react to that state, and keep enough migration context alive after the guest starts for the source to answer requests. Every transition narrows or widens the set of operations that can safely occur concurrently.
In ADVISE, post-copy has been enabled as a possibility, but the destination has not necessarily been told to start it. The destination checks whether the operating system has the support required for post-copy and prepares the RAM mappings that will later be used to detect accesses to missing pages. This is an early failure point: if the destination cannot provide the required memory-fault mechanism, the migration should fail before the guest is placed into the more exposed running phase.
DISCARD begins when the destination receives the first discard command. The purpose is not to throw away the VM's logical contents. It changes how the destination treats memory that still has to be obtained from the source. QEMU also uses this stage to prevent new huge pages from being created during post-copy and to break huge pages that contain discarded regions. The destination is preparing memory to be missing in a controlled way rather than pretending that every page is already resident.
The key concurrency change arrives in LISTEN. The destination starts a listener thread that takes responsibility for receiving page data from the migration stream while the main migration thread continues processing the packaged state. The destination also registers or otherwise sensitizes the relevant RAM so an access to a missing page produces a userfault event. From this point forward, the migration cannot be understood as one thread reading one stream from beginning to end. Page reception and state restoration overlap because device loading or guest progress may need a page before the ordinary stream has delivered it.
QEMU packages part of the post-copy transition before device loading begins. The package includes the command that enters the listen state, device state, other migration sections, and a command that permits the destination to enter RUNNING. Reading the package as a whole matters because device emulation can access guest RAM while it is reconstructing a device. If the stream were consumed by an operation that could not also answer a page request, the device loader could wait on a page while the page that would release the wait remained trapped behind the same stream.
When RUNNING is reached, the destination synchronizes the required CPU and device state and starts the virtual CPUs and I/O devices. The main thread finishes the package and returns to ordinary execution. The listener thread remains responsible for page data until the migration ends. The source does two kinds of work during this phase: it scans pages that are still marked as needing transfer, and it reacts to explicit requests from the destination. When a request arrives, QEMU can restart its scan near the requested address, sending the urgent page and nearby pages sooner on the assumption that the guest may soon touch them as well.
The division between the main thread and the listener is a response to a dependency cycle. Loading a device may need guest memory. Guest memory may be missing. Resolving the missing page needs the migration stream. If the same execution path had to finish device loading before it could receive page data, it could wait for itself. The listener breaks that cycle by allowing page reception to continue while the main thread consumes the package. It does not make the work free; it makes the two responsibilities able to make progress at the same time.
The source-side scan has a similar scheduling problem. It is useful to send pages in bulk because each network transfer and page installation carries overhead. It is also necessary to respond quickly when the running guest exposes a particular gap. QEMU's documented behavior gives explicit requests priority by restarting the scan near the requested address, while still allowing pages to flow without individual requests. This is a compromise between prediction and reaction: background traffic tries to stay ahead of the guest, and fault-driven traffic corrects the prediction when it fails.
Finally, END is reached after the remaining page service is complete. The listener can exit and migration state can be cleaned up. The important point is that the destination has been running for much of the interval during which the source still owns some of the memory needed by that running guest. The lifecycle is therefore a controlled overlap between execution, background transfer, and fault-driven recovery.
A missing page is a conversation
The mechanism behind the conversation is Linux userfaultfd. User space creates a userfaultfd object, registers one or more virtual-memory ranges, and receives a message when a page fault occurs inside those ranges. The interface is deliberately general: a manager can read or poll for fault events and can resolve them with operations that copy page contents, provide a zero page, or continue an already populated mapping. A resolving operation is atomic from the reader's point of view, so a guest thread does not observe a half-installed page.
In post-copy, the destination's userfaultfd is the boundary between a guest access and a migration request. Suppose a guest instruction reaches a page that the destination has marked non-present. The faulting execution cannot continue with unknown data. A QEMU thread reads the event, obtains the fault address, and sends enough information across the migration channel for the source to identify the corresponding page. The source then uses its knowledge of pages still missing at the destination to send that page. When the data arrives, the destination migration path installs it through a page-delivery operation and wakes the faulting execution.
The interface deliberately keeps the guest unaware of the transport. The guest does not receive a special “remote memory” result and does not need to know whether the source or destination currently holds the page. It simply encounters a memory access that cannot yet be satisfied locally. QEMU and the kernel turn that access into an event, match the event to a migration address, deliver ordinary page contents, and make the mapping usable again. This separation is what allowed post-copy support to be built around userfaultfd without modifying KVM guest-memory code for the basic mechanism.
The abstraction also explains why a page fault is not equivalent to a network packet. A packet can be acknowledged when bytes arrive; a fault is complete only when the bytes have been copied or mapped into the right memory range, the relevant translation state is usable, and the blocked execution has been woken. Each step can involve a different owner and synchronization rule. At small scale, those steps may be hidden by fast service. At high-end scale, concurrent faults make their shared management structures visible.
Two details make this more subtle than a simple request-and-response RPC. First, the fault listener and the migration thread can run at the same time. A page may be sent proactively by the background path just as the listener is reporting a fault for it. The fault can then be resolved before the request is acted on; the listener must recognize that the page is no longer missing rather than treating the duplicate event as a new transfer obligation. QEMU's documented path accounts for this kind of race, including a case in which the fault has already been resolved and the parallel reader observes an EAGAIN-style result.
Second, the source is not idle while it waits for requests. QEMU's post-copy source behavior scans the remaining dirty or missing-page information and sends pages without being asked. This active pushing uses the migration bandwidth efficiently when the guest's next accesses can be predicted or when bulk transfer is cheaper than waiting for individual faults. An explicit request changes the priority: the source's scan can restart near the requested location, and the requested page can move ahead of less urgent background work. Demand paging and active pushing thus share a transport but have different timing objectives.
The same distinction explains why device state is packaged before the destination starts device loading. A device can touch RAM on behalf of the guest, and a pass-through device can issue DMA to an address whose contents have not arrived. If the state needed to reconstruct the device were left in an ordinary sequence behind page service, the device loader could become another consumer competing for the same response path. Packaging gives the destination the device information it must parse before it starts the device, while the listener remains able to receive pages that device processing or other state reconstruction needs.
The package is therefore a scheduling boundary, not just a serialization format. The destination reads the device-related material in one controlled operation, starts the listener at the point where page requests can be serviced, and delays guest execution until the required CPU and device state is synchronized. After that point, later I/O buffers can still be recovered through the ordinary fault path. What is avoided is the initial burst in which device reconstruction and the first guest/device accesses would otherwise compete to establish the same state while the guest is already running.
This is also why a post-copy diagram should show more than a single arrow from source to destination. Ownership changes in stages. The source initially owns the complete usable memory state. During switching, the destination receives enough control and device state to start. During running post-copy, the destination owns execution but not necessarily all of the memory contents. The source owns the missing copies and the knowledge of which pages remain to be sent. The destination owns the fault address and the decision that execution must wait. The migration channel connects those responsibilities but does not eliminate their coordination cost.
That split has a sharp failure boundary. If the migration channel is interrupted, QEMU can move both sides into a paused post-copy state. Recovery requires a new or repaired channel, a destination recovery command, a source-side resume, and a synchronization handshake. Pages may have been sent during the interruption but not received, so the handshake must make sure in-flight data is resent. A guest thread that accesses a page already present at the destination may continue, while a thread that needs a missing page can remain blocked until recovery completes. Guest asynchronous page-fault support may let some guest work be scheduled away, but it does not create the missing data.
This failure mode follows directly from the ownership model rather than from an incidental QEMU command. During the pause, the destination cannot safely assume that a request has been satisfied just because the source attempted to send it. Conversely, the source cannot discard its responsibility for a page until the two sides have synchronized their view of what arrived. The recovery handshake re-establishes that shared view before normal post-copy service resumes. It protects correctness, but the guest's visible latency is the time needed to repair the channel and reconcile the split state.
The lifecycle therefore solves one problem and exposes another. It solves the pre-copy convergence race by letting the source stop producing new dirty state. It exposes a live dependency between guest execution, user-space fault handling, page-table updates, and the source's transfer schedule. A correct request path can still be slow if the destination takes too long to install pages or if background and urgent transfers contend for the same locks and page granularity. The next question is where that contention occurs inside the host memory-management system, and why it becomes severe for a high-end VM.