The hidden System VM: how one daemon becomes a platform

ArcBox's promise of one local runtime depends on a machine most users never need to name: the Linux System VM. This chapter follows what happens between the macOS daemon and that guest, focusing on ownership, readiness, and backend choice. Understanding that path lets the reader judge both sides of ArcBox's design: the System VM amortizes the work needed by containers and sandboxes, but it also concentrates their dependencies and failure modes in one place.

Two virtualization layers, two different jobs

The first distinction is between the VM that hosts ArcBox and the microVMs that ArcBox creates. On a Mac, the host daemon owns the top-level runtime. Its Runtime object coordinates machine records, automatic VM lifecycle, the container backend, networking, migration, and—on Apple Silicon—the separate manager for macOS guests. One of its primary responsibilities is to keep the default Linux machine ready for ordinary container traffic.

That default machine is the System VM. It is not a disposable sandbox and it is not just a renamed Docker container. It is the Linux guest in which ArcBox's Docker service, Kubernetes components, guest agent, persistent workload data, and sandbox manager live. When the sandbox tier is used, the System VM becomes the host for another layer: the guest-side computer runtime asks a VM driver to create nested Firecracker microVMs. A command may therefore travel through a macOS process, into the System VM, through a sandbox manager, and finally into a workload-specific microVM. The layers have different owners and different reasons to exist.

That ownership distinction also explains the difference between stored state and live state. The System VM has a persistent Docker data image, so images and container metadata can survive a stop and restart. A nested sandbox has its own record, root filesystem or overlay, network lease, and possibly a checkpoint. A process running inside either environment is more transient: a backend switch can preserve the Docker image while stopping live containers, and a sandbox can preserve a paused overlay while releasing the microVM. ArcBox's reuse is therefore not one undifferentiated persistence guarantee. Each layer decides which state it can safely retain.

The top-level VM monitor, arcbox-vmm, owns the host-side concerns: guest memory, virtual CPUs, device registration, boot images, and the connection between a chosen hypervisor and the guest. The guest-side computer runtime does not know how the host VM is implemented. It talks to a driver interface whose vocabulary is deliberately narrower: boot a VM, observe its state, stop it, and expose optional capabilities such as vsock, snapshots, staging, adoption, or memory ballooning when a driver actually supports them.

This interface is more than a software-engineering convenience. It prevents the sandbox manager from quietly assuming that every virtual machine has the same shape. Firecracker is the reference driver for nested sandboxes, but the port can also describe other VMMs, including an in-process Hypervisor.framework engine or a managed Virtualization.framework VM. A driver reports its capabilities and the handle exposes only the capabilities that both the driver and the VM specification make possible. If checkpoint restore needs a jailed layout, for example, the higher layer can refuse the operation instead of allowing a later path failure to masquerade as a generic boot problem.

It also creates a useful failure vocabulary. A missing optional capability is different from a failed operation. A VM without a vsock device cannot be dialed by a guest-agent client; a driver without checkpoint support should not be asked to save state; a VM launched without the requested device should not advertise that capability merely because another VM of the same type has it. This is how a shared runtime remains honest while supporting several VMM implementations.

The same separation appears in communication. Host services reach the System VM through virtual sockets rather than treating guest processes as local processes. The Docker compatibility layer uses a host Unix socket for the client-facing API and a vsock connection to the guest engine. The sandbox runtime uses its own guest-agent protocol for execution, files, and lifecycle operations. A socket is therefore not the boundary by itself; it is the transport across a boundary whose ownership has already been defined.

That organization gives ArcBox a useful center of gravity. Docker containers and nested sandboxes can share the System VM's boot assets, persistent storage, network datapath, DNS behavior, and guest services. The daemon can expose them through familiar or purpose-built APIs without booting an unrelated Linux environment for every operation. But the center of gravity also becomes a shared failure domain. If the System VM cannot boot, if the agent is not ready, if its route is stale, or if a required runtime binary fails verification, neither the Docker path nor the sandbox path can proceed normally.

sequenceDiagram participant D as Host daemon participant C as Control plane participant A as Asset store participant V as System VM participant G as Guest agent participant S as Runtime services D->>D: Prepare host and acquire daemon lease D->>C: Bind control-plane socket early D->>D: Release stale resources D->>A: Verify and prepare boot assets D->>V: Construct and boot selected backend V->>G: Boot Linux guest and answer readiness G-->>D: Guest-agent ready D->>S: Start DNS, Docker, and optional Kubernetes services S-->>C: Publish network-ready and ready phases D-->>C: Publish failure instead if a required step fails

From launch to a usable System VM

ArcBox treats startup as a control-flow problem, not as a single call to a hypervisor. The daemon first prepares its host directories and configuration, then acquires an exclusive lease. The lease is kernel-managed through an advisory file lock, so a crashed process does not leave a permanently held lock. If another ArcBox daemon is still alive, the new one can ask it to shut down and wait for the lock rather than starting two competing runtimes.

That lease protects more than a process name. The daemon also owns Unix sockets, disk images, network state, and the relationship between host-side records and guest resources. Two daemons booting against the same Docker image could corrupt data even if their process IDs were different. The lock is therefore the first ownership gate in a sequence that later includes resource cleanup, guest readiness, and service publication.

The control plane is bound before the slow parts of startup. That choice looks counterintuitive until the user interface is considered: Desktop or CLI clients need a place to observe progress while stale resource holders release disk images, assets are prepared, or the guest is booting. The socket can answer setup-status calls even while the full runtime is unavailable. A client that connects too late sees fewer phases, so a missing early phase is treated as “already passed,” not as proof that the phase took no time.

After stale resources are released, the daemon prepares the boot assets. ArcBox's Linux guest boots directly into a read-only EROFS root filesystem; there is no initramfs in the documented boot path. The runtime and guest-agent binaries arrive through VirtioFS, but the guest does not execute the transport copy blindly. It verifies binaries against the pinned manifest, copies them into a version-keyed generation on its persistent Btrfs data disk, and executes from that materialized generation. The distinction matters: an application bundle can transport a new runtime generation without turning the persistent Docker and sandbox data into part of the immutable boot image.

The asset manifest is consequently part of the trust and recovery story as well as the boot story. A missing or non-executable guest binary is detected before the VM is presented as ready. A later generation can be materialized beside an older one, with a stable active link inside the guest. The precise files are implementation details; the important boundary is that immutable, checksummed inputs and mutable workload state have different lifecycles.

The daemon then constructs the System VM and waits for the guest agent, rather than treating the creation of a VM process as readiness. The agent's response is the point at which the guest can accept the operations ArcBox needs. Only after that does the daemon bring up the runtime services that make the machine useful: DNS forwarding, the Docker API, critical recovery paths, and, where configured, Kubernetes integration. A setup phase can therefore say “the VM is starting” without claiming that docker run will already work.

Rust types enforce this ordering in the daemon's startup pipeline. Host preparation produces a context that can acquire the daemon lease; the leased context can start the control plane; the control-plane state can wait for resources; and only the asset-ready state can boot the runtime. Reordering one of these transitions is not merely a code-review concern. It becomes a type error. When a required step fails, the daemon publishes a failed setup phase, leaves the error text in the status, waits briefly for connected streams to receive that final event, and exits non-zero.

The sequence is designed to make failure recoverable rather than merely visible. On a graceful exit, ArcBox drains DNS, Docker, and control-plane connections, stops the VM, and removes its sockets while keeping persistent images. After a crash, socket files may remain and Virtualization.framework helpers may still hold disk images. The next startup removes stale sockets, waits for possible image holders for a bounded period, and rebuilds stale network state. That does not eliminate every manual recovery case, but it turns common process failure into a lifecycle transition the daemon knows how to inspect.

The backend decision comes after the ownership model but changes the mechanics of the boot. On macOS, ArcBox supports two host-side routes. The Virtualization.framework route delegates managed virtual-machine execution to Apple's framework. The custom Hypervisor.framework route gives ArcBox direct control over the VM monitor: it allocates guest memory, registers VirtIO devices, starts a thread for each virtual CPU, dispatches memory-mapped I/O traps, injects interrupts through the virtual GIC, and runs dedicated workers for I/O. The first route reduces the amount of host execution machinery ArcBox must own. The second makes it possible to tune device emulation and the networking datapath in ways a managed framework may not expose.

That control is not free. A manual vCPU loop has to know what to do when the guest accesses a VirtIO register, waits for an interrupt, invokes a power-management call, or requests a reset. Network and block workers hold pointers into guest memory, so shutdown must stop and join them before that memory is unmapped. The custom path also needs explicit kicks to bring vCPUs out of a wait state when an I/O worker injects an interrupt. The teardown order—DAX mappings, interrupt controller, VM, then guest memory—is part of correctness, not a performance footnote. A managed VM hides much of this work; it does not necessarily provide the same control surface.

The selected backend also determines how ArcBox handles Linux image compatibility. On the VZ path, linux/amd64 images rely on Rosetta when the System VM is wired for it. On the custom HV path, the guest uses FEX supplied in the verified runtime generation. If the relevant translator is absent, ArcBox is designed to fail closed rather than silently run an unsupported fallback. This is a good example of the shared substrate's promise and limit: a single System VM can host both native arm64 and translated amd64 workloads, but only when the backend, hardware, and assets satisfy the translator's contract.

The System VM can be changed after installation, but not invisibly. Switching between the HV and VZ backends gracefully stops the VM, records the new choice, and boots it again. The persistent Docker data image is preserved, so images and container data are not discarded merely because the host execution path changed. Running containers are stopped during the restart, however, and the machine record may be recreated when backend-specific kernel command-line configuration has drifted. Persistence protects stored state; it does not promise continuity for live processes.

That distinction is the practical meaning of centralization. ArcBox gains one place to manage boot assets, guest readiness, Docker connectivity, DNS, x86 translation, and the substrate on which sandboxes depend. It also inherits one place where failure can block several product surfaces. The architecture is strongest when the shared machine is treated as a managed platform with observable phases, verified assets, and explicit recovery—not as an invisible implementation detail that users should never reason about.

The next boundary is therefore familiar but not trivial. Once the System VM is ready, ArcBox has to make its guest Docker service feel like a local engine without reimplementing everything Docker already knows how to do. That is the job of the compatibility path: a host socket, a selective router, and a guest engine connected by vsock.

References