Inside the Control Plane: Who Owns a Container?
In apple/container, the command-line interface is not the component that directly owns a Linux guest. The important question is how a request travels from the CLI through the project’s API and service boundaries until one per-container runtime assembles a virtual machine. Following that ownership path makes the design easier to understand and gives readers a way to predict where state lives, where a failure will appear, and who is responsible for cleanup.
This matters because the project combines two kinds of work that look like one command to a user. A request such as container run needs image access, persistent container metadata, a kernel and initial filesystem, a network attachment, process I/O, and a live guest. Those resources have different lifetimes. apple/container keeps them coordinated by making the CLI a client of a long-lived control plane and by giving each running container its own runtime service.
Why the command is not the runtime
The CLI is the visible edge of the system. It parses options, prepares a request, and presents output, but it does not need to stay responsible for every process or VM after the request has been accepted. When a user starts the system with container system start, the project launches container-apiserver as a launch-managed service. The API server exposes client APIs through XPC, Apple’s interprocess communication mechanism, and remains available for operations such as listing containers, creating resources, starting processes, collecting statistics, and stopping workloads.
That separation solves a practical coordination problem. A shell command is short-lived and tied to one terminal; a container may be detached, restarted, inspected by another command, or left stopped for later use. If the CLI were also the sole owner of the runtime, detaching would make ownership ambiguous and a terminal exit could be confused with a workload exit. The API server gives the project a stable place to retain configuration and mediate all later requests.
The service lifecycle is independent of any one terminal for another reason: launchd can manage the server and its helpers as operating-system services. Stopping the system therefore means more than closing a CLI process. It deregisters the service group, while a container that was already created can still have durable configuration waiting in the application data area. Conversely, a running container has live resources that must be stopped through the control plane before its runtime helper can disappear cleanly. This is the first sign that “the command ended” and “the container ended” are different events.
The server is not a monolith that performs every operation itself. Its services divide responsibility around the resources that have different state and failure modes. The image and content service manages local OCI data. The network service works with the vmnet-backed network plugin and allocates addresses and interfaces. The volume service manages named guest storage. Other services handle kernels, health checks, disk usage, and container state. The technical overview describes the same arrangement at a higher level: the API server starts helpers for images and networking, while the CLI communicates with the server rather than reaching into those helpers directly.
XPC is the interface boundary that keeps this division operational. A request is routed to a service that understands its resource instead of becoming a direct call from the CLI into a virtual machine. That indirection adds a failure surface—an API service or helper can be unavailable—but it also makes ownership visible. An image problem belongs to a different layer from a missing network attachment; a failure to contact the API server is different from a process that exits inside a healthy guest.
The service model can be summarized without turning the project into a source-tree catalogue:
The arrows do not mean that every operation follows one identical call path. They show the important ownership relationships. The API server is the system-level coordinator; the runtime helper is the owner of one container’s live guest; the guest workload is a process inside that guest. Image, network, and volume services provide resources to the runtime but do not become the owner of its process lifecycle.
The diagram also distinguishes control from possession. The API server can ask a network service for an attachment and can register a runtime helper, but that does not mean the API server contains the guest’s process state. The runtime helper can expose operations over XPC, but the Linux VM remains the place where the workload, its filesystem, and its process signals are realized. Keeping those distinctions intact prevents a common debugging mistake: treating a successful API response as proof that the guest process is healthy, or treating a guest failure as proof that the image store or control plane is broken.
Where a container becomes a guest
The boundary between a stored container and a live VM appears in stages. Creation begins with a configuration that names the image, process, resources, mounts, networks, and runtime choices. The API service persists that configuration in a container bundle and prepares the image filesystem and initial filesystem needed by the runtime. At this point, a container can exist as a stopped resource. There is an identity and a description of what should run, but there is not yet a running guest process.
This two-stage design separates desired state from realized state. The configuration says which image and process should be used and which bridges should be attached. Bootstrap tests whether those instructions can be turned into a runtime: can the helper be registered, can the kernel and initial filesystem be opened, can the network service provide an attachment, and can the virtual machine be created? A failure in any of those steps can prevent the workload from starting even though the container’s saved identity remains valid.
Bootstrap changes the kind of state involved. The API server registers a runtime helper for that particular container with launchd and creates a client for the helper. The helper exposes its own XPC service, identified separately from the main API service. This per-container registration is the point at which apple/container turns a saved description into an independently managed runtime. A failure here can leave creation metadata intact while the guest never reaches the boot stage; it should not be diagnosed as an application-level process failure.
The runtime then assembles the guest. It reads the persisted configuration, selects the kernel and initial filesystem, and creates a VZVirtualMachineManager for the Linux environment. It attaches the container’s root filesystem, allocates its network interfaces, configures mounts and sockets, and prepares standard input and output. The project’s low-level Containerization package supplies the Linux-container and virtualization machinery; the runtime helper supplies the project-specific configuration and service boundary around it.
The network is allocated before the guest is fully usable. The runtime asks the network service for an attachment, converts that attachment into a guest interface, and uses the result to configure the guest’s network and host entries. A published port may also cause the runtime to start a host-side forwarder. This is why a container can be “created” yet have no current IP address: the address belongs to a live network attachment, not merely to the saved image or container name.
The same distinction applies to process state. Bootstrapping starts the VM and its guest support, moving the runtime into a booted state. Starting the container’s init process moves it into a running state. Additional commands such as exec are processes managed inside that already-running guest. A terminal attached to container run is therefore connected to a process through several layers of I/O forwarding; it is not the place where the process itself is born.
Logs follow the boundary as well. The runtime writes guest boot output and container standard output to files in the container bundle, while also forwarding output to the CLI when the invocation is attached. Statistics are obtained from the live Linux container and returned through the runtime API. A stopped container can retain logs and configuration because those are persisted resources, even though its VM, network sessions, process handles, and port forwarders no longer exist.
What survives, and what must be rebuilt
The API server reloads container bundles when it starts. It can recover the identity, configuration, and stopped state of a container from the persisted resource, then wait until a later start operation registers a new runtime helper and recreates the live pieces. This is a useful division between durable intent and ephemeral execution:
- The image reference, process configuration, resource choices, mounts, labels, and other settings describe what should be run.
- The VM, guest agent, process handles, network sessions, and host port forwarders describe what is running now.
The second group cannot simply be treated as a file that survived a daemon restart. It is reconstructed from the first group when the runtime boots. That is also why an IP address may change after a stop and start, and why a helper can be deregistered even though the stopped container remains visible to container list --all.
Exit handling makes ownership especially clear. When the init process exits or the VM stops unexpectedly, the API server’s container service stops tracking the runtime, asks it to shut down when possible, deregisters its launchd service, clears live network state, and marks the container stopped. If the container was created with auto-removal, the service then deletes the container bundle. If it was not, the configuration and logs remain available for inspection or later cleanup.
This layered cleanup is more careful than simply deleting a directory after a process exits. The runtime first stops the guest and closes port forwarders and network sessions. The API service then updates the system-level snapshot and decides whether the durable resource should remain. A failure in one cleanup step is therefore not proof that all cleanup failed, nor is an absent runtime proof that the saved container should be deleted.
The ownership model helps separate common symptoms. If the CLI cannot contact container-apiserver, the control plane or system service is the first suspect. If the API server lists a stopped container but no guest is running, that is a valid persisted state, not necessarily corruption. If bootstrap reaches the runtime but network allocation fails, the image and process configuration may be fine while the network service or host framework is not. If the guest is running but the command exits, the process lifecycle—not the API-server lifecycle—determines the result. The same symptom, such as “nothing is listening,” can therefore have different causes depending on which boundary the request crossed successfully.
There is a corresponding limit to what this architecture promises. The project’s public interfaces for ordinary CLI operations are more stable than internal helper labels and non-public XPC details. The helper names explain the current implementation, but they are not a reason to build an integration that depends on private routes. The durable lesson is responsibility: the API server coordinates resources, the per-container runtime owns one live Linux guest, and the guest process owns its own application behavior.
That responsibility map turns a static architecture into a temporal question. Once the actors are clear, the next question is how they move in sequence: how a Dockerfile reaches the BuildKit builder and OCI store, how a saved image becomes a booted guest, and how a process, its logs, and its cleanup state move through the resulting lifecycle.