Two Kernels, Two Cgroup Stories
This chapter examines Kata Containers’ host-cgroup design: how a workload in a guest VM, its host-side VMM, the shim, and I/O threads are placed under resource controls. The practical question is what sandbox_cgroup_only changes, why the pod-overhead assumption matters, and why an overhead cgroup outside the workload limit is a governance choice rather than evidence that the host is using no resource. The goal is to make process ownership visible before we trace a single memory page through the system.
A conventional process-based container encourages a simple picture. The application’s processes are placed in a cgroup, and the host kernel applies CPU, memory, and other controls to that group. The container runtime is involved in creating the group, but it does not create a second operating system whose work must be represented separately. Kata adds that second system. The workload runs with a guest kernel inside a VM, while the host runs the VMM and the processes that connect guest devices to host resources.
That division does not make the host-side processes incidental. A VMM can consume memory and CPU. I/O threads can spend time moving data or managing queues. The Kata shim remains part of the sandbox’s control path. These processes are not application processes from the guest’s point of view, but they are real host consumers. A resource policy that ignores them can be too small for the sandbox; a policy that includes them without planning can put application work under pressure from its own virtualization layer.
The sandbox has a guest side and a host side
Kata’s host-cgroup documentation describes two layers. Inside the guest, the workload is arranged according to the container and pod structure that the guest runtime exposes. On the host, the VMM and associated threads must also be placed somewhere in the host cgroup hierarchy. The two layers are related, but they do not automatically share the same accounting identity.
The host layer contains several different responsibilities. The VMM provides the virtual machine boundary and manages virtual devices. The shim supervises the sandbox and coordinates with the container runtime. I/O-related threads handle the path between guest requests and host-side services. The vCPU threads represent the guest’s virtual processors and are the closest host-side counterpart to the workload’s computation. A useful resource design does not need to pretend these are the same kind of process. It needs to decide which of them the pod’s limits are intended to govern and what the owner will do with the rest.
The orchestrator contributes an important assumption. In Kubernetes, the pod cgroup is not simply an arbitrary folder into which the runtime can place every process after the fact. The orchestrator sizes it from the container resource requests and may add a configured allowance for sandbox overhead. If that allowance is present and accurate, the pod boundary can include both the application and the machinery needed to run it. If it is absent or too small, putting all sandbox processes inside the pod can make the workload compete with the VMM for a budget that was calculated only for application processes.
This is why the Kata design exposes an explicit choice instead of trying to guess. The setting sandbox_cgroup_only tells the runtime whether the pod cgroup is assumed to have been sized for the sandbox. The setting does not cause the runtime to discover the correct answer dynamically. The infrastructure owner must understand how the upper orchestration layer creates and sizes the pod cgroup, then choose the placement model that matches that contract.
The distinction is easier to see as a setup sequence:
The diagram shows a contract, not a universal runtime behavior. The first branch works only if the pod budget already accounts for the cost of the sandbox. The second branch protects the workload’s application budget from non-vCPU sandbox work, but it moves responsibility for that work to a different boundary. If that boundary is left unconstrained, the host can still carry the cost; it is simply no longer charged against the application’s pod limit.
One flag, two different resource policies
When sandbox_cgroup_only is true, Kata creates a per-sandbox child under the pod’s cgroup hierarchy. The documentation describes the VMM, shim, I/O threads, and vCPU threads as running within that sandbox child. Because the child inherits the parent’s applicable constraints, the pod’s host resource policy applies to the processes that make the sandbox function.
This is the cleanest arrangement when the upper layer has done its part. A pod-level memory limit is then meaningful as a limit on the workload plus the expected sandbox overhead, rather than as a limit on the workload while the VMM is treated as someone else’s problem. Host statistics gathered from the pod hierarchy can include the sandbox work. The resource owner can also reason about noisy neighbors more directly: a Kata workload that consumes excess host memory through its VMM or I/O path remains inside the same governing boundary.
The cost is that the budget must be honest. If a pod was assigned a small memory request because the orchestrator counted only application processes, then placing the VMM and I/O threads under that budget does not create free capacity. It exposes the missing overhead as pressure. The likely symptom may be performance degradation or an out-of-memory event where a process-based container would have appeared to fit. That is not necessarily a failure of the cgroup mechanism. It may be the mechanism enforcing a budget that was never large enough for the VM architecture.
The same placement also improves the meaning of measurements. When the sandbox processes remain in the pod hierarchy, a host observer can ask how much of the pod’s reported resource use comes from the application and how much comes from the sandbox child. The two are still distinguishable, but they share a governing parent. This supports the documented advantages of more complete pod-level statistics and stronger host isolation.
When sandbox_cgroup_only is false, documented as the default mode in the retained design, Kata takes a different view of the pod budget. It keeps the vCPU threads in the pod’s sandbox cgroup, while moving the shim, VMM, and I/O-related processes into per-sandbox groups under a dedicated overhead cgroup. In the documentation’s example, that overhead hierarchy is called /kata_overhead.
This policy assumes that the pod cgroup is sized for the actual container workload, not for every host process required to implement the sandbox. That can prevent the VMM, shim, and I/O threads from consuming the memory that the guest workload expects to use. In particular, separating non-vCPU sandbox processes can reduce the chance that a VMM or its I/O machinery pushes the application’s pod cgroup into a catastrophic out-of-memory condition.
The protection is conditional, however. Kata does not impose constraints on the overhead cgroup in this mode. The infrastructure owner can pre-size an existing overhead group, let Kata create one and leave it unconstrained, or resize it later. If no one governs it, the arrangement converts a visible pod-budget problem into a host-level resource risk. The application can remain within its assigned limit while the VMM and I/O threads consume resources outside that limit. The resource has not disappeared; the accounting and enforcement boundary has moved.
That trade-off is not automatically wrong. A platform may deliberately want a sandbox’s support processes to have room to keep the VM alive even when the application is under memory pressure. It may also want to measure sandbox overhead separately so that future pod sizing is based on evidence rather than a guess. But the separate group must then be treated as a first-class resource domain. “Outside the pod cgroup” cannot mean “outside resource governance.”
The design also explains why Kata creates a per-sandbox child even in environments that do not have a Kubernetes-style pod hierarchy. Docker, for example, does not have the same pod abstraction, yet Kata still needs to associate a sandbox with a single-container use case. A distinct sandbox group gives the runtime an intermediate identity that can be mapped to the workload. The implementation detail matters here only because it illustrates the larger principle: accounting needs an ownership boundary that survives differences between orchestration models.
The two modes therefore answer different questions:
- The fully accounted mode asks, “What is the total host cost of this pod and its sandbox, and can one parent limit govern it?”
- The separate-overhead mode asks, “What resource budget belongs to the application, and how should the infrastructure owner govern the VM machinery alongside it?”
Neither question is inherently more correct in every deployment. The first offers a simpler relationship between pod limits, statistics, and host isolation when the overhead estimate is sound. The second can preserve application capacity when the pod budget intentionally excludes sandbox overhead, but it requires active governance of a second boundary. The dangerous state is not choosing one mode over the other; it is choosing a mode without matching the orchestrator’s sizing assumptions and without observing the resulting host consumption.
Deliberate separation is not the same as mis-accounting
This distinction matters for the resource-accounting problem examined by the USENIX study. A deliberately separate overhead cgroup is a policy decision. The owner knows that certain Kata-created processes are outside the application’s pod budget and can, at least in principle, monitor or constrain that group. A first-touch accounting gap is different: a guest workload may cause a host backend to populate file-backed pages, and the host kernel may charge those pages to the backend because that process first made them resident. The workload’s use and the host’s charge diverge through the operation’s execution path.
The two cases can interact. If a host-side service is placed in an overhead hierarchy, its memory may appear in that hierarchy by design. If the service is outside the expected hierarchy, or if the accounting system cannot associate the resulting pages with the intended tenant, the owner loses a more fundamental attribution signal. Moving a process into a cgroup can improve containment, but it does not automatically explain every page or storage object that the process touches. Conversely, a correct cgroup placement cannot compensate for a storage observer that never sees the object it is meant to charge.
This is why “the container escaped its limit” is too vague to guide a fix. At least three different failures can produce that appearance:
- Sizing failure: the pod limit excludes legitimate sandbox overhead.
- Placement failure: the overhead is placed in a group that nobody constrains or attributes to the tenant.
- Observation failure: a host process or storage object performs work on behalf of the tenant, but the relevant accounting mechanism does not connect it to the tenant’s usage.
The first two are visible in the cgroup design itself. The third requires tracing how an operation creates a page or retains storage. The study’s later memory findings concern that third category, even though the host-cgroup arrangement determines where some of the resulting charges may appear.
The difference also changes what a good statistic means. In the fully accounted mode, a high pod memory reading may indicate that the pod’s application and sandbox together are consuming the budget the orchestrator promised to enforce. In the separate-overhead mode, a low pod reading alongside high overhead-group usage may be expected, but only if the overhead group is monitored and governed. In a delegated-I/O case, low pod usage alongside growing host-backend memory may instead indicate that the observer has lost the relationship between the guest action and the physical page.
Kata’s broader limitations documentation makes a related point without claiming the specific attack findings. In a VM-based system, a constraint may need to be applied inside the guest, to the container inside that guest, and outside the VM to the hypervisor process or the set of host processes it controls. A single number at one layer cannot automatically express the behavior of all the others. Resource governance is a system of linked controls, not a single switch.
Once process placement is explicit, the next question becomes sharper. When a guest file operation causes a host backend to populate memory, is the resulting page charged to the workload, to a deliberately separate sandbox group, or to a host process that the accounting model never associated with the workload? That question leads from cgroup topology to the lifecycle of an individual page.