How Linux Turns a CXL Device into a Memory Tier
NEMO does not create a CXL memory tier. Linux first has to describe the device, place it in the system's memory topology, and expose its capacity through a usable interface. This chapter follows that lifecycle so the boundary around NEMO is clear: Linux makes the region available to the operating system, while NEMO later supplies access signals for policies deciding how that region should be used. The distinction matters because a device can be correctly configured and still be poorly managed if the system cannot see what applications are doing with it.
From a firmware description to a Linux-visible region
A CXL Type-3 memory device enters Linux as more than a block of anonymous capacity. At early boot, firmware and boot configuration determine which parts of the physical address space are available and which are reserved for special handling. The Linux documentation describes CXL memory being set aside through the EFI or E820 memory map and represented as a soft-reserved I/O resource. This is the first ownership transition: the address range is identified as something the CXL subsystem must claim, rather than as ordinary RAM that the general allocator can immediately consume.
The next transition is from a physical range to a locality model. Linux consults ACPI information, including CXL fixed memory window descriptions in the CEDT and proximity domains in the SRAT. Those descriptions allow the kernel to create NUMA nodes associated with the CXL region. NUMA is a way of representing that memory is not equally close to every processor; a node is not simply a device name, but a locality label that later code can use when choosing where to allocate or move pages.
Linux also creates a default memory tier containing the available nodes. That does not mean every placement decision has already been made. It means the kernel has a structure in which nodes can be related as memory resources with different characteristics. A memory tier is therefore a policy-facing abstraction built on top of physical topology. It gives later management code a way to reason about placement without needing to rediscover how the platform's address windows and proximity domains were wired.
After boot has established the memory map and topology, the ACPI and PCI layers identify the device as CXL-capable and arrange for the CXL driver to probe it. The CXL subsystem builds relationships among the device's components: roots and ports represent the path into the fabric, a memory device represents the endpoint, and decoders describe how address ranges are routed. A region becomes meaningful only when those pieces agree about which physical addresses belong to it and how they are associated.
That association can come from firmware-programmed decoders or be built when a user configures and commits them. In the first case, the driver validates the existing configuration and locks the relationship at probe time. In the second, validation and association occur when the decoder configuration is committed. The important point is not the vocabulary of each object; it is that Linux must establish a consistent mapping before it can safely expose the region. A memory range without a valid route would be capacity that the allocator could name but could not reliably serve.
Once the CXL driver has a valid region, it surfaces that region to the DAX layer. DAX is the interface through which this kind of memory can remain directly addressable as a device or be converted into ordinary memory managed by the page allocator. The CXL path has reached a second boundary: the region is no longer just a firmware-described address window, but Linux-managed storage with a selected access model.
The diagram's last arrow is deliberately separated from the earlier transitions. NEMO is not the component that discovers the CXL endpoint, creates a decoder, or decides whether a region is DAX hmem or kmem. Those operations establish what memory exists and how Linux can reach it. NEMO's role begins at a different observation boundary: requests are already flowing through a memory controller, and the operating system needs a timely account of their behavior.
Two ways the region can meet applications
The DAX driver can leave the region in hmem mode. In that mode, the region remains a DAX device that applications can access as a file-backed or device-like memory resource. The lifecycle stops at the DAX boundary because the region is not being presented as ordinary pages in the general page allocator. This path preserves a direct device abstraction, which can be appropriate when the consumer is designed to use DAX semantics.
The other mode is kmem. Here the DAX device is converted into hotplug memory, and the memory-hotplug subsystem represents the region as a set of memory blocks. Those blocks are linked to a NUMA node and brought online into a requested zone, such as NORMAL or MOVABLE. Once online, they participate in the page allocator's normal work: pages can be allocated, reclaimed, migrated, and considered by memory-management policies. The conversion changes who owns the next decision. Instead of an application opening a DAX device directly, the kernel's page-management machinery can place ordinary allocations in the CXL-backed region.
The documentation marks that memory as “driver managed.” This is a practical ownership rule: the region should not be treated as ordinary memory for unrelated operations such as using it as a kernel-update area during kexec. The label does not mean that the driver performs all future placement decisions. It means the system retains the information that this memory came through a device-managed path and must be handled under the corresponding constraints.
The distinction between hmem and kmem also explains why “CXL memory” is not one single runtime experience. In hmem mode, the access model is centered on a DAX device. In kmem mode, the region becomes part of the page allocator's NUMA-visible pool, so memory policy and tiering code can make decisions about ordinary pages. NEMO's evaluated integrations are about the latter kind of policy problem: observing requests so systems such as HeMem or MEMTIS can decide which data should occupy a faster or slower tier.
A useful way to read the lifecycle is to track which layer owns the next decision. Firmware and ACPI describe the physical range and its relationship to the platform. Linux boot code turns that description into an address-space and NUMA model. The CXL driver owns the validity of the device, decoder, and region associations. DAX chooses whether the region remains a device-like resource or becomes a candidate for memory hotplug. The memory-hotplug subsystem then owns the transition from a region to blocks that can be onlined, while the page allocator and memory policy code own the later use of those blocks. NEMO does not take over any of these ownership changes; it observes requests after the relevant memory-controller path exists.
Each transition has a different success condition. Early boot must reserve the right addresses without accidentally hiding ordinary RAM. Topology creation must associate the region with the right proximity information, or a later placement decision could treat a distant resource as if it were local. Decoder validation must make routing and region membership agree. DAX must expose a mode that matches the intended consumer. Memory hotplug must online blocks into a supported zone and retain their NUMA association. A telemetry engine cannot repair a wrong address range, an invalid decoder route, or a region that was never made visible to the allocator; it can only make the behavior of a successfully exposed path more legible.
The zone choice also carries a policy implication without being a policy decision by itself. A region onlined into NORMAL is available to ordinary page allocation, while MOVABLE is intended for pages that can participate in migration. That distinction affects which later mechanisms can rearrange data when the fast tier is scarce. It is another example of why “the device is present” is not a sufficient description of system state: the same physical capacity can become useful to different parts of Linux depending on how the transition is completed.
The physical and logical labels remain useful after the device is online. A NUMA node tells placement code which locality a page belongs to, while the memory tier expresses that nodes may have different performance roles. The page allocator can therefore place ordinary pages in a CXL-backed node without pretending that the node is equivalent to local DRAM. If a policy later migrates a page, it is moving an allocation between resources that Linux already knows how to name; it is not discovering the topology from the access stream.
That separation also prevents a tempting but misleading shortcut. It would be possible to describe the whole CXL path as “memory hotplug,” but doing so would hide the decisions that occur before blocks appear. The boot memory map determines whether a range is available to the CXL driver. ACPI determines how the range is related to processors and other memory. Decoder associations determine whether requests can be routed to the intended region. Only after those facts are consistent can DAX choose how the region should meet an application or the page allocator. Each layer narrows the set of legal later actions.
For a tiering policy, the consequence is practical. The policy can choose a page based on observed accesses only if that page has a valid address, belongs to a known node, and resides in a resource that the policy is allowed to move. NEMO can improve the evidence about which page is hot or which tenant is noisy, but it does not grant permission to migrate a DAX file, repair an offline block, or override the ownership rules attached to driver-managed memory. The quality of a runtime decision is bounded by both the telemetry signal and the resource model on which the signal acts.
That relationship should not be overstated. The Linux overview documents the capacity exposure and configuration lifecycle; it does not document NEMO or claim that every memory-tier policy follows the same path. The connection is architectural. Linux creates a correctly routed, addressable, policy-visible CXL region. A telemetry mechanism such as NEMO can then observe traffic at the memory-controller boundary and export a view that a policy understands. The two systems cooperate, but they own different state.
The ownership boundary is useful when diagnosing failure. If the firmware memory map or ACPI proximity information is wrong, Linux may build the wrong topology before NEMO has anything meaningful to observe. If decoder associations are invalid, the region cannot safely be surfaced. If a DAX region is not converted to kmem, ordinary page-placement code cannot treat it as a normal allocator resource. Conversely, if the region is visible and correctly online but telemetry is stale or too coarse, the capacity path has succeeded while the policy loop still makes poor decisions.
CXL is valuable precisely because its capacity can be added or arranged in ways that change the memory hierarchy. But that value depends on recognizing the hierarchy after boot, not merely enumerating the hardware once. The paper behind NEMO describes requests whose latency and bandwidth vary with their service location; the Linux path gives the operating system a vocabulary for those locations. It does not, by itself, tell the OS which pages are hot, which huge pages are internally skewed, or which tenant is responsible for bandwidth pressure. Those are runtime facts, and they require a second layer of observation.
This is why the next question is not “how do we configure CXL?” but “what can Linux learn once CXL is configured?” A page-table scan, a performance-monitoring sample, and a memory-controller counter may all observe some part of the same traffic while offering very different coverage, timeliness, and attribution. NEMO enters that comparison as a telemetry design—not as another step in device enumeration—and the quality of the resulting tier depends on that distinction.