Overview

This synthesis asks when controller-side telemetry is worth adding to CXL tiering. NEMO can improve placement, huge-page splitting, and tenant isolation when bounded aggregation matches the policy's unit and timing, but Linux still supplies the tier and software still performs migration, splitting, or throttling. The evidence covers named workloads and an Altera Agilex 7 CXL 2.0 Type-3 FPGA prototype; finite SRAM, revisit time, expressiveness, actuation, and ownership boundaries limit generalization.

Learning outcomes

NEMO is a candidate when a policy needs high-rate page, sub-page, or tenant evidence that existing paths cannot provide affordably.
Its value is end-to-end: telemetry can shorten detection, but migration, splitting, cgroup actuation, and policy logic remain.
The evidence is promising but bounded by the named workloads, FPGA prototype, finite state, revisit time, and security assumptions.

Keywords

NEMOCXLTelemetryTieringSRAMActuation

What Better Hardware Telemetry Can—and Cannot—Change

The question running through this series is whether CXL memory tiering can make good decisions without a sufficiently good view of memory behavior. The evidence about NEMO supports a conditional answer: controller-side telemetry can materially improve placement, page splitting, and tenant isolation when its aggregation matches the policy, but it cannot replace Linux's capacity plumbing, the policy's actuation, or the resource needed to hold and revisit state. The final judgment is therefore about fit, not a universal victory for hardware over software.

Capacity creates the choice; telemetry makes it usable

CXL changes the memory hierarchy by making additional capacity available at a location whose service characteristics can differ from local DRAM. Linux turns that physical fact into a managed resource through memory maps, ACPI locality, NUMA nodes, CXL decoder associations, DAX, and, when selected, memory hotplug into the page allocator. That lifecycle answers a necessary question: what memory exists, how can it be reached, and which subsystem owns it?

It does not answer the runtime question: which data should use which tier now? A correctly configured CXL region can still receive a poor placement if the system cannot tell which pages are active. A huge page can be online and valid while wasting fast-tier capacity because only a few of its base pages are hot. A tenant can have a legitimate allocation while its bandwidth bursts damage another tenant's tail latency. These are not failures of enumeration. They are failures of the feedback loop that turns access behavior into a policy action.

NEMO's role is to connect those loops without confusing them. It observes request metadata beside the memory data path, maintains state in controller-adjacent SRAM, and exposes a driver-controlled view to the OS. Linux still decides what a CXL region means to the page allocator. HeMem or MEMTIS still decides how to migrate or split data. A cgroup controller still performs the throttle. NEMO changes the evidence that reaches those decisions, especially when CPU-side sampling or coarse hardware counters cannot preserve the relevant coverage and attribution.

The five dimensions from the NEMO paper make the relationship precise. Coverage matters because a tiering policy must see the traffic that consumes the tier, including traffic that a cache-miss sample may omit. Timeliness matters because a hot set or bandwidth burst can change before a periodic scan catches up. Granularity matters because a socket total cannot identify a page, and a huge-page total cannot reveal skew inside the page. Flexibility matters because hot-set, page-splitting, and tenant policies need different groupings. Overhead matters because observing a policy cannot consume the CPU or memory bandwidth that the policy is meant to protect.

NEMO's controller-side position addresses coverage and host overhead together, while its OS-defined translation addresses meaning. That division is the central architectural contribution. The hardware sees the stream at the point where requests are served; the driver and OS decide whether the stream should become one counter per selected huge page, many counters inside a huge page, or one counter for a tenant's pages. The same request traffic can support different policies because the state mapping is not frozen to one vendor-defined statistic.

That does not make the two loops independent. NEMO can observe only regions that the underlying system has correctly exposed and that the driver is permitted to map. It cannot repair a CXL decoder association, online an unavailable memory block, or grant a policy the authority to move a DAX resource. Conversely, Linux can provide a valid memory tier but cannot infer a current hot set from topology alone. The useful design keeps both facts visible: capacity plumbing establishes what can happen, and telemetry determines what should happen next.

Three cases, three reasons the signal matters

The strongest evidence for NEMO is not one headline number but the way three policies use different telemetry shapes.

Published by DeepMox

FAQ

What should a platform validate before adopting NEMO?

Before adopting NEMO, a platform should validate the policy's required unit and timing, available SRAM and revisit time, ownership controls, actuator delay, workload baseline, and prototype-to-production gap.

What can NEMO not change in a CXL tiering system?

NEMO cannot make a slow tier local, create infinite SRAM, run arbitrary analytics at line rate, or remove migration and actuation time.

Why might a hybrid telemetry design be preferable to NEMO alone?

A hybrid design may be preferable because the evaluations use PEBS for the fast tier and NEMO for slow CXL memory, while one engine may not cover every tier or granularity.

Previous11 minNEMO's CXL Isolation and Scaling Limits

All chaptersBack to Topic

NEMO: CXL Tiering Fit and Limits

What Better Hardware Telemetry Can—and Cannot—Change

Capacity creates the choice; telemetry makes it usable

Three cases, three reasons the signal matters

FAQ