Overview

HeMem uses per-hugepage NEMO counts to detect a changed hot set in the slow CXL tier, while MEMTIS uses basepage counts inside selected 2 MiB huge pages to find split candidates. In the reported FlexKVS setup, recovery was 67 seconds versus 324 seconds for default 0.02% PEBS at 0.89% CPU overhead; MEMTIS-NEMO identified more than twice as many skewed candidates and detected them up to 10.4× faster, with 13% higher Silo throughput at a 1:8 ratio. Results depend on workloads, mappings, schedule, and prototype.

Learning outcomes

HeMem benefits from timely per-hugepage counts when a disjoint hot set replaces the old one; migration bandwidth still bounds recovery.
MEMTIS needs basepage distribution, not only a hugepage total, to identify split candidates as fast-tier capacity tightens.
Reported gains are conditional: NEMO observes the slow CXL tier while PEBS remains for local DRAM, and time-multiplexing limits fine-grained coverage.

Keywords

NEMOHeMemMEMTISPEBSHugepagesTiering

When Counters Become Decisions: HeMem and MEMTIS

NEMO's mechanism matters only if its counters change what a memory-management policy can do. Two experiments make that relationship concrete: HeMem uses per-hugepage telemetry to recover from a changed hot set, while MEMTIS uses counters inside huge pages to decide which regions are worth splitting. Together they show the difference between a cleaner measurement and a different control loop, while keeping the reported gains tied to their workloads, baselines, and CXL prototype.

Hot-set recovery: seeing the change before the policy falls behind

HeMem is a userspace memory-tiering system that uses access evidence to decide which pages should occupy a faster tier. The experiment creates a deliberately difficult transition: a set of initially hot keys becomes cold, and a disjoint set becomes hot. The working set spans fast and slow memory, so HeMem must recognize the new pattern and migrate pages rather than simply continue serving the old placement.

The baseline uses Intel PEBS to sample LLC misses at a 0.02% rate. Every second, HeMem processes the new samples, locates the associated page metadata, updates access counts, and uses those counts to guide migration. This is a coherent policy loop, but its response is bounded by the probability that the new hot pages produce enough samples and by the CPU work needed to process them. Lowering the rate protects the application but delays recognition; raising it provides more evidence at a cost that eventually harms the workload.

The NEMO version changes the observation step, not the policy's basic purpose. For the slow CXL tier, NEMO maintains a counter for each tracked huge page on every relevant LLC miss. HeMem then reads the counters and updates the same page-level access information it already uses. In the prototype, PEBS remains in place for the fast tier, so this is not a claim that NEMO replaced every observation path. It is a targeted offload of the slow-tier signal where sampling was especially costly and incomplete.

sequenceDiagram participant W as Workload participant T as NEMO or PEBS participant H as HeMem or MEMTIS participant M as Fast and slow tiers W->>M: Change accesses or concentrate within a huge page M->>T: Produce telemetry signal T-->>H: Return sampled or counted evidence alt hot-set phase change H->>M: Migrate newly hot pages else within-page skew H->>M: Split and place hot base pages end M-->>W: Change service performance and capacity use

In the FlexKVS hot-set experiment, the NEMO-backed system recovered to steady state in 67 seconds, compared with 324 seconds for the PEBS-backed system at the default sampling rate. The roughly fivefold difference is not the result of a new migration algorithm. The policy still has to move data, and the paper says the NEMO result is bounded by migration bandwidth. The change is that HeMem receives a sufficiently complete page-level signal sooner, so it can begin moving the new hot set rather than waiting for samples to accumulate.

Published by DeepMox

FAQ

How did NEMO change HeMem's hot-set recovery?

In the reported FlexKVS experiment, NEMO helped HeMem recover in 67 seconds versus 324 seconds with default 0.02% PEBS; the policy and migration step stayed the same.

Why does MEMTIS need basepage telemetry inside a huge page?

MEMTIS needs basepage telemetry because a 2 MiB total cannot distinguish activity spread across the region from activity concentrated in a few hot 4 KiB pages, which determines whether splitting saves fast-tier capacity.

What limits the NEMO results in the HeMem and MEMTIS experiments?

The results are limited to named workloads, baselines, capacity ratios, time-multiplexed mappings, and a CXL-attached prototype; PEBS still covers local DRAM and migration or splitting remains.

Previous10 minNEMO's Programmable Controller Pipeline

Next11 minNEMO's CXL Isolation and Scaling Limits

NEMO in HeMem and MEMTIS Tiering

When Counters Become Decisions: HeMem and MEMTIS

Hot-set recovery: seeing the change before the policy falls behind

FAQ