The Hidden Cross-Boundary Effect: How Membars Become a Sensor
The central question for NVIDIA Multi-Instance GPU (MIG) is no longer whether its L2 cache can be divided into separate slices; the preceding architecture shows that it can. The harder question is whether an operation that coordinates memory can still affect what another instance observes. This chapter follows the evidence for that question: first the paper's profiler-based discovery of cross-instance memory barriers, then the timing experiment that turns the effect into a sensor. The distinction matters because a timing sensor does not need access to a neighboring tenant's data.
When ordering reaches farther than storage
A memory barrier, often shortened to membar, is an ordering operation. It does not primarily say “fetch this value” or “evict that cache line.” It says that memory operations issued before the barrier must complete within a defined scope before later operations are allowed to proceed. Programmers use ordering when the sequence of memory visibility matters—for example, when one operation publishes data and another operation must not observe a later state before the earlier state is complete.
That purpose creates a different kind of hardware work from an ordinary load or store. The processor has to track which earlier operations are still relevant, determine the scope in which they must be complete, and hold or coordinate later work accordingly. On NVIDIA GPUs, the SASS instruction set includes MEMBAR with scope qualifiers such as CTA, GPU, and SYS. A CTA scope is local to a thread block; GPU and system scopes cover wider execution domains. The paper also found SM and VC scope values in compiled instructions and reports that they behave, respectively, like the narrower CTA and broader GPU cases in its experiments.
The scope is not a cosmetic suffix. A narrow barrier can be satisfied by coordination within a local part of the GPU, while a GPU-wide or system-wide barrier may require activity in the L2-level memory system. That is why two barriers that look similar in a program can leave different traces outside the issuing thread block. The researchers observed no cross-instance profiler effect from a stressor that repeatedly issued MEMBAR.CTA, but did observe such effects for repeated MEMBAR.GPU and MEMBAR.SYS. The result supports a bounded conclusion: the broader-scope operations tested in this design can engage behavior visible beyond one MIG instance, while the narrower operation did not show that behavior in the same measurement.
The paper first uncovered this difference with a controlled detector and stressor. The detector ran in one MIG instance and executed a simple kernel designed to produce very few L2 requests of its own. A stressor ran in another instance. The researchers varied both direct memory instructions and ordinary activities that pass through the CUDA driver, then used NVIDIA Nsight Compute to inspect the detector's L2-related performance metrics.
The control cases were important. When the stressor repeatedly performed user-level LD, ST, or atomic operations, the detector's profiler results showed no observable change attributable to those activities. Driver-level actions produced a different result. Three classes of behavior substantially increased the L2 requests reported while the detector was being profiled:
- launching CUDA kernels;
- calling tested memory-management APIs such as
cudaFree,cudaMemcpy, andcudaMemset; and - creating or destroying CUDA contexts.
The finding is not that these API names are magical or that every call in the CUDA ecosystem has the same behavior. They are the activities the paper tested and found to generate the relevant effect. The authors then compared them with direct barrier loops to identify the common mechanism. Kernel launches produced both GPU-wide and system-wide membar requests in the profiler results. cudaFree produced system-wide requests in that measurement. The apparent number of barriers also depended on how frequently the activity occurred and, initially, on how many threads were issuing it. Increasing activity density increased the effect until available hardware resources limited further growth.
This experiment exposes hidden coupling between layers. A user thinks of a kernel launch as a request to execute a function on the GPU. The driver and hardware may also perform memory-ordering work around that request. A user thinks of freeing or copying a buffer as a memory-management operation. The memory system may process a barrier with a wider scope as part of that path. The exact internal sequence is not fully documented in the supplied material, but the measured relationship is clear enough for the security argument: ordinary activities can be associated with L2-level membars, and those membars can influence observations in another instance.
The sequence separates cause, mediator, and observation. The victim-side activity is the trigger. The L2-level ordering machinery is the mediator. The profiler or receiver measures the consequence. Nothing in the sequence requires the receiver to read a cache line belonging to the victim. That is the key difference between this path and an eviction attack.
The profiler results also need careful interpretation. Nsight Compute was configured to monitor the detector kernel, not the stressor. The authors hypothesize—and support with direct-instruction experiments—that the hardware counters captured the L2-level operations generated by the stressor but attributed them to whichever kernel was being profiled at the time. In other words, the extra count is evidence that the detector's observation environment changed, not evidence that the detector itself executed all of those membars.
That distinction is more than a bookkeeping detail. If a counter is misattributed, a naive reader might conclude that the detector's own code is unexpectedly issuing barriers. The more accurate conclusion is that the profiler is seeing a cross-instance effect through a counter associated with the detector. Counters remain useful for discovering the effect because they reveal a repeatable difference between “stressor absent” and “stressor present.” They are not, by themselves, a practical tenant-level attack interface.
From a privileged counter to a timing sensor
The paper's next step follows directly from the limitation of profiler evidence. Performance profilers generally require privileged access, and security-sensitive environments may disable them. If a separate tenant cannot use the official profiler, the cross-instance effect matters only if an ordinary workload can detect it through an interface available to that workload. The researchers therefore measured the time taken by memory instructions in the receiver instance while a stressor generated membars elsewhere.
The test changed the detector from a counter-reading tool into a workload that measured its own operations. Threads in the receiver repeatedly performed loads or stores against different addresses and recorded execution-time samples. The stressor, in the other instance, repeatedly issued GPU-wide or system-wide membars, first directly and then indirectly through common GPU activities. The authors varied the number of threads in both workloads and sometimes flushed the receiver's L1 or L2 caches to determine which level of the hierarchy was involved.
The timing effect was conditional rather than a universal slowdown. With only a small amount of receiver-side parallelism, the difference was difficult to observe. When the detector issued a large number of loads or stores at once—about a thousand threads in the reported experiments—the stressor increased their average execution time. The receiver's parallel requests create enough samples and enough pressure on the relevant hardware path for a small per-operation effect to emerge. The configuration is therefore part of the signal: an attacker needs a detector that exposes the interaction, not merely any arbitrary kernel.
The density of the stressor mattered as well. More stressor threads initially produced a larger timing increase because they could issue membars more frequently in parallel. The effect then reached a plateau; adding threads beyond the available processing capacity did not continue increasing it. This is a useful systems clue. The signal depends on a finite service path, so it has a saturation point. It also means that a measured percentage should not be read as a fixed performance tax imposed on every tenant. It is the average difference under a particular detector and stressor configuration.
The paper reports that ordinary loads were more promising than ordinary stores, but individual load-latency samples were unstable. In a trace of repeated measurements, the averages with and without a stressor were different while many individual samples overlapped. An observer that classified one sample at a time would therefore make mistakes. The effect was visible statistically, but statistical visibility is not the same as a clean communication signal.
The researchers then used NVIDIA's LD.STRONG.GPU instruction. The supplied material says that NVIDIA does not fully document the exact semantics of this instruction, but the authors infer stronger ordering behavior from their experiments and from how certain ordered memory operations are compiled. That uncertainty should remain attached to the result. The instruction is not introduced as a fully specified public interface; it is a particularly useful measurement operation discovered through reverse engineering and testing.
Its practical advantage was stability. Compared with ordinary LD, the latency samples for LD.STRONG.GPU showed a much clearer separation when the other instance generated membars. The paper reports observable increases in strong-load latency for kernel launches and the tested memory-management activities, while the corresponding strong-store effects were below the useful threshold in its tables. The exact increases were modest—roughly half to seven-tenths of one percent for the strong-load cases in the reported activity tests—but the samples were separated more reliably than ordinary-load samples. A small and stable signal can be more useful than a larger but noisy one.
The result illustrates a general measurement principle without requiring a general-purpose attack recipe. The observer should choose an operation whose own execution is sensitive to the event of interest. Here, the event is a membar issued in another MIG instance, and the sensitive operation is a parallel strong load. The receiver never asks the victim for a value. It asks its own instruction how long it took, then uses repeated measurements to infer whether the neighboring activity was present.
The distinction between profiler and timing detector also clarifies the security boundary. The profiler sees a named class of hardware request but usually needs elevated access and may assign the event to the wrong kernel. The timing detector sees only a change in its own execution time and can operate under the paper's unprivileged threat model, assuming access to a user-level GPU clock such as CLOCK. That makes the latter more relevant to a tenant who cannot inspect performance counters.
The paper's experiments do not imply that every receiver can observe the effect equally well. The timing increase depends on the receiver's parallelism, the stressor's membar density, whether data hits in L2 or misses, and the amount of unrelated activity. The authors found that L2 hits showed a larger timing increase than misses in the main configuration. Other memory-related instructions, including atomic operations and barriers themselves, also showed small average changes, but their individual samples were too unstable to serve as reliable detectors in the tested setup.
This is why the word “sensor” is more accurate than “leak” at this stage. The mechanism creates an observable channel, but the channel still needs a source with a meaningful pattern and a receiver with enough signal-to-noise ratio. The sensor reports activity indirectly. What that activity means depends on the victim's workload: a burst of kernel launches may correspond to one phase of an application, a quiet interval to another, and a repeated sequence to a recognizable model or algorithm. The next question is how much information those patterns can carry.
The evidence up to this point supports a precise conclusion. MIG's L2 partition prevents the tested cross-instance cache-eviction route, but it does not prevent every L2-level ordering effect from influencing another instance's timing. The effect was first visible through a privileged profiler and then through an unprivileged, timing-based detector built around LD.STRONG.GPU. That is not arbitrary data access. It is nevertheless enough to turn the rhythm of another workload into something an observer can measure.