Why AEGIS Watches the Real Training Run
AEGIS is built around a counterintuitive diagnosis: a GPU can be faulty even when repeated tests appear correct. This chapter examines why silent data corruption is non-deterministic, sensitive to the workload that happens to run, and diverse in the computations where it appears. The practical judgment is not that offline tests have no value; it is that “the test passed” and “the fault is absent” are claims about different bodies of evidence.
A fault can be present without a reproducible symptom
The first source of confusion is the difference between a fault and its manifestation. A hardware fault may exist continuously while producing an incorrect result only under a narrow set of circumstances. If a test does not enter those circumstances, the test can pass honestly. The result is still a poor basis for declaring the device safe for every later computation.
The AEGIS paper illustrates this with repeated executions of nanoGPT on a GPU where an SDC had already been observed. Under identical settings, 50 runs produced only one divergent loss. A reader who sees the other 49 runs may reasonably conclude that the event is rare. It would be a mistake to conclude that the underlying fault was absent during those runs. The experiment separates two questions that ordinary testing often combines: “Did this execution fail?” and “Can this hardware ever produce a wrong result in the relevant execution space?”
A larger Matmul replay makes the same point in a more extreme form. The authors repeated a kernel 9.2 × 10^5 times and observed only three distinct erroneous outcomes. They estimated an overall error rate of 2 × 10^-9 for that experiment. Such a rate explains why a short stress test can miss a fault without being incorrectly implemented. It also explains why simply repeating the same test a few more times does not automatically convert it into a complete health certificate.
This is not a claim that every device has the same probability or that every SDC is equally difficult to reproduce. The measurements come from the paper’s experiments and studied production environment. Their importance is diagnostic: they establish that observation frequency cannot be used as a direct proxy for fault existence. A low count may mean a healthy device, a rare trigger, or a test that never reached the right conditions. More evidence is needed to tell those possibilities apart.
The trigger belongs to the workload
The second problem is input sensitivity. The paper varies the inputs to a Matmul by introducing scaling factors, effectively changing the numerical regime in which the operation runs. The observed SDC appeared in some ranges and not in others. The same operator and the same device therefore did not produce one fixed answer to the question “Does this computation expose the fault?” The answer depended partly on the values being processed.
That matters for LLM training because its tensors are not arbitrary placeholders. Shapes, values, precision modes, and neighboring operations arise from a changing model and data stream. A diagnostic may exercise a representative kernel but still use values that do not activate the relevant failure condition. Even a test designed to resemble the target workload can cover only a sample of the numerical regimes that production will reach.
Input sensitivity also changes how a passing result should be communicated. “This input did not trigger an error on this run” is a narrow observation. “This GPU is correct” is a much broader conclusion. The first statement is supported by the test; the second requires confidence that the test covers the conditions that matter. AEGIS starts from the premise that the live training computation is not merely another benchmark. It is the environment whose particular inputs and execution path may reveal the fault.
The third problem is diversity. The production analysis found that SDC incidents did not share one uniform signature. One incident may appear in a matrix multiplication, another in an attention computation, another in normalization or a memory path. The evidence may be a checksum mismatch, a disagreement between two deterministic executions, a training anomaly, or a pattern useful for later localization. A checker designed around one signature will naturally be strong only where that signature exists.
These properties interact. A fault can be rare to observe, conditional on numerical values, and visible only through one class of computation. That combination defeats the comforting idea that there is a single standard stress workload through which every bad GPU must eventually reveal itself. It also makes a single online rule insufficient: an algorithmic invariant may cover one operator while missing another, and an exact recomputation comparison may exist only where the training system already repeats work.
This creates a coverage problem with two directions. A test can produce a false negative by never reaching the condition that exposes a fault. It can also produce a false positive when its signal is sensitive to ordinary variation that has nothing to do with corruption. The remedies are not mirror images. More representative workloads and more observation opportunities can improve exposure, while better confirmation evidence is needed to decide whether an unusual result is real. Treating both problems as “the threshold is wrong” hides the system choices involved.
The distinction is especially important for a fleet operator. A device that fails a memory-integrity test deserves immediate attention even if the training workload has not yet shown an error. A device that passes that test may still need observation during production because the test did not cover compute-unit behavior or the numerical conditions of the model. Conversely, a production anomaly should not automatically be assigned to memory: its location may be in computation, framework behavior, or another protected path. Diagnosis begins by preserving the scope of each observation.
The diagram is a diagnosis, not yet a solution. It shows why the evidence routes fail in different ways. Offline testing may never enter the triggering regime. A rerun may sample a rare manifestation too sparsely or consume too much computation. A naive checksum may observe the right operation but confuse ordinary arithmetic noise with a fault. AEGIS’s design must address all three gaps without treating them as the same problem.
Every detection method sees only part of the problem
An offline diagnostic suite is strongest when the question is narrow and the test can exercise it directly. NVIDIA’s DCGM GPU Memory plugin provides a useful example. It writes known patterns into framebuffer memory, reads them back, and checks for mismatches. When enabled on supported hardware, it can also test L1-cache data integrity. The framebuffer test allocates a significant portion of memory—75 percent by default—and the documentation records conditions such as an uncorrectable ECC error or a readback mismatch as failures.
That is valuable evidence about the memory property the plugin actually tested. It can expose a data-integrity fault in the tested region and report when the requested test cannot complete. But it is not a test of memory bandwidth, nor is it a replay of a full LLM training step. It requires suitable ECC capability and state, enough free memory, and permission to run. If the test is skipped because the prerequisites are not met, disabled, or interrupted, the result does not establish memory health. Even a successful run says nothing directly about every compute unit, operator, input range, or distributed interaction in a training job.
The distinction is easy to lose when diagnostic output is summarized as a green or red status. A green status is meaningful only with its scope attached: these patterns, this allocation, this execution, and these prerequisites behaved as expected. A red status can be highly actionable, but it still identifies the property that failed rather than automatically explaining every later symptom. For SDC detection, that scoped evidence is a starting point for fleet hygiene, not a substitute for observing production computation.
The paper places broader offline suites in the same general category. They interrupt training and exercise expert-designed workloads. The paper reports that such suites can miss non-deterministic SDCs, and describes prior production experience in which a single faulty-GPU diagnosis can take more than eight hours and achieve about 70 percent recall. Those figures are contextual observations, not universal performance specifications for every vendor suite or configuration. The underlying limitation is more general: a finite, stop-time test cannot exhaustively reproduce a fault whose trigger depends on the exact workload.
Rerun-based testing addresses a different weakness. If two executions are deterministic and comparable, a disagreement can provide strong confirmation. This is a more direct test of the computation than a generic memory pattern. But at LLM-training scale, rerunning substantial work duplicates GPU cycles and can delay the job. The non-deterministic manifestation problem remains as well. A finite set of reruns can fail to sample a rare event, even while a rerun that does observe one can be compelling evidence. Reruns therefore exchange detection cost for stronger confirmation; they do not remove the need to choose a representative execution.
Algorithm-based online detection tries to observe the real operation without duplicating the entire job. A checksum, for example, compares two mathematically equivalent ways of summarizing a result. In exact arithmetic, a mismatch is a clean warning. Modern LLM training commonly uses low-precision values such as bfloat16, however, and ordinary floating-point round-off can make equivalent paths differ even when the computation is healthy. A threshold that treats every difference as a fault will generate false positives. A threshold that tolerates too much noise may hide a small corruption. The method has not failed because it is online; it has encountered a difficult evidence problem in approximate arithmetic.
An online check also has to balance where it looks against what it can retain. Watching every operation would increase exposure, but it could add enough checksum work and saved context to interfere with training. Watching only the most convenient operations reduces cost, but leaves blind spots precisely where a different fault may appear. The useful unit of design is therefore not a detector in isolation. It is a chain that chooses a computation, captures evidence while that evidence exists, and escalates only signals that a later test can interpret. AEGIS’s contribution begins with this chain, before any particular checksum or fingerprint is selected.
This comparison rules out two tempting but opposite conclusions. The first is that offline diagnostics are enough because they test hardware directly. They test some hardware behavior directly, but SDC risk is conditional on more than memory readback. The second is that every online signal should immediately stop training because it observes the production path. An online signal may be more relevant to the actual job and still be only a suspicion. Relevance and certainty are different properties.
What the diagnosis demands from AEGIS
The three observations lead to a specific design requirement. A detection system must be present when the production computation runs, because only then can it observe the inputs and execution regime that trigger an SDC. It must support more than one sensing mechanism, because incidents can surface in different computations and produce different evidence. And it must confirm a suspicious signal before reporting it, because low-precision arithmetic and other benign variation can make a lightweight check over-sensitive.
AEGIS responds by separating sensing from verification. Its inline sensor can watch a selected computation and record a compact signal; a later verifier can use stronger evidence to decide whether that signal represents corruption. This arrangement does not make the diagnostic, rerun, or checksum categories disappear. It gives their useful properties different jobs. An offline test can remain a preflight screen. A checksum can become a cheap way to select suspicious executions. A replay or comparison can be reserved for the smaller set of cases that need confirmation.
The important shift is from asking whether one test can certify a GPU to asking how several evidence paths can meet around the actual training run. A test pass then has a precise meaning. A checksum threshold crossing has a precise but provisional meaning. A verified mismatch has a stronger operational meaning. Keeping those meanings separate is what allows AEGIS to combine coverage with restraint.
That is why the next system boundary is not another catalog of tests. It is the path by which AEGIS moves a signal from a live operation into a trustworthy decision while preserving the context that the training framework may soon release. The central question becomes architectural: who senses, who owns the evidence, who confirms it, and how can that work occur without turning the training job into a sequence of diagnostic interruptions?