Why Large-Scale LLM Training Needs an Online Correctness Layer
This series examines AEGIS, an online silent-data-corruption detection framework built for large-scale large language model training. The opening question is simple but consequential: how can a training system notice a wrong computation that produces no alarm, appears only under particular inputs, and may be buried inside weeks of distributed work? The answer will require more than a definition of GPU faults; it requires a way to distinguish what a test has actually established from what the live training run still leaves unknown.
A bad result can be silent
A silent data corruption, or SDC, is an incorrect result produced by hardware without a hardware or software alarm. That last condition is what makes the problem different from an ordinary crash. A failed kernel, an ECC alert, or a NaN loss gives the system a visible event around which it can stop, diagnose, and recover. An SDC may instead return a plausible-looking value and let the training process continue.
The distinction matters because a large language model is not trained by one isolated calculation. Modern jobs distribute matrix multiplications, attention, and other operations across many GPUs, often running for weeks or months. A single incorrect intermediate can be consumed by later operators, passed through collective communication, and incorporated into parameter updates. The result may be a degraded model, a wasted run, or a failure that appears much later than the computation that caused it. At that point, the loss curve is evidence about the outcome, not a reliable record of every operation that produced it.
The scale also changes the meaning of “rare.” The paper behind AEGIS describes a training incident involving around 10,000 GPUs in which gradient norms and loss values showed pronounced changes. That example is not a universal failure pattern: many SDCs do not create a dramatic spike. It does show why a low-frequency event can become operationally important when the system performs an extraordinary number of operations and when a corrupted value can spread across a distributed job.
A useful first mental model is therefore not “the GPU is either healthy or broken.” It is a chain of evidence. A test exercises some memory, instruction, or workload and observes a result. The live training job exercises a much larger and changing space of computations. A monitoring layer observes that real execution, produces a suspicion when it sees something unusual, and seeks stronger confirmation before turning the suspicion into an incident.
This map is the central relationship in AEGIS. The sensor is close to the computation because that is where the relevant inputs and outputs are visible. The verifier is separated from the critical path because certainty may require replay or comparison. The distinction between those two stages is not a cosmetic software split: it is the way AEGIS tries to make correctness checks compatible with a job whose primary purpose is still to train a model.
What a memory diagnostic can tell you
Offline diagnostics remain useful, but their result must be read at the right scope. NVIDIA’s DCGM GPU Memory plugin is a clear example. It tests data integrity in framebuffer memory and, when enabled on a supported device, in the L1 cache. For the framebuffer test, it allocates a significant portion of GPU memory—75 percent by default—writes known patterns with CUDA kernels, reads the values back, and checks for mismatches or uncorrectable ECC conditions. The documentation also makes clear that this is not a memory-bandwidth test.
That is a meaningful contract. If a value read from framebuffer memory differs from the value written, the plugin can report a memory mismatch. If CUDA reports an uncorrectable double-bit ECC error, the plugin can report that condition. But the test needs the expected ECC capability and state, enough free memory, and permission to run. A disabled or unsupported test, an allocation failure, or an API and device error means the test did not complete or did not produce a memory-integrity result. It is not equivalent to a certificate that the GPU will behave correctly in every workload.
The same logic applies to broader stop-time diagnostic suites. They exercise selected patterns under controlled conditions, often while training is interrupted. A passing result says that the device behaved correctly for those operations, inputs, and timing conditions. It does not expose a fault that manifests only in a production Matmul shape, a particular numerical range, or a computation that the diagnostic never performs. The paper describes prior production experience in which offline suites can take hours and still miss non-deterministic SDCs. Those observations concern the limits of coverage, not the uselessness of diagnostics.
Other alternatives exchange a different resource. Rerunning a deterministic workload can provide strong confirmation by comparing executions, but at LLM-training scale it duplicates substantial computation. Classical algorithm-based checks can run online, but a checksum difference in low-precision arithmetic may be caused by ordinary round-off rather than corruption. If the threshold is too sensitive, the system creates false positives and too much verification work; if it is too forgiving, it may miss a small fault.
The gap is consequently not one missing test parameter. It is a mismatch between the evidence a preflight or generic check can collect and the evidence needed to judge the exact computation that matters. The paper’s observations make that mismatch concrete. On a GPU where an SDC had already been observed, 50 identical nanoGPT runs produced only one divergent loss. In a separate Matmul experiment, 9.2 × 10^5 replays produced three distinct erroneous outcomes with an estimated overall error rate of 2 × 10^-9. The fault was present, but its manifestation was probabilistic.
The inputs mattered too. Scaling the operands changed the range in which the SDC appeared; other numerical ranges were unaffected. A diagnostic that samples one regime can therefore pass while a training run later reaches another. Finally, different incidents surfaced in different computations and left different observable evidence. These properties explain why AEGIS does not rely on a single universal checker.
The missing layer: AEGIS
AEGIS adds an online correctness layer between the live computation and the platform’s operational response. Its design has two deliberate asymmetries. First, sensing is lightweight and inline: it must inspect the real execution without turning every operation into a second full computation. Second, verification is definitive but deferred: it may replay a suspicious computation or compare outputs from executions that training already performs, so it is moved away from the critical path when possible.
The inline component is called cSensor. After a protected computation, it applies a detector suited to that computation and retains compact evidence along with the minimum context needed later. A suspicious signal does not immediately become an incident. Instead, cSensor packages the signal as a verification task, or vTask. The vTask is the common handoff between different sensors and the verifier. One task may carry a checksum difference and replay context; another may carry fingerprints from two self-equivalent executions. The shared interface lets AEGIS buffer and schedule both without pretending that their evidence has the same meaning.
The deferred component is cVerifier. It consumes queued vTasks and applies the stronger test appropriate to each one. For a deterministic comparison, it checks whether fingerprints match. For an algorithmic signal, it replays the suspicious computation using the saved context and compares the result with the recorded evidence. Only after this step does AEGIS report a confirmed SDC. Numerical noise can therefore produce a suspicion without becoming a false incident, provided the verifier has enough evidence to make the comparison.
This arrangement also explains why evidence must be captured early. Large-model training aggressively releases or overwrites intermediate tensors to manage memory. If cSensor waits until a later diagnosis to ask for an input that no longer exists, a cheap signal may be impossible to confirm. AEGIS instead retains a small, detector-specific payload before the training framework releases the larger state. The memory cost is part of the correctness design.
Confirmation then has to fit the schedule of a running job. In pipeline-parallel training, workers periodically wait for tensors from other stages, creating pipeline bubbles. AEGIS uses those naturally occurring idle periods to process vTasks and reserves a small fixed slice at the end of a step for work that remains. This is a best-effort policy, not a claim that verification is free or instantaneous. It is a way to keep expensive confirmation from introducing a new synchronization stall while bounding the amount of deferred work competing with training.
The final output is consequently more useful than a raw alert. A confirmed report includes the associated device and execution context, allowing the platform to isolate a faulty GPU and restart or roll back from a checkpoint. In an SDC incident, detection without a path to containment would only reveal that the training run was at risk. AEGIS links observation to a decision about whether the device should continue participating.
The strongest claim supported by this opening is therefore also a bounded one. AEGIS does not prove that every GPU computation is correct. It responds to a specific gap: rare, silent, workload-sensitive faults can escape tests that do not see the live training regime. By sensing in place, preserving minimal evidence, and verifying off the critical path, it turns that gap into something the training platform can investigate and act on. The next question is how the system can tell a real fault from the ordinary numerical noise of modern low-precision training.