What Online SDC Detection Can—and Cannot—Guarantee
The opening question of this series was whether a large-scale LLM training system can trust a clean loss curve or a passing preflight test when a hardware fault may produce a wrong result without an alarm. AEGIS gives a concrete answer: correctness needs an online observation layer that sees the real computation, preserves compact evidence, and sends suspicious work to a stronger verifier. This final synthesis connects that architecture to the production measurements and fault cases, then draws the boundary around the claim. The result is a reason to deploy online detection as part of a reliability stack, not a promise that one system can certify every operation.
Detection in production means more than finding an anomaly
A preflight diagnostic and an online detector answer different questions because they observe different moments. NVIDIA’s DCGM GPU Memory plugin tests a bounded property: it writes known patterns to framebuffer memory, reads them back, and checks for mismatches or uncorrectable ECC conditions. The documented test allocates 75 percent of GPU memory by default, requires appropriate prerequisites, and can also test L1-cache integrity on supported devices when enabled. A successful result is meaningful within that scope. A skipped, disabled, interrupted, or incomplete test does not establish memory health, and even a completed memory test does not exercise every compute unit, operator, input range, or distributed interaction in an LLM training run.
AEGIS addresses the unobserved moment: the production computation itself. The paper’s observations show why that moment matters. An SDC may appear only occasionally on the same device, only under particular numerical inputs, and in different computations from one incident to another. A test that never reaches the relevant regime can pass without disproving the existence of a fault. A detector that watches only one operator can miss an error in another. Online observation is therefore necessary when the risk is conditional on the actual execution.
But online does not mean conclusive. AEGIS separates the property of relevance from the property of certainty. cSensor runs close to a protected operation and applies a lightweight, detector-specific check. A mixed-precision checksum difference or a self-equivalence fingerprint is evidence tied to the computation that just ran. If the signal is suspicious, cSensor retains the compact context needed later and emits a vTask. cVerifier then performs the stronger comparison or replay away from the critical path. Only a confirmed result becomes an SDC report.
That separation answers a common reliability dilemma. If every anomaly stopped training immediately, low-precision round-off and other benign variation could cause costly false positives. If the system waited for a visible loss spike or NaN, it would surrender the chance to catch silent damage before it spread. AEGIS uses the critical path for relevance and the deferred path for certainty. The architecture does not remove the cost of verification; it concentrates that cost on selected evidence and tries to place it in available slack.
The two sensor families make the design more than a queue around one test. Mixed-precision-aware algorithmic detection uses equivalent Matmul checksum paths or the FlashAttention backward invariant to produce a compact suspicion signal. Its strength is that it can inspect a computation without requiring a naturally repeated copy. Its weakness is that approximate arithmetic creates a background of ordinary differences, so the signal needs tolerance and later confirmation.
Self-equivalence-based deterministic detection uses redundancy that training already creates for memory or kernel efficiency. FlashAttention recomputes intermediates in backward, and activation recomputation repeats framework-level work. When the same segment runs with the same inputs and configuration under deterministic execution, its outputs should be bitwise identical. AEGIS records xorsum fingerprints and compares them later. Its evidence can be sharper than a tolerance-based checksum, but only where the self-equivalence premise is valid.
These methods complement rather than merge. An algorithmic invariant may protect a compute-intensive operation that has no duplicate execution. A deterministic fingerprint may protect a recomputed segment without adding a new full replay. Their signals can share the vTask and cVerifier workflow, but the meaning of a checksum threshold crossing is not the same as the meaning of a mismatch between valid self-equivalent fingerprints. AEGIS gains coverage by preserving that difference.
Memory and scheduling complete the logic. A sensor must save evidence before the framework releases or overwrites the relevant state. A verifier must run without creating a new synchronization stall or an unbounded stream of deferred work.