AEGIS: Online SDC Detection for LLM Training

Overview

AEGIS addresses silent, workload-sensitive data corruption in large-scale LLM training by sensing selected live computations, preserving compact context, and deferring stronger replay or self-equivalence checks. The series explains its Matmul and FlashAttention sensors, redundancy-based fingerprints, adaptive sampling and scheduling, and production trade-offs, while keeping offline diagnostics, uninstrumented paths, transient faults, and fleet-specific results within their limits.

Learning outcomes

Silent data corruption can evade offline diagnostics when manifestations are rare, input-sensitive, and workload-specific.
AEGIS separates inline sensing from deferred verification so relevance and certainty are handled at different costs.
Float32-accumulator checksums, FlashAttention invariants, and deterministic fingerprints provide complementary evidence, not universal proof.
Production results show measured value and overhead, but coverage remains bounded by sampling, instrumentation, fault type, and deployment conditions.

Keywords

AEGISSDCLLMGPUcSensorcVerifierMatmulFlashAttentionbfloat16Checkpoint

Chapters

AEGIS: Why Offline Tests Can Miss SDCs

What live execution reveals before a fault spreads

10 min · What live execution reveals before a fault spreads

10 min

AEGIS: Why SDCs Hide in Real Workloads

Rare triggers, changing inputs, and diverse fault signatures

11 min · Rare triggers, changing inputs, and diverse fault signatures

11 min

AEGIS: From cSensor Signal to Incident

Compact evidence, deferred checks, and explicit data lifetime

10 min · Compact evidence, deferred checks, and explicit data lifetime

10 min

AEGIS: How Evidence Changes Incident Actions

Keep diagnostic, warning, and confirmation scopes distinct

11 min · Keep diagnostic, warning, and confirmation scopes distinct

11 min

AEGIS: Low-Precision Checksums and Attention Invariants

Preserve fault signals before output rounding

13 min · Preserve fault signals before output rounding

13 min

AEGIS: When Recomputation Supports Bitwise Checks

Use xorsum fingerprints for self-equivalent executions

9 min · Use xorsum fingerprints for self-equivalent executions

9 min

AEGIS: Managing Verification Cost in Live Training

Sampling, thresholds, slack, and measured overhead

11 min · Sampling, thresholds, slack, and measured overhead

11 min

AEGIS: What Online SDC Detection Cannot Guarantee

Production evidence, coverage limits, and layered reliability

11 min · Production evidence, coverage limits, and layered reliability

11 min

Preview every chapter · Unlock full text, audio, and video from chapter 4

Plans

Ch.1-3 FreeCh.4-8 Premium

FAQ

Why can a GPU pass diagnostics and still corrupt LLM training?

A GPU can pass diagnostics and still corrupt LLM training because the test may not exercise the rare, input-sensitive workload condition that triggers the SDC.

How does AEGIS confirm a suspicious signal?

AEGIS preserves compact detector-specific context in a vTask, then cVerifier replays the computation or compares fingerprints during available slack; only a confirmed mismatch becomes an incident.

What is the difference between AEGIS’s algorithmic and deterministic detection?

Algorithmic checks use tolerant Matmul or FlashAttention relationships, while deterministic checks compare fingerprints from genuinely self-equivalent executions; they trade flexibility for sharper evidence under different conditions.

What can AEGIS not guarantee in production?

AEGIS cannot certify every GPU operation or guarantee its reported recall, overhead, or incident rate beyond the studied workload, implementation, and coverage. Uninstrumented, transient, software-induced, and communication paths remain subject to the stated uncertainty.