Checkpoint: From a Suspicious Signal to a Trustworthy Incident

AEGIS brings together three ideas that are easy to confuse: a test can be useful without being comprehensive, an online signal can be relevant without being conclusive, and a confirmed mismatch can justify action without proving that every operation is healthy. This checkpoint connects those ideas into one evidence chain. The practical judgment is whether a GPU event has earned screening, investigation, verification, or an incident response—and what uncertainty remains at each point.

One event can have several meanings

Consider a hypothetical training run on a GPU that has passed a memory-integrity diagnostic. That result matters: under the plugin’s allocation, patterns, prerequisites, and execution, the tested memory behavior was acceptable. But the result does not answer whether a matrix unit will produce a wrong value for a particular low-precision input, whether an attention computation will be internally consistent, or whether a distributed training step will remain correct. The test has a scope, and the scope travels with the evidence.

Now suppose the live job produces an unusual checksum difference. This observation is more relevant to the training run because it occurred in the operation and numerical regime that the job actually used. It is still not automatically a confirmed SDC. The difference could be a fault, but it could also be ordinary round-off or another benign variation, especially when the operation uses low-precision values. The online signal earns a place in the verification path; it does not earn the right to stop the job by itself.

A cross-rank outlier warning has a related but distinct meaning. If the same operation and batch produce relatively stable checksum differences across data-parallel ranks, a rank that deviates far beyond the group baseline deserves attention. AEGIS treats that as a separate warning channel because the signal can identify a likely problem without carrying the evidence needed for definitive confirmation. It can also be sensitive to out-of-distribution values during model warm-up. A warning supports investigation; it is not proof.

A cVerifier confirmation is stronger because it applies the test that the initial signal could not afford to run inline. Depending on the vTask, it may compare fingerprints from self-equivalent executions or replay the suspicious computation using retained context. If the stronger comparison confirms the mismatch, AEGIS can report an incident with device and execution information. If it does not confirm corruption, the original signal remains a useful record of an unusual event, but it should not be rewritten as a hardware failure.

These meanings produce different actions. A failed memory-integrity test can justify removing a device from service for diagnosis. A passing test justifies confidence only within its tested scope. A cSensor signal justifies preserving context and scheduling verification. An outlier warning justifies investigation and correlation with other evidence. A cVerifier confirmation justifies an incident report and can support device isolation and training recovery. The actions are not ranked versions of the same alert; they follow from different evidence.