AEGIS in Motion: From Suspicion to Confirmation
AEGIS solves a specific architectural problem: the training system must look at the computation in which an SDC may occur, but it cannot afford to verify every computation immediately. This chapter follows the boundaries that make that compromise workable. By tracing what cSensor observes, what a vTask carries, and what cVerifier is allowed to conclude, the reader can distinguish a fast signal from a confirmed incident and see why memory lifetime and scheduling are part of correctness rather than afterthoughts.
The critical path creates evidence
A training job has a critical path: the sequence of work that must complete before the next dependent operation can proceed. A detection system that adds a large replay or a long diagnostic test to that path may improve confidence while making the training job too slow to run. AEGIS therefore gives its first component a narrower responsibility. cSensor observes a protected computation immediately after it finishes, performs a lightweight check suited to that computation, and records only what a later verifier will need.
The timing is essential. The sensor is not examining a generic copy of the model sometime after the event. It sees the inputs, outputs, execution context, and numerical regime in which the computation actually ran. Those details may be gone a moment later: large-model training commonly releases or overwrites intermediate tensors to control memory use. If a sensor waits for a later investigation, the relevant evidence may no longer be available, and a strong verifier may have nothing to replay or compare.
cSensor’s output is not automatically an incident. It is evidence with a confidence level and a next step. If the check looks ordinary, the training operation continues. If the check is suspicious, cSensor packages the signal into a verification task, called a vTask, and places it in a queue for cVerifier. This distinction keeps the critical path small while preserving a route to stronger judgment.
The interface matters because AEGIS supports more than one kind of signal. An algorithmic detector may produce a checksum difference and need a small replay context. A deterministic detector may produce a fingerprint that will later be compared with a fingerprint from a self-equivalent execution. These payloads have different meanings and different data requirements, but both can cross the same system boundary as a vTask. The interface standardizes the work around a signal without pretending that the underlying evidence is interchangeable.
A vTask carries the minimum context for its own verification, along with metadata that lets the system manage it. The paper identifies information such as the operator, row or column indices, and verification type. This is more than bookkeeping. The operator tells cVerifier which confirmation logic applies; selected indices identify the compact tensor slice retained for replay; and the verification type determines whether the next step is a fingerprint comparison or a recomputation.
The packing choice is shaped by tensor lifetime. For some forward and input-gradient matrix multiplications, AEGIS can retain a row of an input and the corresponding output row associated with the largest numerical discrepancy. For weight-gradient cases, it uses row-and-column checksums and retains selected rows, columns, and corresponding output elements. Attention verification needs a different context, including items such as the saved log-sum-exp value, gradients, and relevant inputs. These are examples of a general rule: the sensor must preserve the smallest evidence bundle that still makes its assigned confirmation possible.
That rule also explains why the vTask abstraction is not merely a message format. It is a state boundary. Before the handoff, cSensor owns the immediate observation and decides whether the local signal is strong enough to create deferred work. After the handoff, cVerifier owns the task’s confirmation. The training framework may continue releasing ordinary intermediates, but it must not invalidate the compact context that the vTask promises to carry.
A useful failure case is a suspicious signal whose context was not retained in time. cSensor might still know that a checksum difference exceeded a local tolerance, but cVerifier could be unable to reproduce the operation or reconstruct the comparison. In that situation, the architecture cannot honestly upgrade the signal to a confirmed SDC. The paper describes the retention policy as a way to avoid this problem for the supported detectors; it does not establish a universal memory-cost or evidence-retention policy for every training framework.
The boundary is also intentionally selective. Instrumenting every operation would increase coverage in one sense, but it would add checksum work, metadata, and saved context throughout a memory-constrained job. AEGIS instead focuses on selected, compute-intensive paths and uses detector-specific packing. Selection is therefore a coverage decision with a cost, not a claim that uninstrumented work is safe.
It helps to view a vTask as a promise with a deadline rather than as a dump of everything that happened. At creation, cSensor has the richest view of the operation, but only a short window in which intermediate state is available. It must choose the evidence that can survive. At consumption, cVerifier may be running on a different schedule and may have no access to the original large tensor. The task is useful only if the information crossing the boundary is sufficient for the verifier’s particular test.
That lifecycle also clarifies why detector diversity does not require separate infrastructure for every detector. The sensing logic owns the meaning of its local signal; the shared queue owns the fact that work is pending; and the verifier owns the transition from pending evidence to a confirmation result. A checksum difference should not be interpreted by the queue, and a fingerprint mismatch should not be treated as just another counter. The queue coordinates work while the detector and verifier preserve semantics.
The producer and consumer can operate at different rates. cSensor may create a task while training is busy, whereas cVerifier waits for a suitable idle interval. This decoupling is useful only if the task remains valid while it waits. A task that loses its replay context, is mistaken for a different verification type, or is processed after its associated state has been overwritten cannot deliver the certainty the architecture promises. The paper addresses this through compact, detector-specific packing; the general lesson is that asynchronous verification needs an explicit data-lifetime contract.
The same contract limits what a confirmed result says. cVerifier can establish that the protected computation failed its assigned comparison or replay. It does not thereby inspect every neighboring operation, prove that every tensor in the training step is correct, or explain every model-quality change. The report is precise because the evidence path is precise. That precision is a strength: it gives the platform a defensible reason to isolate a device without pretending that one confirmed mismatch is a complete forensic account of the run.
The sequence makes the ownership rule visible. Training invokes the observation; cSensor decides whether to emit deferred work; the queue holds the handoff; cVerifier decides what the evidence means; and the platform acts on a confirmed report. The design does not require cSensor to solve the entire diagnosis before the training job can proceed.
The deferred path turns evidence into a decision
cVerifier is the authority for confirmation because the first signal may be intentionally economical. For a deterministic vTask, it compares fingerprints from two executions that should be bitwise identical under the same inputs and configuration. For an algorithmic vTask, it replays the suspicious computation using the saved context and checks the result against the recorded evidence. A mismatch in the appropriate comparison supports an SDC report; a signal that cannot survive the stronger test remains unconfirmed rather than being promoted by urgency alone.
This division protects two different properties. cSensor protects relevance: it watches the real operation and can notice a condition that an offline test never exercises. cVerifier protects meaning: it applies a stronger test before the platform treats the event as corruption. If those responsibilities were merged, either sensing would become too expensive or reporting would rely on a noisy signal. AEGIS keeps the distinction explicit in the control flow.
The queue is what allows the two time scales to coexist. Suspicious work can be created during forward or backward execution, but cVerifier does not have to run at that exact instant. The paper describes a scheduler that treats vTasks as deferred work and uses naturally occurring idle periods, especially pipeline bubbles in interleaved pipeline parallelism. During a bubble, a worker is stalled waiting for tensor data from another stage; that interval can absorb verification work without adding a new synchronization stall to the training schedule.
The benefit is conditional on the timing actually being available. A bubble is not an unlimited compute pool, and a run with an unusually high volume of suspicious signals could create a queue backlog. AEGIS uses a best-effort policy: workers process tasks opportunistically during idle periods, and a small fixed time slice at the end of a training step handles tasks that were not completed there. The system therefore preserves throughput by placing work where slack already exists while acknowledging that deferred work still consumes GPU time.
That bounded policy changes how confirmation latency should be understood. A task may be detected inline but verified later. “Online detection” describes when the relevant computation is observed, not a promise that every incident is confirmed before the next instruction or step. The separation is useful precisely because immediate confirmation is often too expensive. But operators still need to consider queue pressure, the size of the residual time slice, and what happens when the rate of suspicious work exceeds available idle capacity. The paper establishes the scheduling strategy and its measured behavior in the described pipeline setting; it does not provide a universal backlog guarantee for every deployment.
Once cVerifier confirms an SDC, the result leaves the local detection loop. AEGIS reports the incident with the associated device and execution context so the platform and users can respond. The intended operational path includes isolating the faulty GPU and restarting or rolling back training to a previous checkpoint, minimizing the amount of computation that must be discarded. The report is valuable because it turns an abstract mismatch into a resource-management decision: this device and this execution should no longer be trusted in the same way.
The distinction between confirmation and action remains important. Confirmation says that the protected computation failed the relevant strong comparison or replay. Isolation is a platform policy based on that evidence and the cost of continued participation. Recovery uses checkpointing and other training infrastructure; cVerifier does not itself recreate the entire job. AEGIS supplies the timely, device-linked signal that lets those systems act before silent damage compounds.
The paper’s description also leaves some production questions deliberately outside this control loop. It establishes queueing within the running detection workflow, but it does not present the vTask queue as a durable store across process failure or specify a general retry protocol for incomplete verification. Nor does a report prove that every other operation on the device is correct. Those are not contradictions in the architecture. They are reminders that AEGIS is an online detection layer with defined interfaces, not a complete replacement for checkpointing, diagnostics, ECC, or platform failure management.
The complete path can now be stated without collapsing its stages. A real training computation creates an opportunity for an error. cSensor observes it with a low-cost, detector-specific check. A vTask preserves the evidence that might otherwise disappear. cVerifier uses a stronger comparison or replay during available slack. Only a confirmed result becomes an incident report, and the platform can then isolate the device and recover the training state. Each boundary exists because speed, memory, certainty, and operational action are different requirements.
That architecture sets a standard for the mechanisms that follow. A useful sensor must be cheap enough to run near the operation, precise enough to select worthwhile verification, and deliberate about what state it saves. A useful verifier must be strong enough to filter benign numerical variation and scheduled well enough not to become a new source of training instability. AEGIS does not solve reliability by choosing one perfect test; it makes several kinds of evidence meet at a controlled interface.