How AEGIS Fits Verification into a Live Training Job

AEGIS is not production-ready merely because its sensors can detect a mismatch. The system must control how many signals become verification tasks, fit those tasks around a running distributed job, and show what that protection costs. This chapter examines the operational choices that connect a checksum or fingerprint to time and resource budgets: adaptive thresholds, sampling, outlier warnings, pipeline-bubble scheduling, and the measured overhead of the reported implementation. The useful judgment is not whether verification is free, but whether its cost and coverage are visible enough to manage.

Control the number of events before they reach the queue

An inline detector operates on a stream of computations. If every small deviation became a replay task, cVerifier could create a backlog large enough to compete with training. AEGIS therefore controls event volume before the vTask queue, while keeping the meaning of a confirmed result separate from the rules that decide which signals deserve attention.

Tolerance must follow the live numerical background

For algorithmic detection, AEGIS maintains a runtime estimate of the normal checksum-difference distribution. The reason is direct: the size of a benign difference is related to the numerical behavior of the computation, including the scale of its activations. That background tends to be stable over a local period of training but can change enough that a fixed ideal error bound is either too sensitive or too permissive.

When a checksum difference exceeds the current tolerance and suggests a possible SDC, AEGIS can temporarily raise the tolerance—for example, by doubling it—to avoid turning a short-lived variance increase into a flood of verification tasks. If later checks are quiet, it can decay the tolerance, for example by halving it toward a baseline. These are practical adaptive heuristics described by the paper, not universal calibration laws. Their job is to regulate the amount of work entering cVerifier; they do not decide that a raised threshold makes an unusual computation correct.

This creates a useful separation between control and semantics. The threshold controls escalation. cVerifier controls confirmation. A signal below the current threshold is not proof that no fault occurred; it is an event that the chosen sensing policy did not escalate. A signal above it is not proof of corruption; it is an event whose evidence is worth spending resources to verify. The system can adjust its workload without changing what “confirmed SDC” means.

Sampling turns coverage into a budget

Checksum calculation is itself non-negligible, especially when it is attached to compute-intensive operators throughout a large model. AEGIS therefore assigns a sampling rate to an operator: the checksum validation runs with a chosen probability rather than on every eligible execution. In the production deployment, the sampling rate is used to find a workable point between detection exposure and performance overhead.

The trade-off is unavoidable. Lower sampling reduces the number of checksum operations, memory copies, and candidate tasks. It also reduces the fraction of qualifying executions that are observed. A sampled detector can therefore meet a performance target while leaving more of the workload outside its direct exposure.