The State AI Leaves Behind

AI infrastructure is often discussed as if security ends when a job has been admitted and a model has produced an answer. The harder question is what remains after each transition: which data must be recovered, which state may be reused, and which software is allowed to continue operating. This chapter follows training data, project information, outputs, accelerator state, and user-developed software through use, failure, and reuse. The aim is to distinguish durable assets from disposable intermediate work and to make recovery and isolation decisions visible.

That distinction matters because AI workloads do not produce one undifferentiated kind of data. They may use valuable training data and model weights, create intermediate results that change frequently, depend on configuration that makes a run reproducible, and leave state on shared computing resources. Treating all of it alike either wastes capacity on low-value copies or loses the information needed to restore a useful system.

Recovery follows value and responsibility

A training project begins with inputs: datasets, project information, configuration, and software that define what the workload is meant to do. During execution it may create temporary files, checkpoints, intermediate results, logs, and other artifacts. At the end, it may produce model weights, inference outputs, and results that a researcher needs to preserve. These objects have different recovery value. A changing intermediate file can be large but easy to regenerate; a configuration record or authoritative input may be smaller but essential to reproducing the result.

NIST says that full backup may be prohibitive for HPC systems because of their data volume and because intermediate results often change and have little intrinsic value. It recommends prioritizing configuration data and critical project information so the system can be restored to a functional state. The point is not that intermediate data is always unimportant. It is that backup is a decision about mission and recovery, not a contest to copy the greatest number of bytes.

Consider two failures. In the first, a system loses a temporary result but retains its configuration and authoritative input data. Re-running the work may be expensive, but the project can reconstruct the result. In the second, the final model output remains while the configuration or critical project data is lost. The surviving file may not be enough to restore the process that produced it or to establish what the result means. The source does not prescribe a universal retention policy, but its prioritization principle explains why the second loss can be more damaging even if the first file was larger.

Recovery also has more than one owner. NIST describes reconstitution as a shared responsibility. Operators are responsible for system and configuration data, while data owners may need to establish processes for their project data and outputs. The distinction follows from control. An operator can maintain the environment that starts the job, but may not know which research input is authoritative or which output must be retained for a project. A data owner can know that value without controlling the storage infrastructure. If the two responsibilities are not explicit, each party can assume that the other has preserved the information needed after a failure.

The same logic applies to an alternate processing site.