Why Self-Generated Data Changes the Target

The Adaptive Data Flywheel makes model-assisted data creation look useful: production failures can be diagnosed, corrected, and expanded into more training examples. The model-collapse research asks what changes when those generated examples stop supplementing an external reference and begin replacing it. This chapter explains that mechanism in plain language. The goal is to distinguish two kinds of error—finite-sample error and model-approximation error—and to show why a larger generated dataset may still contain less independent information about the distribution an enterprise agent is meant to serve.

The issue is not that a model produces text or that a flywheel uses synthetic data. The issue is recursive dependence. A model learns a distribution, generates data from its learned approximation, and supplies that data to the next model. If some events are missing or distorted in the first generation, the next generation treats the altered record as if it were the world. The feedback loop then changes not only the model but the target against which later models are judged.

The generational loop

Imagine a clean collection of human-produced examples. A first model is trained on that collection and learns an approximation of the underlying distribution: which questions, phrases, events, or patterns occur, and with what variety. The model then generates a new dataset. A later model is trained on that generated dataset, generates another one, and passes it forward.

This process differs from ordinary evaluation. In evaluation, a model is compared with a reference that is meant to remain outside the model's control. In generational training, the model's output becomes part of the next reference by entering the next training set. The distinction is easy to lose because each generation may appear internally coherent. A dataset can be grammatical, plausible, and large while still being a biased sample of the distribution from which it descended.

The paper calls the resulting degenerative process model collapse. In its early form, the tails of a distribution begin to disappear. “Tails” means low-probability events: unusual but real questions, less common meanings, minority cases, and exceptions. In a later form, different modes of the original distribution become entangled and the model converges toward a narrow, low-variance approximation that carries little resemblance to the starting distribution.

That vocabulary is deliberately narrower than “the model got worse.” Model collapse is not every hallucination, every distribution shift, or ordinary overfitting. It is the generational setting in which generated data pollute the training data of later models. The source also distinguishes it from catastrophic forgetting. In catastrophic forgetting, one model loses access to previously learned tasks or examples as it learns something new. In model collapse, successive models reinforce an increasingly distorted account of what is real.

The difference matters for an Adaptive Data Flywheel. A system can improve a component against a current test while quietly making its future evidence less diverse. If the loop keeps only the latest model-shaped examples, the problem is not merely that a checkpoint forgot an old case. The lineage has stopped carrying the information needed to remember that the case was possible.

Why the tail disappears first

The paper's simplest argument uses a discrete distribution, like a histogram of possible events. Suppose an event has probability q and the next generation draws M samples. If q is around or below 1/M, the expected number of appearances is less than one. The event may be genuine and important, but a finite sample can contain no instance of it. Once it is absent from the generated training data, a later fitting step has no direct evidence that it exists.