Risks & safety
Model Collapse
Also known as: model autophagy
The degradation that can happen when AI models are repeatedly trained on AI-generated data, gradually losing quality and diversity.
Model collapse is what can go wrong when AI learns mostly from other AI's output. As more of the internet fills with AI-generated text and images, new models risk training on that instead of genuine human data, like making a photocopy of a photocopy. Over generations, quality and variety can quietly degrade, with rare details disappearing first. It is a reason human-made and well-curated data stays valuable.
Model collapse describes how models can degrade when successive generations are trained heavily on data produced by earlier models, rather than on original human data. Errors and biases get amplified, and diversity shrinks, uncommon cases ("the tails" of the distribution) fade first.
Why it is a concern
- The web increasingly contains AI-generated content, which can end up in future training sets.
- Synthetic data is useful and widely used, but leaning on it recursively without care risks compounding problems.
The nuance
It is not inevitable. Carefully filtered synthetic data, mixed with fresh human data and good curation, avoids collapse. The lesson is about data quality and provenance, not "synthetic data is always bad".
Model collapse (a.k.a. model autophagy disorder) is the degenerative process whereby training on the outputs of prior generative models causes progressive loss of distributional fidelity, first at the tails (rare events, minority modes), then in the core, as sampling error, bias and finite-sample effects compound across recursion (Shumailov et al. 2023). It manifests as reduced diversity, amplified artefacts and eventual quality decline.
The practical driver is the rising fraction of AI-generated content on the open web feeding future pretraining corpora. However, evidence indicates collapse is avoidable: it is mitigated by retaining/accumulating real data, careful filtering and provenance tracking, and disciplined use of synthetic data (which, well-curated, demonstrably improves models). The core implications are for data governance, valuing verified human data, tracking provenance/watermarking, and treating recursive self-training as a risk to manage rather than a free lunch.