Foundations
Training Data
Also known as: dataset, corpus
The collection of examples a model learns from; its quality and content largely determine what the model becomes.
Training data is everything a model learns from, the text, images or examples fed to it during training. It shapes the model more than almost anything else: an AI is, in a real sense, a reflection of its training data. Good, varied, accurate data makes a capable model; narrow or biased data makes a biased one. "Garbage in, garbage out" very much applies.
Training data is the dataset a model learns from. For LLMs this is typically a vast text corpus (web pages, books, code); for image models, image-text pairs. Its scale, quality, diversity and cleanliness heavily determine the model's capabilities and flaws.
Why it is decisive
- Capability: more and better data (well-balanced with model size) yields better models.
- Bias: models absorb the biases and gaps in their data.
- Knowledge cut-off: the model only "knows" what was in its data, up to a point in time.
- Provenance: where the data came from raises copyright and privacy questions.
Increasingly, curated and synthetic data is used to raise quality.
Training data is the empirical foundation of a model; under self-supervised pretraining, the data distribution effectively defines the learned distribution. Its composition, scale, quality, deduplication, domain mixture and filtering, is a primary determinant of capability, often more so than architecture, and scaling laws tie data quantity to compute-optimal model size.
Critical concerns: bias and representational gaps propagated from data; benchmark contamination inflating scores; memorisation of PII/copyrighted text (privacy, copyright); temporal cut-off necessitating retrieval; and provenance/consent. Best practice emphasises documentation (datasheets/data cards; Gebru et al. 2018), rigorous cleaning and deduplication, decontamination against evaluations, and increasingly heavy use of curated and synthetic data, the latter carrying model-collapse risk if used recursively without fresh real data. Data curation is now a central lever of frontier model quality.