Training
Pretraining
Also known as: pre-training
The first, largest training phase where a model learns general knowledge and skills from a huge amount of data, before any task-specific tuning.
Pretraining is the big, foundational stage of building an AI, where it reads or sees an enormous amount of data and absorbs general knowledge and skills. It is like a long, broad education before any specialisation. This phase is by far the most expensive and time-consuming part, and it is where a language model gains most of what it "knows".
Pretraining is the initial phase in which a foundation model learns from a very large, broad dataset, before any task-specific adjustment. For LLMs, this means learning to predict the next token across a huge text corpus, which teaches grammar, facts, reasoning patterns and more, all without human labels (the data supplies its own answers).
Pretraining vs fine-tuning
- Pretraining: broad, general, huge data, done once, very costly. Builds general capability.
- Fine-tuning: narrow, specific, smaller data, cheaper. Specialises behaviour.
Why knowledge is "frozen"
Because pretraining happens at a point in time, the model's knowledge has a cut-off date. Anything newer must be supplied at use time via retrieval or tools.
Pretraining optimises a model on a large, broad corpus under a self-supervised objective, for LLMs, autoregressive next-token prediction, which requires no manual labels and thus scales to internet-sized data. It is where the bulk of world knowledge, linguistic competence and in-context learning ability is acquired, and it dominates total training cost. The payoff scales predictably with model size, data and compute per empirical scaling laws, with compute-optimal analyses prescribing balanced parameter/token budgets.
Downstream, pretraining is followed by adaptation, supervised fine-tuning and preference optimisation (RLHF/DPO), which reshape behaviour but add relatively little new knowledge. Key considerations at this stage are data quality, deduplication and mixture, contamination of evaluation sets, and the temporal knowledge cut-off that makes retrieval and tool use necessary for current information. The base checkpoint produced here is the raw foundation model prior to alignment.