Training
Epoch
Also known as: training epoch
One complete pass of the training process through the entire training dataset.
An epoch is one full read-through of all the training data. Training usually involves going over the data several times (several epochs), each pass letting the model refine what it has learned, a bit like re-reading a textbook a few times to really absorb it. Too few passes and it under-learns; too many and it starts memorising instead of understanding.
An epoch is one complete pass of gradient-descent training over the whole training dataset. Training typically runs for multiple epochs, each further reducing the error.
The balance
- Too few epochs: the model underfits (has not learned enough).
- Too many epochs: it risks overfitting, memorising the training data rather than generalising.
Practitioners watch performance on held-out data and often stop when it stops improving ("early stopping").
A frontier-model note
The biggest LLMs are often trained on so much data that they see much of it only once (roughly a single epoch), because there is more high-quality data than compute to reuse it, so "epochs" matter more for smaller models and fine-tuning than for frontier pretraining.
An epoch is one full traversal of the training set by the optimiser; within an epoch, updates occur per minibatch. Epoch count is a training hyperparameter governing the underfitting/overfitting trade-off, tuned via validation curves and early stopping.
Scale changes the picture: compute-optimal pretraining of large LLMs often approximates single-epoch (or sub-epoch) training over massive corpora, since data abundance and the diminishing returns and memorisation risk of repetition favour more unique tokens over repeated passes (with documented limits on how many times data can be safely repeated). Multi-epoch training remains standard for smaller datasets, fine-tuning and vision. Related bookkeeping terms, step/iteration (one minibatch update) and batch size, interact with epoch count and learning-rate schedules to define the total optimisation budget.