Training
Overfitting
Also known as: over-fitting
When a model memorises its training data instead of learning general patterns, so it does great on what it has seen but poorly on anything new.
Overfitting is when an AI "learns the answers" instead of understanding the subject, like a student who memorises last year's exam word-for-word but is lost when the questions change. The model looks brilliant on the data it trained on, then fails on new, real-world data. The goal of good training is the opposite: learning general patterns that hold up on things the model has never seen.
Overfitting happens when a model fits its training data too closely, capturing noise and quirks rather than the underlying pattern. The tell-tale sign is a large gap: low error on training data but high error on new (validation/test) data.
Why it happens and how to prevent it
- Causes: too little data, too large/flexible a model, training too long.
- Fixes: more (and more varied) data, regularisation (penalising complexity), early stopping, dropout, and holding out a test set to check honestly.
The opposite problem is underfitting, when a model is too simple to capture the pattern. Good training balances the two, and honest evaluation on unseen data (or a benchmark) is how you catch overfitting.
Overfitting is the regime where a model's capacity lets it fit sample-specific noise, minimising training loss while generalisation error rises, the high-variance end of the bias-variance trade-off (Hastie et al. 2009). It is diagnosed by a widening train/validation gap and controlled by regularisation (L2/weight decay, dropout, data augmentation), early stopping, and capacity/data balancing.
Two modern caveats complicate the classic picture. First, deep networks often generalise well despite being heavily overparameterised, and can exhibit double descent, where test error falls again beyond the interpolation threshold, so parameter count alone is a poor overfitting predictor. Second, at foundation-model scale the pressing failure is often evaluation contamination (test data leaking into pretraining) inflating benchmark scores, a form of overfitting to the benchmark rather than the task. Rigorous held-out evaluation, decontamination, and out-of-distribution testing are the practical guards.