Training
Scaling Laws
Also known as: neural scaling laws
Predictable patterns showing how AI models get better as you increase their size, training data and compute.
Scaling laws are the discovery that AI models get predictably better as you make them bigger, feed them more data and give them more computing power. This reliability is a big reason the field poured resources into ever-larger models: researchers could forecast, roughly, how much a bigger model would improve before building it. It is a bit like knowing in advance how much faster a bigger engine will make a car.
Scaling laws are empirical relationships showing that a model's performance improves smoothly and predictably as three ingredients grow: the number of parameters, the amount of training data, and the compute budget. Crucially, the improvement follows a regular curve, so it can be extrapolated.
Why they matter
- They justified building very large models: bigger, trained well, reliably meant better.
- They guide how to spend a fixed budget, how big a model, on how much data.
The Chinchilla correction
An influential 2022 study found many large models were undertrained: for a given compute budget, they should have used a smaller model with more data. The lesson, "balance model size and data", reshaped how frontier models are built.
Neural scaling laws describe test loss falling as a power law in parameters N, dataset size D and compute C, over many orders of magnitude (Kaplan et al. 2020). Their practical value is predictability: performance at scale can be forecast from smaller runs, enabling budget allocation and architecture bets before committing to expensive training.
The compute-optimal refinement (Hoffmann et al. 2022, "Chinchilla") showed the loss-minimising allocation scales N and D roughly proportionally for a fixed C, implying prior flagship models were oversized and undertrained; this reoriented practice toward more tokens per parameter. Caveats: laws are objective- and distribution-specific, describe pretraining loss rather than downstream utility (which can show discontinuous/emergent behaviour), and interact with data quality and repetition limits. They do not promise indefinite returns, and debate continues over data ceilings and where diminishing returns bind, motivating complementary axes such as test-time compute (reasoning models).