Training
Loss Function
Also known as: objective function, cost function
The formula that scores how wrong a model's prediction is; training works by making this number as small as possible.
A loss function is the model's report card, a single number that says how wrong its answer was. A big loss means a bad prediction; a small loss means a good one. Training is simply the process of changing the model to make this number as small as possible. Choosing the right loss function matters, because a model becomes good at exactly what you measure, so what you reward is what you get.
A loss function measures the gap between a model's prediction and the desired output, producing a number that training tries to minimise. It is the objective the whole training process is aimed at.
Common examples
- Cross-entropy: for classification and for predicting the next token in language models.
- Mean squared error: for predicting numbers (regression).
Why it is a design choice
The loss defines what "good" means for the model. A poorly chosen loss can make a model optimise the wrong thing, and even a good loss on training data can hide overfitting. The loss provides the signal that backpropagation and gradient descent use to update the model.
The loss (objective/cost) function defines the scalar training criterion whose gradient drives optimisation. Its choice encodes the task and its assumptions: cross-entropy/negative log-likelihood for categorical prediction (including autoregressive next-token modelling), MSE for Gaussian regression, plus task-specific and structured losses. Loss design also incorporates regularisation (weight decay, label smoothing) and auxiliary terms that shape generalisation and calibration.
Crucially, the loss is a proxy for what we actually want, minimising it does not guarantee the intended behaviour, which is the technical seed of specification and reward-hacking problems in alignment (see also RLHF's learned reward as a loss surrogate). Practical concerns include differentiability, numerical stability (log-sum-exp, clipping), class imbalance handling, and the mismatch between a smooth training loss and non-differentiable evaluation metrics, which motivates surrogate losses and, in RL/preference settings, learned or ranking-based objectives.