Training
Backpropagation
Also known as: backprop
The algorithm that works out how much each of a network's many settings contributed to its error, so they can all be adjusted efficiently.
Backpropagation is how a neural network figures out what to fix. After it makes a mistake, backprop traces that error backwards through all the network's layers to assign blame: how much did each internal setting contribute to the error? With that information, every setting can be nudged in the right direction. It is the workhorse that makes training large networks practical.
Backpropagation is the method for computing how the error (the loss) changes with respect to every parameter in a neural network. It runs the error backwards from the output layer to the input, applying the chain rule of calculus layer by layer.
Why it is essential
A big network has billions of parameters. Backprop computes all their gradients efficiently in roughly one backward pass, rather than testing each parameter separately (which would be impossibly slow). Those gradients then feed gradient descent, which does the actual updating.
One-line summary
Backprop = "who is responsible for the error, and by how much"; gradient descent = "adjust everyone accordingly".
Backpropagation is reverse-mode automatic differentiation applied to a network's computational graph: it computes gradients of a scalar loss with respect to all parameters in time linear in the graph size by propagating adjoints from output to input via the chain rule (Rumelhart, Hinton & Williams 1986). This efficiency, one forward and one backward pass yields every gradient, is what makes training billion-parameter models tractable; the naive alternative scales with parameter count.
Implementation caches forward activations for reuse in the backward pass (a memory cost addressed by gradient checkpointing), and numerical stability motivates normalisation, residual connections and careful initialisation to avoid vanishing/exploding gradients in deep stacks. Backprop supplies the gradients that first-order optimisers (gradient descent, Adam) consume; it is a differentiation algorithm, not an optimiser, a distinction worth keeping clear. It generalises across architectures wherever operations are differentiable.