Training
Gradient Descent
Also known as: stochastic gradient descent, SGD
The core optimisation method that trains neural networks by repeatedly nudging their settings in the direction that reduces error.
Gradient descent is how an AI actually learns. Picture standing on a foggy hillside wanting to reach the lowest point: you feel which way is downhill and take a small step, over and over. Training an AI is the same idea, "downhill" means "less error", and each step adjusts the model's internal numbers a little to make it more accurate. Do this millions of times and the model gets good.
Gradient descent is the optimisation algorithm behind training neural networks. It works by: 1. Measuring the model's error with a loss function. 2. Computing the gradient, the direction in which the error increases fastest, via backpropagation. 3. Adjusting the model's parameters a small step in the opposite direction.
The step size is the learning rate: too large and it overshoots, too small and training crawls.
In practice
Real training uses stochastic gradient descent (SGD): it estimates the gradient from small batches of data at a time, which is far more efficient. Modern optimisers like Adam improve on plain SGD with adaptive step sizes.
Gradient descent minimises a differentiable loss by iteratively updating parameters θ ← θ − η∇L(θ), with learning rate η and gradients obtained by backpropagation. Stochastic/minibatch variants estimate ∇L from data subsets, trading gradient noise for computational efficiency and, often, better generalisation. Convergence and stability depend on learning-rate scheduling, batch size, initialisation and normalisation.
Deep-learning practice relies on adaptive first-order methods, momentum, RMSProp, Adam/AdamW, which precondition or smooth updates for faster, more robust training on non-convex landscapes. Key phenomena include saddle points (more problematic than local minima at scale), the loss-landscape geometry's effect on generalisation, and the interaction of batch size with learning rate. Gradient descent's scalability to billions of parameters, enabled by efficient autodiff and hardware, is what makes training large models feasible at all.