Foundations
Neural Network
Also known as: artificial neural network
A computing system loosely inspired by the brain, made of layers of simple connected units that learn by adjusting the strengths between them.
An artificial neural network is a computer system loosely modelled on how brain cells connect. It is built from many tiny units ("neurons") arranged in layers and wired together. Learning happens by adjusting the strength of those wires: when the network gets something wrong, the connections are nudged so it does a little better next time. Do that millions of times and it becomes good at a task, without anyone writing explicit rules.
A neural network is the core building block of modern machine learning. It consists of layers of simple units. Each unit takes numbers in, multiplies them by adjustable weights, adds them up, applies a simple non-linear function, and passes the result on.
How it learns
- Data flows forward through the layers to produce an output.
- The output is compared to the desired answer, giving an error.
- That error is sent backwards ("backpropagation") to adjust every weight slightly.
- Repeat over huge amounts of data until the errors are small.
A network with many layers is called a deep network. The adjustable weights are the model's parameters, and modern models have billions of them.
An artificial neural network is a composition of parameterised affine maps interleaved with non-linear activation functions, forming a differentiable function trained by gradient descent. Learning uses backpropagation, reverse-mode automatic differentiation of the loss with respect to the weights (Rumelhart, Hinton & Williams 1986), to compute gradients that an optimiser (SGD, Adam) uses to update parameters.
Non-linearity (e.g. ReLU) is what lets stacked layers approximate complex functions; the universal approximation results establish expressive power, while depth improves parameter efficiency for compositional structure. Architecture encodes inductive bias, convolutions for spatial locality, recurrence or attention for sequences, and the transformer's self-attention is the design that scaled to today's large language models. Practical training depends on normalisation, residual connections, careful initialisation and regularisation to remain stable at depth and scale.