Models
Transformer
Also known as: transformer architecture
The neural-network design, based on an "attention" mechanism, that powers virtually all modern large language models.
The transformer is the engine inside almost every modern AI that works with language. Its clever trick is called "attention": when processing a piece of text, it can weigh how much each word relates to every other word, so it captures meaning and context rather than reading in a rigid straight line. This design, introduced in 2017, is what made today's powerful chatbots possible.
The transformer is a neural-network architecture introduced in 2017 in a paper titled "Attention Is All You Need". It is the foundation of large language models and much of modern generative AI.
The key idea: attention
Older networks processed text word by word in sequence, which made long-range connections hard. The transformer's self-attention mechanism lets every token look at every other token at once and decide which ones are relevant. This both captures context better and allows massive parallel training, which is why transformers scaled up so successfully.
From tokens to output
Text is split into tokens, each turned into an embedding, passed through many layers of attention and processing, and finally converted into a probability for the next token.
The transformer replaces recurrence and convolution with stacked self-attention and position-wise feed-forward layers (Vaswani et al. 2017). Self-attention computes, for each position, a weighted sum over all positions using query-key-value projections and scaled dot-product similarity, giving O(1) path length between any two tokens at the cost of O(n²) compute over sequence length, which is the central constraint on the context window. Multi-head attention runs several attention subspaces in parallel; residual connections and layer normalisation stabilise deep stacks; and since attention is order-invariant, positional information is added explicitly (learned or rotary/RoPE encodings).
Most contemporary LLMs are decoder-only transformers trained with a causal mask for autoregressive next-token prediction. The architecture's decisive property is favourable scaling: its parallelism and capacity let performance improve predictably with model size, data and compute, the empirical basis of the current frontier-model era. Efficiency variants (flash, sparse and linear attention, mixture-of-experts) target the quadratic bottleneck while preserving the core design.