Models
Attention
Also known as: self-attention
The mechanism that lets a model weigh how relevant each part of the input is to every other part; the core idea inside the transformer.
Attention is the trick that lets AI understand context. When reading a sentence, the model asks, for each word, "which other words matter for understanding this one?" and focuses on those. In "the trophy did not fit in the suitcase because it was too big", attention helps the model figure out that "it" means the trophy, not the suitcase. This ability to focus on the relevant parts is the heart of modern language AI.
Attention is a mechanism that lets a model, for each token, look at all the other tokens and decide how much each one matters. Self-attention means the tokens attend to each other within the same sequence. It is the central component of the transformer.
Why it was a breakthrough
Older models read text strictly in order, which made distant connections hard and training slow. Attention lets every token connect directly to every other token, regardless of distance, and lets the whole sequence be processed in parallel. That combination is what allowed LLMs to scale up.
The cost
Because every token attends to every other, the work grows with the square of the sequence length. That is the main reason a model's context window has limits, and a big focus of efficiency research.
Attention computes, for each query position, a weighted sum of value vectors with weights given by a softmax over scaled dot-product similarities between the query and all key vectors: softmax(QKᵀ/√d)V (Vaswani et al. 2017). Self-attention derives Q, K, V from the same sequence, giving O(1) path length between any two positions, permutation-equivariance (hence explicit positional encodings), and full parallelism over the sequence, properties that made it superior to recurrence for scaling.
Multi-head attention runs several attention subspaces in parallel to capture different relational patterns. The defining cost is O(n²) in sequence length for both compute and memory, the primary constraint on context-window size and a driver of research into efficient variants (sparse, linear, sliding-window, and flash attention, plus KV-cache management at inference). Causal masking restricts each position to attend only to earlier ones, yielding the decoder-only design used by most LLMs.