Infrastructure
Context Window
Also known as: context length
The maximum amount of text, measured in tokens, that a model can take into account at one time.
The context window is how much an AI can keep in mind at once, its short-term memory for the current conversation. Everything you type, plus everything it has said so far, has to fit inside this window. When a chat gets very long, the earliest parts can fall out of view, which is why an assistant sometimes seems to "forget" what you said at the start.
The context window is the maximum span of text, measured in tokens, that a language model can process in a single request. It holds your instructions, any documents you paste, the conversation history and the answer being generated, all together.
Practical consequences
- It is a hard limit: exceed it and the oldest content must be dropped or summarised.
- Bigger is not automatically better: models can struggle to use information buried in the middle of a very long window, an effect researchers have called "lost in the middle".
- It is not permanent memory: once the request ends, the model does not retain the window unless the application deliberately stores and re-supplies it.
To work with more material than fits, applications use techniques like retrieval-augmented generation to fetch only the most relevant pieces.
The context window is the maximum sequence length the model attends over in one forward pass. It is bounded architecturally by positional encoding and, more practically, by the compute and memory cost of self-attention, which scales quadratically with sequence length in the vanilla transformer (Vaswani et al. 2017); much long-context research targets this cost via sparse, linear or flash attention and improved positional schemes (e.g. RoPE scaling).
Two caveats matter operationally. First, effective use degrades non-uniformly across position, with mid-context recall often weakest (Liu et al. 2023), so placement of critical content matters. Second, the window is stateless across calls: any persistence is an application concern, implemented by re-injecting history, summarising, or retrieving relevant chunks via RAG. Context length also directly drives latency and cost, since both scale with the number of tokens processed.