Infrastructure
Token
Also known as: subword unit
The small chunk of text, often a word or part of a word, that a language model reads and generates one at a time.
A token is a little piece of text, roughly a short word or part of a longer word, that an AI reads and writes in. Language models do not think in whole sentences; they handle text one token at a time, like reading and writing with small tiles. This matters because the price of using AI tools and the amount of text they can handle are both measured in tokens.
A token is the basic unit of text for a language model. Before the model sees your text, a tokenizer splits it into tokens: common words often become a single token, while rarer or longer words are broken into pieces. A rough rule of thumb for English is that one token is about four characters, so 100 tokens is roughly 75 words.
Why tokens matter in practice
- Cost: most AI services bill per token, for both what you send and what you get back.
- Limits: a model's context window is measured in tokens, that is how much it can consider at once.
- Languages: some languages and scripts use more tokens per word, which affects both cost and limits.
The model generates its answer one token at a time, each choice influenced by all the tokens before it.
Tokens are the discrete symbols of a model's vocabulary, produced by a subword tokenizer, commonly byte-pair encoding (BPE) or unigram/SentencePiece, that balances vocabulary size against sequence length by merging frequent character sequences (Sennrich et al. 2016). Each token maps to an integer id and then to a learned embedding vector that enters the network.
Tokenization has real consequences: it determines sequence length (and therefore the quadratic attention cost over the context window), it fragments rare words, numbers and non-Latin scripts unevenly (a source of cross-lingual cost and performance disparities), and it defines the granularity at which the model computes its next-token distribution. Generation is autoregressive over this vocabulary, with the chosen token at each step appended and fed back in; sampling behaviour at that step is governed by decoding parameters such as temperature.