Infrastructure
Tokenizer
Also known as: tokenization
The component that splits text into tokens before a model can process it, and reassembles tokens back into text.
A tokenizer is the little translator that sits in front of an AI model. It chops your text into the small pieces (tokens) the model works with, and turns the model's tokens back into readable text at the end. You never see it, but it decides how your words get carved up, which quietly affects cost, speed and how well different languages are handled.
A tokenizer converts text into a sequence of tokens (and back again). It has a fixed vocabulary of tokens learned in advance; common words often map to one token, while rare or long words are split into pieces.
Why it matters
- Cost and limits: the tokenizer determines how many tokens your text becomes, which sets both price and how much fits in the context window.
- Language fairness: some languages and scripts get split into more tokens, making them more expensive and effectively giving them less room.
- Odd behaviours: because the model sees tokens, not letters, tasks like counting letters or exact spelling can trip it up.
The tokenizer implements the mapping between text and the model's discrete vocabulary, almost always via subword algorithms, byte-pair encoding (BPE; Sennrich et al. 2016), byte-level BPE, WordPiece or unigram/SentencePiece, chosen to balance vocabulary size against sequence length and to guarantee coverage (byte-level variants avoid out-of-vocabulary failures). Vocabulary and merges are learned from a training corpus, so tokenization is model-specific.
Consequences are pervasive: token counts drive context-window usage, inference cost and the quadratic attention budget; uneven fragmentation across scripts produces cross-lingual cost and performance disparities; and sub-word granularity underlies character-level failure modes (counting, spelling, arithmetic on digit-split numbers). Each token id indexes a learned embedding as it enters the network. Tokenizer choice is a real design lever, affecting efficiency, multilingual equity and certain systematic errors, and mismatches between a tokenizer and its model are a common integration bug.