Infrastructure
KV CacheKV cache
Also known as: key-value cache
A memory trick that makes AI text generation faster by storing intermediate results so they don't have to be recomputed for every new word.
The KV cache is a speed trick inside AI text generation. Because a model writes one word at a time and each new word depends on all the previous ones, it would be wasteful to redo all that work every step. So the model stores its intermediate calculations and reuses them, making generation much faster. You never see it, but it is a big reason chatbots can reply at a readable pace.
When a model generates text one token at a time, each step's attention needs information ("keys" and "values") from all earlier tokens. Recomputing that every step would be very slow. The KV cache stores those keys and values so each new token only computes its own, reusing the rest.
Why it matters
- Speed: it makes step-by-step generation far faster (lower latency).
- Memory cost: the cache grows with the length of the conversation, so long contexts use a lot of memory, often the real limit on how much a system can handle at once.
Managing the KV cache efficiently is a major focus of AI serving systems.
The KV cache stores per-layer key/value tensors for all past positions so autoregressive decode computes attention only for the new token, turning per-step cost from quadratic-in-length recomputation into an incremental update, essential for practical inference latency (Pope et al. 2022). It is what makes decode memory-bandwidth-bound rather than compute-bound.
Its footprint scales with batch × layers × heads × head-dim × sequence length, making it the dominant memory consumer for long contexts and large batches, and thus a primary serving bottleneck. Optimisations include paged attention (virtualised, fragmentation-free cache memory), multi-query/grouped-query attention (sharing K/V across heads to shrink the cache), cache quantization, eviction/compression for long contexts, and prefix caching to reuse shared prompts. KV-cache management is central to throughput/latency trade-offs and to how many concurrent requests a given accelerator can serve.