Infrastructure
Throughput
Also known as: tokens per second
How much work an AI system can handle in a given time, for example tokens generated or requests served per second.
Throughput is how much an AI system gets done per second, how many words it can produce or how many users it can serve at once. It is different from speed-for-one-person (latency): a system can serve many people at high total throughput while each individual still waits a moment. For running an AI service affordably, high throughput is what keeps costs down.
Throughput measures the volume of work per unit time, commonly tokens per second across all users, or requests served per second. It is the "how much at once" metric, complementing latency (the "how fast for me" metric).
The key trade-off
Throughput and latency pull against each other. Batching, processing many requests together, raises throughput (and lowers cost per request) but can add a little latency for each individual. AI serving is largely about tuning this balance to meet response-time targets while keeping cost per token low.
Why it matters
Higher throughput on the same hardware means lower cost and the ability to serve more users, directly affecting an AI product's economics.
Throughput quantifies served work per unit time (aggregate tokens/sec, requests/sec) and, together with latency, defines the operating point of an inference system. The dominant lever is batching: larger batches amortise weight loads and improve accelerator utilisation, raising throughput and lowering cost-per-token, at the expense of per-request latency, so systems use continuous/in-flight batching to add and retire requests dynamically (Pope et al. 2022).
Because decode is memory-bandwidth-bound, throughput is often limited by memory movement and KV-cache capacity (which caps concurrent sequences) rather than raw FLOPs. Further gains come from paged attention, tensor/pipeline parallelism, quantization, speculative decoding and prefix caching. Capacity planning targets throughput per accelerator subject to a latency SLO; reported "tokens/sec" must specify whether it is per-request or aggregate, since batching makes the two diverge sharply.