Infrastructure
Latency
Also known as: response time
How long an AI takes to respond; a key factor in how usable and pleasant an AI product feels.
Latency is simply the wait, how long the AI takes to answer after you ask. Low latency feels snappy and natural; high latency feels sluggish. It matters a lot for experience: a chatbot that starts replying instantly feels better than one that pauses for several seconds, even if the final answer is the same. It is why answers often "stream" word by word rather than appearing all at once.
Latency is the time between sending a request and getting a response from a model. For chat, two measures matter: - Time to first token: how quickly it starts replying (this drives the feeling of responsiveness). - Inter-token latency / tokens per second: how fast the rest streams out.
What affects it
- Output length: more tokens to generate means longer total time.
- Input length: a bigger context window takes more work to read.
- Model size and hardware (GPUs).
- Extended thinking by reasoning models adds latency in exchange for accuracy.
Streaming the answer as it is generated is the main trick for making high-latency generation feel fast.
Latency in LLM serving decomposes into prefill (processing the prompt, roughly compute-bound and scaling with input tokens) and decode (autoregressive generation, memory-bandwidth-bound and scaling with output tokens). Standard metrics are time-to-first-token (TTFT), inter-token latency (ITL) and end-to-end latency; TTFT dominates perceived responsiveness, motivating token streaming (Pope et al. 2022).
Latency trades off against throughput and cost under batching: larger batches raise throughput but can increase per-request latency, so serving systems tune batch policy (continuous batching), KV-cache management and parallelism to hit latency SLOs. Model size, context length, quantization, speculative decoding and hardware all move the operating point, and test-time-compute methods (reasoning) deliberately spend latency for quality. In agentic systems, end-user latency compounds across multiple sequential model calls and tool invocations, making step count a first-order design concern.