Infrastructure
Inference
Also known as: model serving
Actually using a trained model to produce an answer, as opposed to training it; every time you send a prompt, you are running inference.
Inference is the moment an AI actually does its job: you ask, it answers. Training is the long, expensive process of building the model; inference is using the finished model, which happens every single time you send it a prompt. It is the difference between studying for years (training) and answering one question (inference).
Inference is running a trained model to generate output for a given input. Training happens once (and is enormously costly); inference happens constantly, every request from every user. Because it runs so often, its speed and cost are what mostly determine the price and responsiveness of an AI product.
What affects speed and cost
- Number of tokens: longer inputs and outputs take more work.
- Context window used: more context means more computation.
- Model size: bigger models are slower and pricier per token.
- Hardware: specialised chips (GPUs/TPUs) speed inference up.
You will often see two speed measures: time to first token (how quickly it starts replying) and tokens per second (how fast it streams the rest).
Inference is the forward-pass execution of a trained network to produce predictions. For autoregressive LLMs it is sequential, one token per step, and splits into two regimes: a compute-bound prefill over the prompt, and memory-bandwidth-bound decode generating tokens one at a time. The KV-cache stores per-token key/value tensors to avoid recomputation, trading memory for speed and making decode dominated by memory movement.
Optimisation spans batching (especially continuous/in-flight batching), quantisation, speculative decoding, paged attention, and multi-device parallelism (tensor/pipeline) to fit and accelerate large models (Pope et al. 2022). Key metrics are latency (time-to-first-token and inter-token latency) and throughput (tokens/sec, requests/sec), which trade off against cost and against each other under batching. Inference now often dominates the lifetime compute and expense of deployed models, and test-time-compute methods (long reasoning traces) further shift cost from training to inference.