Infrastructure
Batch Inference
Also known as: batching
Running many AI requests together in one go to use the hardware efficiently, which lowers cost per request but can add a little delay.
Batch inference means handling many AI requests together instead of one at a time, like a bus carrying many passengers on one trip rather than a taxi per person. It makes far better use of the expensive hardware, so each request costs less. The trade-off is that an individual might wait slightly longer to "board". It is a key trick behind affordable AI services, and behind big offline jobs like processing a whole database of documents.
Batch inference processes multiple inference requests together in one pass. There are two contexts: - Online batching: a live service groups concurrent user requests to use the GPU efficiently, raising throughput and lowering cost per request, at a small latency cost per user. - Offline batch jobs: run a large set of inputs (e.g. classify a million records) in bulk, where latency does not matter and efficiency is everything, often at a discount.
Why it works
GPUs are most efficient when kept fully busy. Processing one request at a time wastes capacity; batching fills it, which is central to the economics of serving AI.
Batching amortises weight loads and maximises accelerator utilisation by processing multiple sequences per forward pass, the primary lever for inference throughput and cost efficiency, at the expense of per-request latency. For autoregressive generation, static batching is inefficient because sequences finish at different lengths; continuous (in-flight) batching dynamically admits and retires requests at the token level to keep the batch full (Orca; Yu et al. 2022), and is standard in modern serving stacks.
Constraints include KV-cache memory (which caps concurrent sequences), latency SLOs (bounding batch size for interactive use), and workload heterogeneity (mixed prompt/output lengths). Offline batch inference (throughput-only, latency-insensitive) is often offered at reduced price and suits bulk pipelines. Effective batching interacts with paged attention, scheduling, and prefill/decode disaggregation; it is a core determinant of the cost-per-token and concurrency a deployment can achieve on given hardware.