Infrastructure
Quantization
Also known as: model quantisation
Shrinking a model by storing its numbers at lower precision, making it smaller and faster to run with usually a small quality cost.
Quantization makes an AI model smaller and faster by storing its numbers less precisely, using rougher approximations instead of very exact values. Think of rounding prices to the nearest euro instead of tracking every cent: you lose a little accuracy but everything becomes lighter and quicker. This is what lets big models run on smaller computers, even a laptop or phone.
Quantization reduces the numerical precision of a model's parameters (and sometimes its computations), for example from 16-bit numbers to 8-bit or 4-bit. Fewer bits per number means less memory and faster inference.
The trade-off
- Gain: smaller download, less memory, faster and cheaper to run, can fit on modest hardware.
- Cost: some loss of accuracy, usually small with good methods, larger if pushed too far (e.g. very low bit-widths).
Two flavours
- Post-training quantization: compress an already-trained model (quick, common).
- Quantization-aware training: account for quantization during training for better quality.
It pairs naturally with LoRA (as QLoRA) to fine-tune large models on limited hardware.
Quantization maps high-precision weights/activations to lower-bit representations (e.g. FP16→INT8/INT4), reducing memory footprint and bandwidth, the dominant cost in memory-bound autoregressive inference, and enabling integer or low-precision kernels. Post-training quantization (PTQ) is cheap but sensitive to outlier activations; methods like mixed-precision decomposition (LLM.int8(); Dettmers et al. 2022) and calibration-based schemes (GPTQ, AWQ) preserve quality at 8- and 4-bit. Quantization-aware training (QAT) simulates quantization in the loop for better low-bit fidelity at higher training cost.
Key considerations are what to quantize (weights vs activations vs KV-cache), granularity (per-tensor/channel/group), and outlier handling, since a few large activations dominate error. Effects are non-uniform across tasks and can interact with reasoning. Quantization underpins on-device and cost-efficient serving and, combined with low-rank adaptation (QLoRA), makes fine-tuning of large models feasible on single accelerators.