Infrastructure
FLOPsFLOPs
Also known as: floating-point operations
A measure of raw computation, the number of arithmetic operations, used to quantify how much compute training or running a model takes.
FLOPs measure raw calculation, how many arithmetic operations a computer performs. Since AI is basically an enormous amount of arithmetic, FLOPs are a handy way to say how much "number-crunching" a model needs. Training a big model can take an almost unimaginable number of them, which is why it needs thousands of chips running for weeks. For most people it is just a way to grasp the sheer scale involved.
FLOPs (floating-point operations) count arithmetic operations. Two related uses cause confusion: - FLOPs (operations): a total, e.g. how much compute a training run consumed. - FLOP/s (operations per second): a rate, e.g. how fast a GPU is.
Why it matters
- Training budgets: frontier models are described by their total training FLOPs, a proxy for cost and capability.
- Scaling laws: relate performance to compute measured in FLOPs.
- Hardware: chip speed is quoted in FLOP/s (with a precision, e.g. FP16).
FLOPs are an imperfect measure, real speed is often limited by memory, not arithmetic, but they remain the standard shorthand for compute scale.
FLOPs quantify arithmetic work; the field uses both cumulative operation counts (training/inference budgets) and throughput (FLOP/s at a given numeric precision, FP32/FP16/BF16/FP8). A common training estimate is C ≈ 6ND (N parameters, D tokens), and total training FLOPs anchor scaling-law analyses (Kaplan et al. 2020) and increasingly regulatory thresholds for frontier systems.
Caveats are important: realised performance is frequently memory-bandwidth- or communication-bound rather than FLOP-bound (especially autoregressive decode), so peak FLOP/s overstates usable speed, and model FLOPs utilisation (MFU) measures how much of a device's peak is actually achieved. Precision matters when comparing hardware or runs (a FP8 FLOP is not an FP32 FLOP). FLOPs are thus a useful, standardised, but incomplete metric, best paired with memory, bandwidth and utilisation to reason about real compute cost.