Risks & safety

Existential Risk (from AI)x-risk

Also known as: catastrophic AI risk

The debated worry that extremely advanced AI could, in the long run, pose catastrophic or even civilisation-threatening dangers.

Choose your level

Existential risk from AI is the idea that, someday, AI could become so powerful and hard to control that it poses a catastrophic danger to humanity. It is speculative and hotly debated: some serious researchers think it deserves real attention, others think it is overblown and distracts from AI's here-and-now harms like bias and misinformation. Either way, it is why a lot of effort goes into AI safety and keeping advanced systems under human control.

"Existential risk" (x-risk) refers to the concern that highly capable future AI, especially hypothetical AGI or superintelligence, could cause catastrophic, potentially irreversible harm, for instance if a very powerful system pursued goals misaligned with human interests and could not be controlled.

The debate

Those concerned point to the difficulty of aligning and controlling systems more capable than us, and argue caution is warranted given the stakes.
Sceptics argue it is speculative, unfalsifiable, or a distraction from concrete present harms (bias, disinformation, deepfakes, job impacts).

Why you hear about it

In 2023 many leading researchers and executives signed statements calling AI extinction risk a global priority, bringing the topic into the mainstream, though consensus on likelihood and timelines remains absent.

AI existential risk concerns low-probability-or-uncertain, extreme-severity outcomes from highly capable systems, canonically framed around loss of control over misaligned superintelligence, but also encompassing catastrophic misuse (e.g. AI-enabled bio/cyber threats) and structural/systemic destabilisation. The technical crux is alignment and scalable oversight under a capability gap, where standard evaluation and correction mechanisms may fail.

The field is genuinely divided on probability, timelines and tractability; a prominent 2023 statement equating AI extinction risk with other global priorities signalled mainstream seriousness, while critics contend the framing is speculative and diverts attention and resources from demonstrable near-term harms (bias, privacy, labour, disinformation). Productive treatments avoid both dismissal and certainty: they separate near-term safety from long-term x-risk, note deep uncertainty, and support "no-regret" measures, evaluations, interpretability, governance, dangerous-capability testing, that improve safety across scenarios regardless of one's probability estimate.

← Back to the glossary