Risks & safety
AI Safety
Also known as: safe AI
The field working to make AI systems reliable, controllable and beneficial, and to prevent harm from both today's models and more capable future ones.
AI safety is the work of making sure AI does good and avoids harm, both the everyday harms of today's tools (wrong advice, bias, misuse) and the bigger risks that could come with much more powerful systems in the future. It combines careful engineering, testing, and sensible rules, so that as AI becomes more capable, it stays trustworthy and under human control.
AI safety is the field concerned with building AI systems that are reliable, controllable and beneficial, and with reducing the risks they pose. It spans present-day and future concerns:
- Today: hallucinations, bias, privacy, misuse (scams, disinformation), and unsafe actions by agents.
- Longer term: keeping increasingly capable and autonomous systems aligned with human intentions.
How it is pursued
Through technical work (alignment, RLHF, red-teaming, evaluations, interpretability), through operational practice (human oversight, staged deployment, monitoring), and through governance (standards, policy such as the EU AI Act). A recurring principle is keeping a human in the loop for consequential decisions.
AI safety is the interdisciplinary effort to ensure AI systems behave as intended and do not cause unacceptable harm, encompassing technical alignment, robustness, monitoring, and governance. An influential framing enumerated concrete problems, avoiding negative side effects, reward hacking, scalable oversight, safe exploration and robustness to distributional shift, that remain live research areas (Amodei et al. 2016).
Contemporary practice combines preference-based alignment (RLHF/DPO, constitutional methods), capability and dangerous-capability evaluations, red-teaming, interpretability, and deployment controls (rate limits, guardrails, staged release, incident response). It divides loosely into present-harm safety (bias, toxicity, privacy, security, agent side-effects) and frontier/catastrophic-risk safety (loss of control, misuse of highly capable systems), with debate over how to weight them. Governance layers, standards bodies, safety institutes, and regulation like the EU AI Act, increasingly formalise requirements, while the core technical bottleneck remains scalable oversight: reliably supervising systems approaching or exceeding human evaluative ability.