Risks & safety
ExplainabilityXAI
Also known as: interpretability
The degree to which we can understand why an AI produced a particular result; hard for modern models, which are largely "black boxes".
Explainability is being able to say why an AI gave a certain answer or decision. With modern AI this is genuinely hard: the models are so complex that even their makers cannot fully trace a specific output, they are often called "black boxes". That matters when AI affects people's lives (loans, jobs, medical advice), where "the computer said so" is not good enough and we need reasons we can check.
Explainability (and the closely related interpretability) is about understanding how and why an AI system reaches its outputs. Because neural networks have billions of interacting parameters, their internal reasoning is not directly readable.
Why it matters
- Trust and accountability: especially for high-stakes decisions.
- Debugging: understanding failures, catching bias.
- Regulation: some laws require explanations for automated decisions.
A caution about chatbots
When an AI "explains its reasoning", that explanation is itself generated text and may not reflect what actually happened inside the model. A plausible story is not a guaranteed-true account, an important limit to keep in mind.
Explainability/interpretability seeks faithful, human-understandable accounts of model behaviour, spanning post-hoc attribution (feature-importance methods like SHAP, LIME; saliency) and, for neural networks, mechanistic interpretability that reverse-engineers internal circuits and features. A core difficulty is the lack of an agreed, rigorous definition and evaluation of "interpretability" itself (Doshi-Velez & Kim 2017), and the tension between fidelity (truthful to the model) and comprehensibility (useful to humans).
For LLMs, a critical subtlety is that verbalised rationales (including chain-of-thought) are generated outputs and can be unfaithful to the underlying computation, so they aid usability more than they guarantee explanation. Interpretability is increasingly framed as an AI safety and alignment tool, verifying internal objectives, detecting deception or bias, auditing capabilities, while remaining an open research frontier. Governance drivers include rights to explanation for automated decisions and high-risk-system transparency obligations.