Risks & Governance
Why AI hallucinates, and how to work around it
AI's most important flaw is that it can be confidently wrong. Understanding why, it optimises for plausible text, not truth, tells you exactly how to protect yourself.
Sometimes an AI tells you something that sounds completely convincing, and is simply false: a made-up fact, a fake quote, a book that does not exist. This is called a hallucination, and it is the single most important thing to understand about AI.
Here is why it happens. An AI chatbot is not looking up facts in a database. It writes by predicting what words most likely come next, based on patterns it learned. It has no built-in sense of what is true. So when it does not "know", it does not stop, it produces something that sounds right. Usually that is fine. Sometimes it is confidently wrong.
You cannot fully switch this off, but you can protect yourself: - Give it the facts. If you paste the real document and ask it to answer from that, it has far less room to invent. - Ask for sources, and then actually check them. - Verify anything that matters, numbers, names, dates, legal or medical points, before you rely on it.
Think of AI as a brilliant, fast, but sometimes unreliable assistant. You are still the editor.
A hallucination is a confident but false or unsupported answer. It is not a bug that will be patched away; it follows from how language models work.
Why it is built in
A model is trained to produce likely text, not verified text. It generates one word at a time based on patterns, with no internal fact-checker and no reliable sense of its own uncertainty. A fluent, well-formed sentence can therefore be completely wrong, and the model states it just as confidently as a correct one.
Common forms
- Invented facts, dates or statistics.
- Fake citations, quotes or references.
- Plausible but incorrect answers, especially on niche topics where its training data was thin.
How to reduce it
- Ground it with real information (RAG): have it answer from provided documents, not memory alone. This is the single most effective fix.
- Prompt well (prompt engineering): give context, ask for sources, and explicitly allow "I don't know".
- Use tools: let it use a calculator or search instead of guessing.
- Lower the temperature for factual tasks to reduce random wandering.
How to catch it
Always verify facts, figures and citations for anything consequential. Cross-check against a trusted source. The golden rule: AI drafts, you verify.
Hallucination, generated content unfaithful to source or world facts despite fluency, is a structural consequence of likelihood-maximising generation over a parametric LLM, which optimises plausibility, not veracity, and lacks a calibrated notion of its own uncertainty (Ji et al. 2023). A common taxonomy separates intrinsic (contradicting provided input) from extrinsic (unverifiable against input) hallucination.
Contributing factors
- Objective mismatch: next-token likelihood ≠ truth.
- Knowledge gaps and staleness: thin or outdated training coverage; frozen knowledge cut-off.
- Decoding: higher temperature/sampling increases the chance of low-probability, unsupported tokens.
- Alignment side-effects: helpfulness pressure and, sometimes, sycophancy can push a model to answer rather than abstain; alignment can improve or, if mis-tuned, worsen calibration.
Mitigations (layered, not a cure)
- Grounding: retrieval-augmented generation conditions output on verifiable passages and enables attribution (Lewis et al. 2020); tool use offloads facts/computation to reliable systems.
- Decoding & self-checking: lower temperature for faithfulness; self-consistency and verification passes.
- Alignment for calibration: training that rewards well-calibrated abstention ("I don't know").
- Prompting & context: supply context, request citations, constrain scope (prompt engineering/context engineering).
- Post-hoc verification: faithfulness/attribution checks, and human review for high stakes.
The bottom line
None of these eliminate hallucination; they reduce frequency and make it catchable. Evaluation should therefore measure faithfulness and attribution, not fluency, and system design should assume the model can be wrong, keeping human oversight and grounding wherever correctness matters. Treating the model as a conditional text sampler, not a knowledge base, is the mental model that makes this failure predictable and manageable.