Risks & safety
Prompt Injection
Also known as: prompt injection attack
A security attack where hidden or malicious instructions in the input trick an AI into ignoring its real instructions and doing something it shouldn't.
Prompt injection is a way of tricking an AI by sneaking instructions into whatever it reads. Because an AI treats all the text it sees as potential instructions, a malicious message, or even hidden text on a web page the AI is asked to summarise, can say "ignore your rules and do this instead". It is a serious and still-unsolved security problem, especially for AI that can take actions or browse the web.
Prompt injection is an attack that exploits a basic fact: a language model cannot fully separate its trusted instructions (system prompt) from untrusted content it processes. Malicious instructions placed in that content can hijack the model's behaviour.
Two forms
- Direct: a user types instructions designed to override the system prompt (overlaps with jailbreaking).
- Indirect: malicious instructions are hidden in external data the AI reads, a web page, an email, a document, so the AI is compromised without the user doing anything wrong.
Why it is dangerous
For an AI agent that can send emails, run code or access data, a successful injection can cause real harm: leaking secrets, taking unwanted actions. It is considered one of the top security risks for AI applications.
Prompt injection arises because instructions and data occupy the same untyped token channel, so a model cannot reliably distinguish developer/system intent from adversarial content in its context. Indirect prompt injection embeds payloads in third-party data the model ingests (retrieved documents, web pages, tool outputs), compromising LLM-integrated applications without direct attacker access (Greshake et al. 2023); direct injection targets the user channel.
Impact scales with capability and privilege: for agents with tools, consequences include data exfiltration, unauthorised actions and lateral movement. There is no complete fix; layered mitigations include instruction-hierarchy training, strict separation and labelling of trusted vs untrusted content, input/output guardrails and content sanitisation, least-privilege and human-in-the-loop confirmation for side-effecting tools, and sandboxing. It is closely related to but distinct from jailbreaking (which targets safety policy) and is widely regarded as a foundational, unsolved LLM-security problem.