Training
Red-teaming
Also known as: adversarial testing
Deliberately attacking or stress-testing an AI to find its weaknesses, harmful outputs, jailbreaks, failures, before real users or bad actors do.
Red-teaming is hiring people (and increasingly other AIs) to try to break an AI, to make it say harmful things, get around its rules, or fail, so those problems can be fixed before release. It borrows the idea from security, where a "red team" plays the attacker. It is a key part of making AI safer: you cannot fix weaknesses you have not found.
Red-teaming is the practice of adversarially probing an AI system to surface harmful, unsafe or unintended behaviour: jailbreaks, harmful content, bias, privacy leaks, or prompt-injection vulnerabilities. Findings feed back into training and guardrails.
How it works
- Human red teams: experts and testers try to elicit failures.
- Automated red-teaming: other models generate attacks at scale.
- Results guide alignment training, filters and policies.
Why it matters
Safety cannot be assumed; it has to be tested. Red-teaming is a core step in responsible model development and pre-deployment evaluation, and is increasingly expected (or required) for powerful frontier models.
Red-teaming is structured adversarial evaluation aimed at eliciting harmful, policy-violating or failure behaviours to inform mitigation, spanning manual expert probing, crowdsourced testing, and automated/model-generated attacks that scale coverage (Ganguli et al. 2022). It targets jailbreaks, harmful-content generation, prompt injection, privacy/data leakage, bias, and dangerous capabilities, with results feeding alignment training, classifier guardrails and deployment policy.
It is a central component of responsible-scaling and pre-deployment safety processes and of external evaluation by safety institutes for frontier models. Methodological considerations include coverage and diversity of attacks, reproducibility, quantifying residual risk (absence of found failures is not proof of safety), and the arms-race dynamic as models and attacks co-evolve. Automated red-teaming and adversarial-training loops improve scale but can overfit to known attack templates, so human creativity and continual, updated testing remain essential.