L16. AI Red Teaming: Goals, Scope & Team Structure
Course outlineLesson 16 of 21
AI red teaming is a structured adversarial testing discipline that identifies risks in AI systems before attackers do. It differs from traditional penetration testing in its goals, methods, and what constitutes a finding.
What AI Red Teaming Is
AI red teaming is the practice of using adversarial techniques to probe AI systems for harmful, unexpected, or unsafe behavior. The goal is not just to find security vulnerabilities (though that is included) but to find any behavior that the deploying organization would not sanction if they saw it.
This distinguishes AI red teaming from traditional penetration testing in an important way: a traditional pentest defines success as gaining unauthorized access. AI red teaming defines success as finding any output the organization would not want produced, which includes: harmful content, biased outputs, privacy leaks, safety bypasses, and security vulnerabilities.
Scope Definition
Before starting an AI red team engagement, the scope must define: Target system components:
- The model and its version
- The system prompt and any prompt templates
- The tools and integrations available to the model
- The user interfaces through which the model is accessed
- Safety harms (instructions for dangerous activities, self-harm content)
- Security harms (prompt injection vectors, data exfiltration, tool abuse)
- Privacy harms (PII disclosure, training data memorization)
- Fairness harms (discriminatory outputs, biased decision-making)
- Integrity harms (misinformation, hallucinated facts presented as authoritative)
- Direct user input manipulation
- Indirect injection via retrieved content
- Multi-turn conversation manipulation
- Tool call manipulation
Team Structure
Effective AI red teams combine three types of expertise:
| Role | Expertise | Contribution |
|---|---|---|
| Domain expert | Knows the deployment context deeply | Identifies what "harm" means for this specific application |
| Security researcher | Traditional offensive security skills | Finds prompt injection, authentication bypasses, tool abuse |
| Social engineer | Human psychology and manipulation | Designs roleplay scenarios, fictional framings, authority claims |
Rules of Engagement
- Document all findings, including failed attempts (they establish what does not work)
- Use a standardized finding template: attack vector, reproduction steps, harm category, severity, suggested mitigation
- Define what constitutes a finding vs. expected behavior
- Establish handling procedures for genuinely sensitive findings (e.g., if the model produces CSAM or detailed weapon instructions, stop and report immediately)
- Debrief the development team: red team findings should drive concrete mitigations, not just reports
- ✓AI red teaming goal: find any output the organization would not sanction, including safety harms, privacy leaks, biased outputs, and security vulnerabilities
- ✓AI red teaming differs from traditional pentesting: success is finding unsanctioned outputs, not just unauthorized access
- ✓Scope must define: target components, harm categories, and attack vectors in scope
- ✓Effective teams combine domain expertise, offensive security skills, and social engineering; diversity of backgrounds finds more harm categories
- ✓Document all findings including failed attempts; use standardized templates with attack vector, reproduction steps, harm category, severity, and suggested mitigation
1. How does AI red teaming differ from traditional penetration testing in its definition of a finding?
2. Why does team diversity matter in AI red teaming?
Recommended: Pluralsight
Pluralsight's AI security courses cover threat modeling, governance, and practical red teaming for AI systems to complement what you learn here.