L20. Incident Response for AI Systems
Course outlineLesson 20 of 21
AI incidents have different characteristics from traditional security incidents: the attacker may have manipulated the system rather than broken into it, the blast radius is often unclear, and containment options are different. IR playbooks must be adapted accordingly.
How AI Incidents Differ
Traditional incident response assumes a clear breach: an attacker gained unauthorized access at a point in time. AI incidents are often more ambiguous:
- No clear breach point: a prompt injection attack may have operated through legitimate API calls, leaving no authentication failure in logs
- Uncertain scope: it may be unclear which sessions were affected, whether data was actually exfiltrated, and what instructions the model followed
- Unusual evidence: the evidence is in conversation logs and tool call logs, not firewall logs and authentication events
- Response options are different: you cannot "quarantine" a model the way you quarantine a compromised endpoint
AI Incident Types and Response Priorities
| Incident Type | Primary Response Priority | Key Evidence |
|---|---|---|
| Prompt injection with data exfiltration | Identify what data left the system and through what channel | Tool call logs, external HTTP requests |
| Safety filter bypass producing harmful content | Preserve evidence, assess distribution, apply emergency mitigation | Conversation logs, safety filter logs |
| Model extraction / systematic probing | Identify scope of probing, assess IP exposure, rate-limit or block source | API access logs, query patterns |
| Training data poisoning discovered in production | Assess affected model versions, determine rollback point | Training pipeline logs, model version history |
| Compromised MCP server | Identify which sessions connected, what the server returned | MCP connection logs, tool call results |
Containment Options
Unlike traditional IR where you can isolate a host, AI containment options include:
- Rate limiting or blocking: block the source IP or user account sending anomalous queries
- Disabling tools: remove specific tool access from an agent that is being abused
- Emergency system prompt update: add mitigations to the system prompt for immediate effect while a proper fix is developed
- Model version rollback: revert to a previous model version if the current version has a critical vulnerability
- Disabling the AI feature: take the AI capability offline entirely if the risk is severe enough
Post-Incident Activities
- Root cause analysis: determine how the incident occurred and what controls failed or were missing
- Evidence preservation: archive conversation logs, tool call records, and system state in an immutable store before they age out of retention
- Stakeholder notification: if user data was disclosed, follow the breach notification obligations applicable to your jurisdiction
- Control improvement: update detection rules, add to regression test suites, improve the relevant layer of defense
- Playbook update: document what worked and what did not in the response for the next incident
- ✓AI incidents often have no clear breach point: prompt injection exploits legitimate API calls, leaving no authentication failure in logs
- ✓Key evidence for AI incidents: conversation logs, tool call logs, external HTTP requests, safety filter trigger logs, and model version history
- ✓AI containment options: rate limit/block source, disable specific tools, emergency system prompt update, model version rollback, feature takedown
- ✓Emergency system prompt changes are temporary mitigations: always follow with root-cause analysis and a proper fix
- ✓Evidence preservation must happen before retention policies delete conversation and tool call logs that are the primary forensic record
1. Why is identifying the "breach point" often difficult in a prompt injection incident?
2. Which containment action provides the fastest mitigation for an active prompt injection attack while a proper fix is developed?
Recommended: Pluralsight
Pluralsight's AI security courses cover threat modeling, governance, and practical red teaming for AI systems to complement what you learn here.