L1. The AI Attack Surface: What Changed and Why
Course outlineLesson 1 of 21
AI systems introduce attack vectors that do not exist in traditional software: they can be manipulated through their inputs, their training data, and the emergent behaviors that arise from learned weights. This lesson frames the new attack surface security architects must account for.
Why AI Changes the Security Equation
Traditional software systems are deterministic: given the same input, you get the same output. The attack surface is well-understood: code execution, memory corruption, authentication bypasses, injection. AI systems break this model.
A large language model's behavior is shaped by billions of learned parameters. That means the attack surface is not just the API endpoint in front of it, it includes every prompt it receives, every document it retrieves, every tool it can call, and the entire dataset it was trained on.
Four New Attack Classes
| Attack Class | What It Targets | Example |
|---|---|---|
| Prompt injection | Inference-time inputs | Malicious text in a retrieved document hijacks the model's response |
| Model theft / extraction | Model weights or behavior | Repeated queries reconstruct proprietary model behavior |
| Training data poisoning | The training pipeline | Attacker-controlled data embedded in the training corpus shifts model behavior |
| Inference-time attacks | Statistical properties | Membership inference reveals whether specific data was in the training set |
Expanded Trust Boundaries
In a conventional web application, trust boundaries are at network edges, authentication layers, and API contracts. In an AI system, trust boundaries include:
- The prompt boundary: anything the model reads is part of the attack surface, including tool outputs, retrieved documents, and user messages
- The tool boundary: when a model can call external APIs or run code, it becomes a potential pivot point for lateral movement
- The training pipeline boundary: the data preprocessing, annotation, and fine-tuning stages are all potential poisoning surfaces
- The deployment boundary: the model itself is an asset that can be stolen via extraction attacks
Architecture Implication
Security architects must treat the model as a semi-trusted component, not a trusted one. Outputs from the model must be validated before they trigger actions, especially in agentic workflows where the model can write files, call APIs, or execute commands.
The key mental model shift: move from "is the API authenticated?" to "can a malicious actor influence what the model decides to do, and what is the blast radius of that influence?"
- ✓AI systems are non-deterministic: the attack surface includes inputs, training data, retrieved context, and tool call results
- ✓Four new attack classes: prompt injection, model theft/extraction, training data poisoning, and inference-time attacks (membership inference)
- ✓Trust boundaries expand beyond network edges to include the prompt boundary, tool boundary, training pipeline, and model weights
- ✓Treat the AI model as a semi-trusted component: validate its outputs before they trigger downstream actions
- ✓Agentic AI dramatically increases blast radius: a compromised model that can call tools or write files is a pivot point
1. Which AI attack class targets the model at training time rather than inference time?
2. What is the most important mental model shift for security architects designing AI systems?
3. In an agentic AI system, why does the tool boundary matter for security architects?
Recommended: Pluralsight
Pluralsight's AI security courses cover threat modeling, governance, and practical red teaming for AI systems to complement what you learn here.