L2. Model Theft, Poisoning & Inference Attacks
Course outlineLesson 2 of 21
Three attack classes that operate below the application layer: model extraction steals proprietary behavior, data poisoning corrupts training, and membership inference leaks private data. Each requires different defensive controls.
Model Extraction / Theft
Model extraction (also called model stealing) is the process of reconstructing a model's behavior through repeated queries. An attacker sends carefully chosen inputs and records the outputs, then trains a surrogate model to approximate the target.
The threat is primarily intellectual property theft: a proprietary model trained on months of compute and curated data can be replicated at a fraction of the cost. But extraction can also be a precursor to other attacks: once an attacker has a local surrogate, they can run unlimited offline experiments to find jailbreaks and adversarial inputs without triggering your rate limits or logging. Defensive controls:
- Rate limiting and query volume anomaly detection
- Output perturbation (adding calibrated noise without harming utility)
- Watermarking model outputs to detect surrogate training
- Monitoring for systematic input patterns (grid scans, boundary probing)
Training Data Poisoning
Poisoning attacks target the training pipeline. An attacker who can influence training data (through web scraping pipelines, open-source contribution, or supply chain compromise) can embed backdoors or degrade model performance. Backdoor attacks are the most dangerous variant: the attacker embeds a specific trigger pattern (a phrase, token sequence, or image artifact) that causes the model to behave maliciously when the trigger appears at inference time, while behaving normally otherwise.
| Poisoning Variant | Goal | Detection Signal |
|---|---|---|
| Backdoor | Trigger-activated malicious behavior | Anomalous output on specific inputs |
| Label flipping | Degrade classifier accuracy | Performance degradation on validation set |
| Gradient-based | Influence model parameters toward attacker goal | Outlier gradients in training run |
- Data provenance tracking and integrity verification
- Anomaly detection on training data distributions
- Differential privacy during training to limit memorization
- Independent validation sets held out from all data collection pipelines
Membership Inference
Membership inference attacks determine whether a specific data record was in a model's training set. This matters because if sensitive data (patient records, private communications, proprietary contracts) was inadvertently included in training, an attacker can confirm its presence.
The attack works by exploiting the fact that models tend to be more confident on examples they were trained on than on unseen data. A classifier trained on the target model's confidence scores can predict membership with accuracy significantly above random chance. Defensive controls:
- Differential privacy training (adds calibrated noise to gradients, provides formal membership privacy guarantees)
- Data minimization: audit what goes into training pipelines before training begins
- Output confidence suppression: return top-1 predictions without raw probability distributions where possible
- ✓Model extraction reconstructs proprietary model behavior via repeated queries; a surrogate model enables unlimited offline attack research
- ✓Backdoor poisoning embeds a trigger pattern that causes malicious behavior only when the trigger appears at inference time
- ✓Membership inference exploits higher model confidence on training examples to confirm whether sensitive data was in the training set
- ✓Differential privacy training provides formal mathematical guarantees against membership inference by adding calibrated noise to gradients
- ✓Output confidence suppression (returning only top-1 predictions) reduces the signal available for both extraction and membership inference attacks
1. What does a model extraction attack primarily steal?
2. Which defensive control provides formal mathematical privacy guarantees against membership inference?
Recommended: Pluralsight
Pluralsight's AI security courses cover threat modeling, governance, and practical red teaming for AI systems to complement what you learn here.