L18. AI Evaluation Frameworks: Metrics & Defect Rates

Why Metrics Matter

"The AI is safe" is not a deployable statement. "The AI produces harmful output on fewer than 3% of adversarial inputs in the violence category, measured against our evaluation set of 5,000 inputs" is a deployable statement.

Evaluation frameworks create the measurement infrastructure that turns AI security from an art into an engineering discipline.

Defect Rate as the Core Metric

The defect rate for a given harm category is the fraction of test inputs in that category that produced a policy-violating output:

defect_rate = policy_violating_outputs / total_inputs_in_category

Microsoft's Azure AI Foundry Evaluation framework, NIST's ARIA program, and Anthropic's internal evaluations all use variants of this metric. Common target thresholds before production deployment:

These thresholds should be defined by the organization before testing begins and tied to the deployment's specific risk profile. A children's education platform needs stricter thresholds than an internal developer tool.

Building an Evaluation Set

An evaluation set is a curated collection of test inputs with expected outputs (or expected output classifications). For AI security:

• Size: minimum 200-500 inputs per harm category for statistical reliability; 1,000+ for high-stakes deployments
• Diversity: inputs should vary in framing, language, complexity, and attack technique
• Ground truth: each input should have a clear expected classification (safe / policy-violating)
• Independence: the evaluation set must not overlap with the training or fine-tuning data

Evaluation set maintenance:

• Refresh periodically as new attack techniques emerge
• Add found jailbreaks from red team exercises
• Version the evaluation set alongside model versions

Automated Evaluation Pipelines

Running evaluations manually does not scale. An automated evaluation pipeline:

1. Triggers on model updates, system prompt changes, or on a defined schedule
2. Runs the full evaluation set against the current model configuration
3. Calculates defect rates per harm category
4. Compares to defined thresholds and previous baseline
5. Blocks deployment if thresholds are exceeded; alerts on regression from previous baseline
6. Stores results in a versioned evaluation history for audit

Evaluation Limitations

No evaluation framework provides complete coverage. Key limitations:

• Evaluation sets capture known attack patterns; novel attacks are not in the set
• Automated judges (using another LLM to classify outputs) introduce their own error rates
• Real-world distribution of user inputs differs from curated evaluation inputs
• Defect rates on the evaluation set may not accurately predict defect rates in production