Azure AI Foundry Red Teaming: Evidence-Based Adversarial Testing Workflow

What This Workflow Proves, and What It Cannot Prove

Azure AI Foundry red teaming should produce evidence about the exact application you plan to ship: the model deployment, prompt version, retrieved content paths, tool permissions, guardrail configuration, evaluator version, and remediation decision. It should not be treated as proof that prompt injection risk is absent.

OWASP LLM01:2025 separates prompt injection into direct attacks, where the user attempts to change model behavior through input, and indirect attacks, where instructions arrive through external content such as websites or files. OWASP also warns that RAG and fine-tuning do not fully mitigate prompt injection. That matters for Foundry because a text-only assistant, a RAG assistant, and a tool-using agent do not need the same evidence package.

The goal is not to rank one attack type as universally highest severity. OWASP ties impact to business context and the application's degree of agency. A harmful answer, an unauthorized tool call, a sensitive-data disclosure, and a bypassed approval step are different outcomes. Treat the red-team run as evidence about reachable outcomes, not as a generic score.

Start With an Action-Boundary Threat Model

Before running a scanner, write the prohibited outcomes in application terms:

• The assistant discloses data the current user is not authorized to see.
• The agent calls a tool outside the user's approved task.
• The model produces content that violates the application's policy.
• The workflow skips a human approval requirement for a high-risk action.
• The application accepts untrusted retrieved content as authority over system or developer instructions.

For platform identity, RBAC, and data-governance prerequisites, use the separate Azure AI Foundry security threat model. This article stays at the evaluation and evidence layer. If the deployment also depends on private connectivity, validate the boundary design with the Azure AI Foundry Private Link setup guide before using red-team results as release input.

Choose the Test Path by Capability

Do not start with a generic prompt-injection checklist. Start with what the application can do.

This framing keeps severity tied to evidence. A successful attack prompt can show unsafe output without proving a real security impact. Conversely, a modest-looking prompt can matter if it reaches a privileged tool or bypasses an approval boundary.

Build Owned, Harmless Adversarial Fixtures

Create fixtures for every untrusted input path the application actually processes. Keep them harmless and owned by your team:

• A direct user prompt that requests a prohibited outcome.
• An uploaded document containing instructions that should be treated as untrusted content.
• A retrieved chunk from a controlled RAG corpus.
• A web-result fixture if the application consumes external pages.
• A tool-response fixture if the agent reads tool output before making another decision.

Each fixture should have a revision ID, expected retrieval path, expected containment behavior, and owner. For a RAG assistant, the evidence is incomplete unless you can show that retrieval actually happened and the application still respected the intended boundary. For a tool-using agent, the evidence is incomplete unless the tool permissions, server-side authorization, schema validation, approval requirements, and audit trail were tested after the attack attempt.

Configure and Verify Foundry Guardrails

Microsoft documents Prompt Shields in Foundry for user-prompt attacks and document attacks. Document attacks can arrive through documents, emails, or web pages. That makes Prompt Shields relevant to both direct prompt input and untrusted content flowing into RAG or agentic systems.

Treat Prompt Shields as a control to verify, not as authorization. Microsoft documents configuration dependencies, assignment requirements, intervention points, annotation results, and false positives. Guardrail applicability also differs across model and agent surfaces, and the guardrails overview identifies tool-call and tool-response intervention points as preview and agent-specific.

Use this verification sequence:

1. Assign the guardrail to the deployment or agent where the untrusted content enters.
2. Confirm the intervention point matches the application path being tested.
3. Run in annotation mode where operationally appropriate.
4. Preserve the request ID, guardrail annotation, test fixture revision, and response evidence.
5. Verify the configured action in the real intervention point.
6. Manually review false positives and false negatives before using the result as release evidence.

If you evaluate Spotlighting, document its limitations. Microsoft describes Spotlighting as preview, off by default, available only for Chat Completions API models, and subject to token use and input-size pressure. Microsoft also notes that encoding references can appear in responses. Those tradeoffs belong in the release record, not in a footnote.

Run a Local Baseline Scan, With Preview Limitations

Microsoft's local AI Red Teaming Agent guidance is useful for repeatable baseline evidence, but it must be labeled correctly. The local agent is preview, and Microsoft does not recommend preview features for production workloads. The documented local Red Team API supports single-turn, text-only interactions.

That scope is fine for a text-only baseline. It is not enough for multi-turn agents, multimodal workflows, RAG retrieval validation, or tool-use boundary testing. Use the local run to collect row-level evidence, then add application-level tests for the paths the local API does not cover.

Use the current package names from Foundry guidance:

uv pip install "azure-ai-evaluation[redteam]"
pip install "azure-ai-projects>=2.0.0"

Use the current red-team API shape as a small integration point rather than copying a large orchestration template:

import os
from azure.identity import DefaultAzureCredential
from azure.ai.evaluation.red_team import RedTeam

red_team = RedTeam(
    azure_ai_project=os.environ["AZURE_AI_PROJECT"],
    credential=DefaultAzureCredential(),
)

result = await red_team.scan(target=application_callback)

The documented local example generates 10 attack prompts for each of four default risk categories, for 40 prompts. That number supports repeatability for that documented example only. It does not create a universal threshold, defect-rate target, or production release gate.

Use Cloud Red Teaming for Agentic or Scheduled Evaluation

For broader testing, Microsoft documents a cloud AI Red Teaming Agent workflow that supports pre-deployment larger scans, scheduled post-deployment runs, and agentic scenarios in a minimally sandboxed environment. Cloud red-team runs can use strategies including Flip, Base64, and IndirectJailbreak with a configured turn count and an approved taxonomy.

Use the cloud workflow when the application needs evidence beyond single-turn text:

• The agent can call tools.
• The application has multi-turn behavior that changes state over time.
• The release process requires a larger pre-deployment scan.
• Post-deployment scheduled evaluation is part of the operating model.
• The target taxonomy needs review before execution.

The configured turn count is a test parameter, not a severity formula. Review the row-level attack-response evidence against the prohibited outcomes you wrote at the start.

Validate RAG, Tool, and Approval Boundaries End to End

RAG testing is not complete when a scanner produces an attack prompt. You need to show the full chain:

1. The owned malicious-document fixture was available in the retrieval corpus.
2. The user query retrieved the fixture or relevant chunk.
3. Prompt Shields or other configured controls produced the expected annotation or action at the correct intervention point.
4. The application preserved the system boundary despite the untrusted content.
5. The test record includes the retrieval evidence, response, request ID, fixture revision, and reviewer decision.

Tool-using agents need a different proof. OWASP recommends least privilege, validation outside the model, content separation, human approval for high-risk actions, and adversarial testing. In practice, that means the model should not be the authority for access control. Server-side authorization, schema validation, least-privilege tool scopes, approval requirements, and auditability must still work after a manipulated prompt or tool response.

For approval-boundary tests, a safe test tenant matters. The point is to verify whether an attack attempt can cross an authorization or action boundary, not to run dangerous operations.

Treat Evaluation Output as Evidence

The local red-team scorecard can include row-level attack-response evidence, attack strategy, complexity, risk category, and success assessment. Preserve that detail. A summary score without examples is weak release evidence because it hides false positives, missed detections, and the exact failure mode.

For each finding, record:

• Model deployment and version.
• Prompt version.
• Guardrail configuration and intervention point.
• Fixture revision.
• Evaluator or scanner version.
• Attack strategy and risk category.
• Request ID and response evidence.
• Human reviewer decision.
• Remediation decision.
• Retest result.

Manual review is not optional. A model-evaluator score is evidence, not final adjudication. A clean automated run establishes coverage only for the tested scenarios.

Instrument Production Safely With Foundry Tracing

Replace unsupported endpoint-log assumptions with Foundry tracing through Application Insights and OpenTelemetry. Microsoft documents that Foundry traces can include prompts, outputs, retrieval operations, tool calls, and intermediate steps. That is exactly the evidence you need to reproduce and remediate verified findings, but it can also collect sensitive inputs, outputs, and tool arguments.

Design telemetry controls before broad test logging:

• Redact or minimize sensitive prompt, output, and tool data.
• Restrict Application Insights access to the team that needs investigation evidence.
• Define retention before enabling broad tracing.
• Validate telemetry queries in the reader's tenant and deployment path.
• Keep request IDs linked to fixture revisions and remediation records.

Production monitoring should support reproduction and remediation. Do not rely on fixed request-count heuristics or unsupported behavioral assertions as if they were vendor-backed detection logic.

Release Checklist: Evidence Required for Each Capability

Use this checklist as a release evidence template:

• [ ] Prohibited outcomes are written in application terms.
• [ ] Each untrusted input path has an owned harmless fixture.
• [ ] Prompt Shields or other guardrails are assigned to the correct deployment or agent.
• [ ] Intervention points are verified for the exact model or agent surface.
• [ ] False positives and false negatives are manually reviewed.
• [ ] Local Red Team API results are labeled preview and single-turn text-only.
• [ ] Cloud red-team runs use a reviewed taxonomy and sandboxed target where agentic or scheduled testing is needed.
• [ ] RAG tests prove retrieval occurred and containment held.
• [ ] Tool tests prove least privilege, server-side authorization, schema validation, approval requirements, and auditability.
• [ ] Trace collection has redaction, access control, retention, and tenant-validated queries.
• [ ] Row-level evidence is retained with request IDs and remediation decisions.
• [ ] Retest results are recorded after each material remediation.

Limitations, False Positives, and Retest Triggers

Prompt Shields are probabilistic controls, not authorization. Authorization must remain outside the model. Guardrail support differs across model and agent surfaces, so test the exact deployment and intervention point rather than assuming a setting applies everywhere.

Spotlighting has documented preview, token, context-size, and response-quality tradeoffs. Tracing can collect sensitive data. The local red-team API is preview and single-turn text-only. A successful attack prompt may demonstrate unsafe output without demonstrating a real security impact. A clean run does not prove absence of prompt injection risk.

Retest when a material input changes:

• Model deployment changes.
• System or developer prompt changes.
• Retrieval corpus or fixture set changes.
• Guardrail configuration changes.
• Tool scope, schema, or approval behavior changes.
• The application adds a new untrusted input path.
• Telemetry access, redaction, or retention changes.

Primary Sources

• OWASP LLM01:2025 Prompt Injection
• Prompt Shields in Microsoft Foundry
• Microsoft Foundry guardrails and controls overview
• Run AI Red Teaming Agent locally
• Run AI Red Teaming Agent in the cloud
• Set up tracing for AI agents
• NIST AI 600-1
• microsoft/PyRIT