How to Secure Azure OpenAI Network Traffic: A Private Endpoint & Terraform Guide

The Problem: Public Network Exposure

If you're running Azure OpenAI in production, there's a good chance your API calls are traversing the public internet right now. Even with API keys and Azure AD authentication, this exposes your organization to risks:

• Data exfiltration: Sensitive prompts and responses travel over public networks
• Man-in-the-middle attacks: Despite TLS, public endpoints increase attack surface
• Compliance violations: Many regulations require private network paths for sensitive data
• Service tags aren't enough: While Azure service tags help with firewall rules, traffic still flows through public endpoints

The solution? Azure Private Link with Private Endpoints. This keeps all traffic on Microsoft's backbone network, never touching the public internet.

Architecture Overview (The Protego Approach)

Here's what we're building:

The flow: Your application (VM, App Service, AKS) connects to the Private Endpoint IP address. DNS resolves your Azure OpenAI hostname to that private IP. Traffic never leaves the Microsoft backbone.

Prerequisites

Before we start, ensure you have:

• Terraform installed (v1.5+)
• Azure CLI authenticated (az login)
• An existing Virtual Network, or we'll create one
• Appropriate Azure permissions (Contributor + User Access Administrator)

Step 1: Defining the Azure OpenAI Resource (Terraform)

Let's start with the Cognitive Services account for Azure OpenAI. The critical security setting is public_network_access_enabled = false.

providers.tf:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.85"
    }
  }
}

provider "azurerm" {
  features {}
}

variables.tf:

variable "resource_group_name" {
  description = "Name of the resource group"
  type        = string
  default     = "rg-openai-private"
}

variable "location" {
  description = "Azure region for resources"
  type        = string
  default     = "eastus"
}

variable "openai_name" {
  description = "Name of the Azure OpenAI resource"
  type        = string
  default     = "openai-private-demo"
}

variable "vnet_address_space" {
  description = "Address space for the VNet"
  type        = list(string)
  default     = ["10.0.0.0/16"]
}

main.tf:

resource "azurerm_resource_group" "main" {
  name     = var.resource_group_name
  location = var.location
}

# Virtual Network
resource "azurerm_virtual_network" "main" {
  name                = "vnet-openai-private"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  address_space       = var.vnet_address_space
}

# Subnet for Private Endpoints
resource "azurerm_subnet" "private_endpoints" {
  name                 = "snet-private-endpoints"
  resource_group_name  = azurerm_resource_group.main.name
  virtual_network_name = azurerm_virtual_network.main.name
  address_prefixes     = ["10.0.1.0/24"]
}

# Subnet for compute resources (VMs, App Services, etc.)
resource "azurerm_subnet" "compute" {
  name                 = "snet-compute"
  resource_group_name  = azurerm_resource_group.main.name
  virtual_network_name = azurerm_virtual_network.main.name
  address_prefixes     = ["10.0.2.0/24"]
}

# Azure OpenAI Service
resource "azurerm_cognitive_account" "openai" {
  name                  = var.openai_name
  location              = azurerm_resource_group.main.location
  resource_group_name   = azurerm_resource_group.main.name
  kind                  = "OpenAI"
  sku_name              = "S0"
  custom_subdomain_name = var.openai_name

  # CRITICAL: Disable public network access
  public_network_access_enabled = false

  network_acls {
    default_action = "Deny"
  }

  identity {
    type = "SystemAssigned"
  }

  tags = {
    environment = "production"
    managed_by  = "terraform"
  }
}

# Deploy a GPT model
resource "azurerm_cognitive_deployment" "gpt4" {
  name                 = "gpt-4"
  cognitive_account_id = azurerm_cognitive_account.openai.id

  model {
    format  = "OpenAI"
    name    = "gpt-4"
    version = "turbo-2024-04-09"
  }

  scale {
    type     = "Standard"
    capacity = 10
  }
}

Critical Note: The public_network_access_enabled = false setting is essential. Without this, your Azure OpenAI resource will still accept connections from the public internet, defeating the purpose of your Private Endpoint.

Step 2: Configuring the Private Endpoint and DNS Zone

This is where most architects struggle. The Private Endpoint alone isn't enough - you need proper DNS configuration so your applications resolve the Azure OpenAI hostname to the private IP.

private-endpoint.tf:

# Private DNS Zone for Azure OpenAI
resource "azurerm_private_dns_zone" "openai" {
  name                = "privatelink.openai.azure.com"
  resource_group_name = azurerm_resource_group.main.name
}

# Link the DNS Zone to your VNet
resource "azurerm_private_dns_zone_virtual_network_link" "openai" {
  name                  = "openai-dns-link"
  resource_group_name   = azurerm_resource_group.main.name
  private_dns_zone_name = azurerm_private_dns_zone.openai.name
  virtual_network_id    = azurerm_virtual_network.main.id
  registration_enabled  = false
}

# Private Endpoint for Azure OpenAI
resource "azurerm_private_endpoint" "openai" {
  name                = "pe-openai"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  subnet_id           = azurerm_subnet.private_endpoints.id

  private_service_connection {
    name                           = "openai-privateserviceconnection"
    private_connection_resource_id = azurerm_cognitive_account.openai.id
    is_manual_connection           = false
    subresource_names              = ["account"]
  }

  private_dns_zone_group {
    name                 = "openai-dns-zone-group"
    private_dns_zone_ids = [azurerm_private_dns_zone.openai.id]
  }

  tags = {
    environment = "production"
    managed_by  = "terraform"
  }
}

Why DNS is Painful Here

When you create a Private Endpoint, Azure assigns it a private IP (e.g., 10.0.1.4). But your application still calls your-openai.openai.azure.com. Without proper DNS:

1. Public DNS returns the public IP
2. Your request goes to the public endpoint
3. Azure OpenAI rejects it (public access is disabled)
4. You get a 403 Forbidden error

The privatelink.openai.azure.com Private DNS Zone solves this:

1. Your VNet is linked to the Private DNS Zone
2. DNS queries from within the VNet hit this zone first
3. It returns the private IP for your Azure OpenAI resource
4. Traffic flows privately through the Private Endpoint

Step 3: Verifying Connectivity (The CloudOps Check)

Don't just deploy and assume it works. Here's how to verify your traffic is truly private.

Deploy a Test VM

test-vm.tf (optional - for verification):

resource "azurerm_network_interface" "test" {
  name                = "nic-test-vm"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name

  ip_configuration {
    name                          = "internal"
    subnet_id                     = azurerm_subnet.compute.id
    private_ip_address_allocation = "Dynamic"
  }
}

resource "azurerm_linux_virtual_machine" "test" {
  name                = "vm-test-openai"
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location
  size                = "Standard_B2s"
  admin_username      = "azureuser"
  network_interface_ids = [
    azurerm_network_interface.test.id,
  ]

  admin_ssh_key {
    username   = "azureuser"
    public_key = file("~/.ssh/id_rsa.pub")
  }

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }

  source_image_reference {
    publisher = "Canonical"
    offer     = "0001-com-ubuntu-server-jammy"
    sku       = "22_04-lts"
    version   = "latest"
  }
}

Verification Steps

SSH into your VM and run these commands:

1. Verify DNS Resolution

nslookup your-openai.openai.azure.com

Expected output (private IP):

Server:    168.63.129.16
Address:   168.63.129.16#53

Non-authoritative answer:
your-openai.openai.azure.com canonical name = your-openai.privatelink.openai.azure.com.
Name:      your-openai.privatelink.openai.azure.com
Address:   10.0.1.4

If you see a public IP, your DNS configuration is wrong.

2. Test Connectivity

curl -I https://your-openai.openai.azure.com/

You should get a response (even if it's a 401 Unauthorized - that means connectivity works, you just need authentication).

3. Test API Call

curl https://your-openai.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview \
  -H "Content-Type: application/json" \
  -H "api-key: YOUR_API_KEY" \
  -d '{"messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 50}'

Common Errors and Troubleshooting

Error: 403 Forbidden

Cause: Traffic is hitting the public endpoint, which is disabled.

Fix:

• Verify the Private DNS Zone exists and is linked to your VNet
• Check that DNS resolution returns a private IP (10.x.x.x)
• Ensure your VM/application is in a subnet that's linked to the DNS zone

Error: DNS Resolution Failed

Cause: Private DNS Zone not linked to VNet, or VNet link not active.

Fix:

az network private-dns link vnet list \
  --resource-group rg-openai-private \
  --zone-name privatelink.openai.azure.com

Verify the link shows provisioningState: Succeeded.

Error: Virtual Network Link Missing

Cause: Terraform apply didn't create the VNet link, or it was deleted.

Fix:

az network private-dns link vnet create \
  --resource-group rg-openai-private \
  --zone-name privatelink.openai.azure.com \
  --name openai-vnet-link \
  --virtual-network vnet-openai-private \
  --registration-enabled false

Error: Connection Timeout

Cause: NSG rules blocking traffic to the Private Endpoint subnet.

Fix: Ensure your NSG allows outbound traffic to the Private Endpoint subnet on port 443.

Outputs

Add these outputs to easily retrieve important values:

outputs.tf:

output "openai_endpoint" {
  description = "Azure OpenAI endpoint URL"
  value       = azurerm_cognitive_account.openai.endpoint
}

output "private_endpoint_ip" {
  description = "Private IP of the Azure OpenAI Private Endpoint"
  value       = azurerm_private_endpoint.openai.private_service_connection[0].private_ip_address
}

output "openai_id" {
  description = "Resource ID of Azure OpenAI"
  value       = azurerm_cognitive_account.openai.id
}

Conclusion

By implementing Private Endpoints for Azure OpenAI, you achieve:

• Zero public internet exposure: All traffic stays on Microsoft's backbone
• Compliance alignment: Meet requirements for private data paths
• Reduced attack surface: No public endpoint to target
• Infrastructure as Code: Reproducible, auditable deployments

The complete Terraform configuration creates everything you need: VNet, subnets, Azure OpenAI with public access disabled, Private Endpoint, and Private DNS Zone with VNet link.

Next steps:

1. Clone this configuration to your repository
2. Customize variables for your environment
3. Run terraform plan to review changes
4. Deploy with terraform apply
5. Verify with nslookup from inside your VNet

For the complete Terraform files, check out the examples in this guide and adapt them to your specific requirements.