Leveraging CI/CD for Azure Foundry Agents
Introduction
This is part of a blog series regarding my recent experimentation on creating an Azure Foundry quickstart that allows users to update a .yaml file and deploy a multi-tiered chatbot backed by Azure Foundry. Additionally this quickStart includes:
- Leveraging Azure Deployment Environments as part of the CI/CD process
- Creating a Dev Box that comes preinstalled and configured with all the necessary tools
- Backing the infrastructure by Azure Verified Modules (AVM) when possible
- Partnering with GitHub Coding Agent for content creation
- Diving into MCP to get the most out of GitHub CoPilot
The complete and latest codebase can be found at https://github.com/JFolberth/ai-in-a-box
For the introduction into this series refer to my post: AI-In-A-Box: A Foundry Project Rooted in Developer Technologies
Background
If you managed to attend Microsoft Build `25 you undoubtedly came across a session involving Azure Foundry. Per Microsoft Azure Foundry is the ability
“Create the future of AI using prebuilt and customizable models, tools, and safeguards available in popular developer workspaces.”
This sounds awesome. Even more so if you saw some of the incredible demos where presenters were able to quickly spin up agents and connected them all in the blink of the eye….via the portal. This left the question how could we tap into this power without leveraging the portal. This is a requirement for well built and maintained CI/CD processes.
In addition to configuring and deploying these agents via the Azure portal, several demonstrations showcased creating Foundry Agents directly from application code. Again, is this a best practice? It’s hard to definitively say, as this area is rapidly evolving. In my experience, however, most enterprise customers hesitate to let languages like Python, JavaScript, or C# directly create Azure resources.
Is there another way? The answer is yes.
Foundry
Azure Foundry is actually a type of Cognitive Services offering so as such we are required to create a Cognitive Services Account. To denote this as Foundry the type of account for the Cognitive Services is AIServices.
Based on my experience with Foundry, I recommend developing it similarly to Azure Data Factory or Logic Apps, using either the integrated portal or the VS Code extension. The Foundry portal can be directly accessed via: ai.azure.com
Model
Setting up an Agent requires selecting a Model. Among all components, Models might be the most familiar—and Foundry offers many options. In fact it’s one of the key differentiates of the product. If you are unfamiliar with this then spend some time researching what model to use Azure AI Foundry Models | Microsoft Azure as different models are optimized for different use cases.
A Model is defined in the Azure Resource Manager as a subresource of a Cognitive Services Account. Microsoft.CognitiveServices/accounts/deployments would be the resource type that is required. What’s interesting here is that models is not its own separate type rather of a type of deployments where the model properties are then set. Pay special attention here to your capacity as this is where we need to pay attention around quota settings.
A Model within Foundry, since it is part of the account resource type, can be associated at the project or at the workspace level. Speaking of projects….
Project
Rcently, there’s been a trend with newer Azure Resources such as Azure Deployment Environments, Azure Dev Box, or really any of the products under the Microsoft Dev Portal, there is a logical and security boundary defined as a Project.
Items such as the model, connected services such as Bing Search, and access to the agents can be defined within the Project scope. This means a team within an organization could effectively share a single Foundry instance and rely on the the security of the Project to control access while also being able to share a model across multiple projects.
When bringing this back to the resource provider and IaC this project is defined as Microsoft.CognitiveServices/accounts/projects and the complete schema of features and settings is available.
One piece I will stress that feeds back into the notion of security is that, similar to Dev Center Projects, each project can be configured to leverage a Managed Identity to control access to additional resources that live outside the scope of the project.
Now, how do we perform actions within a Project using a Model in Foundry?.,…
Agent
This is where things diverge slightly from previous components. An agent lives inside a project and leverages a model. The agent is not defined, at this time, via the Azure Resource Manager. The ability to create an agent is only available in the various SDKs.
This explains why the majority of demonstrations out there have the code creating the agent. Within the Azure Portal this is just happening via REST calls with a .json payload:
Thus if we want to automate this outside of the application code we are going to have to still leverage the SDKs to make these calls. This is where we can get a little fancy with some PowerShell or other scripting options.
Defining An Agent As YAML
Now that we have an understanding of the various components we should also understand how deploying an managing an agent differs from other Azure Resources. To this end we need to switch our mentality for deployments from infrastructure centric.
This part I’ve struggled with. Is an Agent code? Is it Infrastructure? Is it Data? Something different? My current thoughts say it could very well end up being something different; however, if I had to pick I’d side with it’s closest to how we handle API deployments.
There are overlapping characteristics. Our Agent, similar to our API, should have the same code across all the environments. It should be housed in a centralized location like a Foundry Project, for APIs this is APIM. Both of these centralized locations would have their own identities and properties that connect to other resources within the environment. So for now I will treat an Agent like an API in API Manager.
Even with this analogy, we typically don’t recommend allowing API source code to directly create backend infrastructure or APIM configurations. (Pause I know you can scream at me that Pulumi allows for this; however, in my experience a lot of large organizations would not be on board for various governance/practices concerns). Upon further digging into how the portal and the REST API works I noticed the Azure AI Foundry Extension for VS Code allows you to define an agent as a YAML file. Great how can we replicate this process to accommodate our deployments?
While doing some digging I stumbled upon the YAML schema definition for the agent. This is a start. I honestly don’t need to define it as YAML as we already discussed there is a REST endpoint and it is expecting a .json payload; however, when possible I like to stick the native tooling format. This approach allows direct interaction with the agent using the Foundry VS Code extension. Therefore, we need to convert the YAML definition into JSON and send it to the same REST endpoint used by the portal and VS Code.
We can accomplish this by examining network traces from the Foundry portal and reverse-engineering the calls. The endpoint for updating an agent will follow the pattern: https://<Foundry-InstanceName>/api/projects/<ProjectName>/assistants/<assistantID>?api-version=APIVersion
If you’re creating an agent for the first time, omitting /assistantID from the endpoint URL will trigger the creation of a new agent.If you’re creating an agent for the first time, omitting /assistantID from the endpoint URL will trigger the creation of a new agent.
So now that there is an outline to this process, what do we need to know? Well for this next part GitHub CoPilot was immensely helpful. We had to take the .yaml file defining our agent and convert it to a .json file. Well luckily I was able to let GitHub CoPilot do it’s thing and build a parsing process out in addition to multiple error checks.
Here’s the end result for a deployment script:
[CmdletBinding()]
param(
[Parameter(Mandatory = $true)]
[string]$AiFoundryEndpoint,
[Parameter(Mandatory = $false)]
[ValidateSet("human", "json")]
[string]$OutputFormat = "human",
[Parameter(Mandatory = $false)]
[string]$AgentYamlPath = "src/agent/ai_in_a_box.yaml",
[Parameter(Mandatory = $false)]
[ValidateSet("Error", "Warning", "Information", "Verbose")]
[string]$LogLevel = "Information",
[Parameter(Mandatory = $false)]
[string]$AgentName = "",
[Parameter(Mandatory = $false)]
[switch]$Force
)
# Set error action preference
$ErrorActionPreference = "Stop"
# Determine the workspace root (project root directory)
$scriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
$workspaceRoot = Split-Path -Parent $scriptDir
# Resolve agent YAML path relative to workspace root
$agentYamlFullPath = Join-Path $workspaceRoot $AgentYamlPath
# Configuration constants
$DefaultModelName = "gpt-4o-mini"
# Initialize logging
function Write-Log {
param(
[string]$Message,
[ValidateSet("Error", "Warning", "Information", "Verbose")]
[string]$Level = "Information"
)
$timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
$logMessage = "[$timestamp] [$Level] $Message"
if ($OutputFormat -eq "human") {
switch ($Level) {
"Error" {
Write-Error $logMessage
}
"Warning" {
if ($LogLevel -in @("Warning", "Information", "Verbose")) {
Write-Warning $logMessage
}
}
"Information" {
if ($LogLevel -in @("Information", "Verbose")) {
Write-Host $logMessage -ForegroundColor Green
}
}
"Verbose" {
if ($LogLevel -eq "Verbose") {
Write-Verbose $logMessage
}
}
}
}
}
# Function to output final result
function Write-Final-Result {
param(
[bool]$Success,
[string]$AgentId = "",
[string]$AgentName = "",
[string]$Endpoint = "",
[string]$ErrorMessage = "",
[string]$Model = "",
[string]$OperationType = ""
)
if ($OutputFormat -eq "json") {
if ($Success) {
$result = @{
success = $true
agentId = $AgentId
agentName = $AgentName
endpoint = $Endpoint
model = $Model
operationType = $OperationType
wasExistingAgent = ($OperationType -eq "updated")
}
}
else {
$result = @{
success = $false
error = "Deployment script failed: $ErrorMessage"
timestamp = Get-Date -Format "yyyy-MM-ddTHH:mm:ssZ"
}
}
$jsonOutput = $result | ConvertTo-Json -Compress
Write-Output "AGENT_DEPLOYMENT_RESULT: $jsonOutput"
}
else {
if ($Success) {
Write-Log "🎉 Agent deployment completed successfully!" -Level "Information"
Write-Log " Agent ID: $AgentId" -Level "Information"
Write-Log " Operation: $OperationType" -Level "Information"
}
else {
Write-Log "💥 Agent deployment failed: $ErrorMessage" -Level "Error"
}
}
}
Write-Log "🚀 Starting AI Foundry agent deployment" -Level "Information"
try {
# =========== VALIDATION ===========
Write-Log "🔍 Validating prerequisites..." -Level "Information"
# Validate YAML file exists
if (-not (Test-Path $agentYamlFullPath)) {
throw "Agent YAML file not found: $agentYamlFullPath"
}
# Read YAML content
try {
$yamlContent = Get-Content -Path $agentYamlFullPath -Raw
if ([string]::IsNullOrEmpty($yamlContent)) {
throw "YAML file is empty"
}
Write-Log "✅ YAML file loaded successfully" -Level "Information"
}
catch {
throw "Failed to read YAML file: $($_.Exception.Message)"
}
# Validate endpoint URL format
try {
$uri = [System.Uri]$AiFoundryEndpoint
if ($uri.Scheme -notin @('http', 'https')) {
throw "Invalid endpoint scheme. Must be http or https."
}
Write-Log "✅ Endpoint URL format is valid" -Level "Information"
}
catch {
Write-Log "❌ Invalid AI Foundry endpoint URL: $AiFoundryEndpoint" -Level "Error"
throw "Invalid AI Foundry endpoint URL format: $($_.Exception.Message)"
}
# Function to get access token for API calls
function Get-AccessToken {
try {
# Try managed identity first
$tokenUri = "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://ai.azure.com/"
$headers = @{ 'Metadata' = 'true' }
$tokenResponse = Invoke-RestMethod -Uri $tokenUri -Headers $headers -Method Get -TimeoutSec 10
Write-Log "✅ Retrieved access token via managed identity" -Level "Verbose"
return $tokenResponse.access_token
}
catch {
Write-Log "⚠️ Managed identity token retrieval failed, trying Azure CLI..." -Level "Verbose"
# Fallback to Azure CLI
try {
$token = az account get-access-token --scope "https://ai.azure.com/.default" --query "accessToken" -o tsv 2>$null
if ($LASTEXITCODE -eq 0 -and -not [string]::IsNullOrEmpty($token)) {
Write-Log "✅ Retrieved access token via Azure CLI" -Level "Verbose"
return $token
}
else {
throw "Azure CLI token retrieval failed"
}
}
catch {
Write-Log "❌ Failed to get access token from Azure CLI: $($_.Exception.Message)" -Level "Error"
throw "Unable to retrieve access token. Ensure you're authenticated with Azure CLI or running with managed identity."
}
}
}
# Function to parse YAML agent configuration
function Read-AgentYaml {
param([string]$YamlContent)
try {
Write-Log "✅ Processing YAML content ($($YamlContent.Length) characters)" -Level "Information"
# Parse YAML fields using regex
$name = if ($YamlContent -match 'name:\s*(.+)') { $matches[1].Trim() } else { "" }
$version = if ($YamlContent -match 'version:\s*(.+)') { $matches[1].Trim() } else { "1.0.0" }
# Extract description (handle multiline literal scalar with |)
$description = ""
if ($YamlContent -match 'description:\s*\|\s*\r?\n((?:\s{2}.*\r?\n?)*?)(?=\r?\n\s*#|\r?\n\s*[a-zA-Z_]+:|$)') {
$description = $matches[1] -replace '^\s{2}', '' -replace '\r?\n\s{2}', "`n" -replace '\r?\n$', ''
$description = $description.Trim()
}
elseif ($YamlContent -match 'description:\s*"([^"]*(?:\\.[^"]*)*)"') {
$description = $matches[1] -replace '\\r\\n', "`r`n" -replace '\\n', "`n" -replace '\\"', '"'
}
elseif ($YamlContent -match 'description:\s*(.+)') {
$description = $matches[1].Trim()
}
# Extract instructions (handle multiline literal scalar with |)
$instructions = ""
if ($YamlContent -match 'instructions:\s*\|\s*\r?\n((?:\s{2}(?!#).*\r?\n?)*?)(?=\r?\n\s*[a-zA-Z_]+:|$)') {
$instructions = $matches[1] -replace '^\s{2}', '' -replace '\r?\n\s{2}', "`n" -replace '\r?\n$', ''
$instructions = $instructions.Trim()
}
elseif ($YamlContent -match 'instructions:\s*"([^"]*(?:\\.[^"]*)*)"') {
$instructions = $matches[1] -replace '\\r\\n', "`r`n" -replace '\\n', "`n" -replace '\\"', '"'
}
elseif ($YamlContent -match 'instructions:\s*(.+)') {
$instructions = $matches[1].Trim()
}
# Extract model configuration
$modelId = $DefaultModelName # Default fallback
if ($YamlContent -match 'model:\s*\n\s*#[^\n]*\n\s*id:\s*(.+)') {
$modelId = $matches[1].Trim()
}
elseif ($YamlContent -match 'model:\s*\n\s*id:\s*(.+)') {
$modelId = $matches[1].Trim()
}
# Extract model options if present
$temperature = 1
$topP = 1
if ($YamlContent -match 'temperature:\s*(.+)') {
$temperature = [double]$matches[1].Trim()
}
if ($YamlContent -match 'top_p:\s*(.+)') {
$topP = [double]$matches[1].Trim()
}
# Extract tools
$tools = @()
$result = @{
name = $name
description = $description
instructions = $instructions
model = $modelId
temperature = $temperature
topP = $topP
tools = $tools
version = $version
}
Write-Log "✅ Parsed agent configuration:" -Level "Information"
Write-Log " 📝 Name: $name" -Level "Information"
Write-Log " 🤖 Model: $modelId" -Level "Information"
Write-Log " 🌡️ Temperature: $temperature" -Level "Information"
Write-Log " 🎯 Top_p: $topP" -Level "Information"
Write-Log " 📋 Instructions length: $($instructions.Length) characters" -Level "Information"
return $result
}
catch {
throw "Failed to parse YAML content: $($_.Exception.Message)"
}
}
# =========== READ AGENT CONFIGURATION FROM YAML ===========
Write-Log "📖 Reading agent configuration from YAML..." -Level "Information"
$agentConfig = Read-AgentYaml -YamlContent $yamlContent
# Extract configuration values (allow override from parameter)
$finalAgentName = if (-not [string]::IsNullOrEmpty($AgentName)) { $AgentName } elseif (-not [string]::IsNullOrEmpty($agentConfig.name)) { $agentConfig.name } else { "AI in A Box Agent" }
$agentDescription = if (-not [string]::IsNullOrEmpty($agentConfig.description)) { $agentConfig.description } else { "AI in A Box intelligent assistant agent" }
$agentInstructions = $agentConfig.instructions
$modelName = if (-not [string]::IsNullOrEmpty($agentConfig.model)) { $agentConfig.model } else { $DefaultModelName }
# Validate essential configuration
if ([string]::IsNullOrEmpty($agentInstructions)) {
throw "Agent instructions are empty in YAML file"
}
if ([string]::IsNullOrEmpty($finalAgentName)) {
throw "Agent name is empty in YAML file"
}
Write-Log "✅ Agent configuration loaded successfully" -Level "Information"
Write-Log " Agent Name: $finalAgentName" -Level "Information"
Write-Log " Model: $modelName" -Level "Information"
# =========== GET ACCESS TOKEN ===========
Write-Log "🔑 Obtaining access token..." -Level "Information"
$accessToken = Get-AccessToken
if ([string]::IsNullOrEmpty($accessToken)) {
throw "Failed to obtain access token"
}
Write-Log "✅ Authentication successful" -Level "Information"
# =========== PREPARE AGENT PAYLOAD ===========
Write-Log "📦 Preparing agent payload..." -Level "Information"
$agentPayload = @{
name = $finalAgentName
description = $agentDescription
instructions = $agentInstructions
model = $modelName
temperature = $agentConfig.temperature
top_p = $agentConfig.topP
tools = $agentConfig.tools
metadata = @{
created_by = "deploy-agent-script"
created_date = (Get-Date -Format "yyyy-MM-ddTHH:mm:ssZ")
yaml_version = $agentConfig.version
deployment_method = "unified-script"
}
}
# Include model options (temperature, top_p) which are supported by the Azure AI Foundry API
# These parameters control response randomness and token selection behavior
$agentPayloadJson = $agentPayload | ConvertTo-Json -Depth 10
Write-Log "✅ Agent payload prepared" -Level "Information"
if ($LogLevel -eq "Verbose") {
Write-Log "🔍 DEBUG: Full payload being sent:" -Level "Information"
Write-Log "$agentPayloadJson" -Level "Information"
}
# =========== CHECK FOR EXISTING AGENT ===========
Write-Log "🔍 Checking for existing agent..." -Level "Information"
# Prepare headers
$headers = @{
'Authorization' = "Bearer $accessToken"
'Content-Type' = 'application/json'
'User-Agent' = 'AI-Foundry-Deploy-Agent-Unified/4.0'
}
# Construct the assistants API endpoint
$cleanEndpoint = $AiFoundryEndpoint.TrimEnd('/')
$agentsEndpoint = "$cleanEndpoint/assistants?api-version=2025-05-15-preview"
Write-Log "📡 Final agents endpoint: '$agentsEndpoint'" -Level "Information"
$existingAgent = $null
try {
Write-Log "🔍 Attempting to list existing assistants..." -Level "Information"
$existingAgents = Invoke-RestMethod -Uri $agentsEndpoint -Method Get -Headers $headers -ErrorAction Stop
if ($existingAgents -and $existingAgents.data) {
$existingAgent = $existingAgents.data | Where-Object { $_.name -eq $finalAgentName } | Select-Object -First 1
if ($existingAgent) {
Write-Log "✅ Found existing agent: $($existingAgent.id)" -Level "Information"
}
else {
Write-Log "ℹ️ No existing agent found with name '$finalAgentName'" -Level "Information"
}
}
}
catch {
Write-Log "⚠️ Could not check for existing assistants: $($_.Exception.Message)" -Level "Warning"
Write-Log "🔄 Proceeding with agent creation..." -Level "Information"
}
# =========== CREATE OR UPDATE AGENT ===========
$response = $null
$isUpdate = $false
try {
if ($existingAgent) {
Write-Log "🔄 Updating existing AI Foundry agent..." -Level "Information"
try {
# Update existing agent
$updateEndpoint = "$cleanEndpoint/assistants/$($existingAgent.id)?api-version=2025-05-01"
$webResponse = Invoke-WebRequest -Uri $updateEndpoint -Method Post -Body $agentPayloadJson -Headers $headers -ContentType "application/json" -ErrorAction Stop
$response = $webResponse.Content | ConvertFrom-Json
$isUpdate = $true
Write-Log "✅ Successfully updated existing agent" -Level "Information"
}
catch {
Write-Log "⚠️ Failed to update existing agent: $($_.Exception.Message)" -Level "Warning"
Write-Log "🔄 Attempting to create new agent instead..." -Level "Information"
# If update fails, try to create new agent
$webResponse = Invoke-WebRequest -Uri $agentsEndpoint -Method Post -Body $agentPayloadJson -Headers $headers -ContentType "application/json" -ErrorAction Stop
$response = $webResponse.Content | ConvertFrom-Json
$isUpdate = $false
}
}
else {
Write-Log "🤖 Creating new AI Foundry agent..." -Level "Information"
# Create new agent with detailed error handling
try {
$ProgressPreference = 'SilentlyContinue' # Suppress progress bars
$webResponse = Invoke-WebRequest -Uri $agentsEndpoint -Method Post -Body $agentPayloadJson -Headers $headers -ContentType "application/json" -ErrorAction Stop
$response = $webResponse.Content | ConvertFrom-Json
$isUpdate = $false
Write-Log "✅ Successfully created new agent" -Level "Information"
}
catch {
Write-Log "🔍 Full Exception Details:" -Level "Error"
Write-Log " Type: $($_.Exception.GetType().FullName)" -Level "Error"
Write-Log " Message: $($_.Exception.Message)" -Level "Error"
if ($_.ErrorDetails) {
Write-Log "🔍 PowerShell ErrorDetails: $($_.ErrorDetails)" -Level "Error"
}
throw $_.Exception.Message
}
}
if ($response -and $response.id) {
$agentId = $response.id
$operationType = if ($isUpdate) { "updated" } else { "created" }
Write-Log "✅ Successfully $operationType agent with ID: $agentId" -Level "Information"
Write-Log "🎯 Agent name: $($response.name)" -Level "Information"
# Output results
Write-Final-Result -Success $true -AgentId $agentId -AgentName $response.name -Endpoint $AiFoundryEndpoint -Model $response.model -OperationType $operationType
}
else {
throw "Agent operation succeeded but no agent ID returned in response"
}
}
catch {
$errorDetails = $_.Exception.Message
Write-Log "❌ Failed to create agent: $errorDetails" -Level "Error"
throw $errorDetails
}
}
catch {
$errorMessage = $_.Exception.Message
Write-Final-Result -Success $false -ErrorMessage $errorMessage
exit 1
}
finally {
Write-Log "🏁 Agent deployment script completed" -Level "Information"
}
For the latest version refer to my GitHub containing this script.
Conclusion
Effectively this script will take a yaml file that defines the agent, the endpoint to deploy the agent, and the agent ID if one does already exists. Once the script has gathered this information, it initiates the deployment. This means we can incorporate this script into our CI/CD process to deploy an Agent, defined as yaml, across multiple environments.
Discover more from John Folberth
Subscribe to get the latest posts sent to your email.