Azure DevOps, Professional, YAML Pipelines

Deploy Terraform to AWS w/ Microsoft Azure DevOps

by John Folberth

Introduction

That is correct you read accurately. This post will guide you through how to deploy AWS resources via Azure DevOps. This feels like a bit of a mashup; however, in this modern age organizations constantly crave the flexibility and capability of being able to deploy their resources to multiple clouds.

Background

We are going to build a simple Azure DevOps pipeline to deploy an AWS S3 account. For those unfamiliar with AWS an S3 (Simple Storage Solution) is the comparable service to Azure Storage Accounts. We are also going to leverage the Azure Pipelines Terraform Task found on the market place and one I have written about before, Azure DevOps Terraform Task.

This process will involve a remote state file which will also be contained in an additional S3 bucket.

If you follow me then you will also know that a static one use pipeline has a limited use. We will write this in such a way to introduce new task, job, and stage YAML Pipelines into my TheYAMLPipeline Repository. If this is new to you free to check out my additional posts on my blog and/or the Microsoft Community series of blog posts on the topic.

I am gong to exclude how to write an S3 Terraform block as this is something that can easily be deciphered from Hashicorp’s documentation. My one piece of advice is the provider will be “aws” and the backend will be “s3”

Getting Started

Fortunately, for us, the Marketplace task supports BOTH AWS and Azure (as well as GCP….maybe will do that one day). So for this to work we will require the following to be in place:

AWS Toolkit for Azure DevOps

Once we have the AWS Toolkit for Azure DevOps marketplace extension in stalled, we will have ability to create a Service Connection to AWS:

Now I am only going to use the Access Key ID and Secret Access Key as those are also the most familiar for those coming from an Azure background. These would be the Azure Service Principal ID and Secret.

Something to understand if you aren’t verse in AWS. This Access Key ID is attached to an Amazon Resource Name, ARN, which is AWS’s equivalent to an Azure Resource ID. This access Key ID then will have an Access Key associated with it for signing the request. The ARN will have a role in IAM which will have the required access to create and manage the S3 account.

Azure Pipelines Terraform Task

Now that we have this service connection we can write the Terraform Task. The documentation for this task is honestly well written. Effectively we can reuse the pattern from Terraform, CI/CD, Azure DevOps, and YAML Templates. The only difference is we need to use the AWS specific parameters.

Here is an example of the init task:

parameters:
- name: serviceName
  type: string
- name: TerraformDirectory
  type: string
- name: backendServiceAws
  type: string
- name: backendAwsRegion
  type: string
- name: backendAwsBucket
  type: string

  
steps:
  - task: TerraformCLI@1
    displayName: 'Terraform : init'
    inputs:
      command: init
      backendType: aws
      workingDirectory: ${{ parameters.TerraformDirectory }}
      backendServiceAws: ${{ parameters.backendServiceAws }}
      backendAwsRegion: ${{ parameters.backendAwsRegion }}
      backendAwsBucket: ${{ parameters.backendAwsBucket }}
      backendAwsKey: ${{ parameters.serviceName }}.tfstate

For brevity, since Terraform, CI/CD, Azure DevOps, and YAML Templates already covers a lot of the structure and templating I am going to refer to the AWS specific tasks here which ultimately transpile into the finished pipeline.

The fully expanded YAML can be found here

The source code for this s3 bucket can be found on my public repo cicd_aws_s3

Results

So now we will hook up the cicd_aws_s3 to Azure DevOps and Run a pipeline:

And confirmation of our apply working:

Even better the features we’ve come to love from the Azure Pipelines Terraform Task are available with AWS. Here are the plan change view:

And our built in condition to deploy if only an apply was detected is still working:

Learnings

First, this was easy! I didn’t realize the learning curve was so small. Yes, Terraform is multi cloud so that helped. We aren’t looking at the actual service deployments like an App Service vs a Lambda. Additionally the Azure Pipelines Terraform Task abstracted out a lot of the nitty gritty for us.

For me I didn’t realize that when writing the TheYAMLPipelineOne I assumed I’d only ever deploy to Azure…..well now I am using it to deploy to AWS. As such I will need to be more specific when naming my job/stages/task to include which cloud the deployment is going to.