Automated Microservice Deployment to AWS Serverless Compute Engine with Azure DevOps
An increased number of organisations start adopting to multi cloud models as part of their cloud strategy to avoid vendor lock-in and benefit from best-of-breed solutions. With multi-cloud, you can run workloads across multiple public or private clouds based on requirements and priorities. As a part of multi cloud strategy, some organisations prefer to use Azure DevOps as their CI/CD toolset to simplify continuous integrations (CI) and continuous delivery (CD) in microservices development process and AWS serverless compute engines for hosting microservices.
Azure DevOps is one of the popular Software as a service (SaaS) platform from Microsoft based on Azure Cloud that provides complete toolchains for project management, version controlling, automated build and test and release management. Azure DevOps Pipelines enables you to automate build and test of your code project and implement continuous delivery of built artefacts to any cloud, including Azure, AWS and GCP.
In a multi cloud environment, the best practice is to use one CI/CD toolset for consistency and auditability. If an organisation has already heavily invested on Azure Cloud and starts investing on AWS cloud to create multi-cloud environment, it is recommended to continue using Azure DevOps Pipelines across the organisation to deploy the code without introducing another new CI/CD toolset.
AWS is the leader in the cloud computing and offers the best serverless compute services for running code without having to provision or manage servers. AWS Lambda and AWS Fargate are the prominent serverless compute platform from AWS, and both allow to run Docker images. AWS-Fargate is, by its nature, container based, and it is a technology used with Amazon ECS, Amazon EKS and AWS Batch to run containers without having to manage servers or clusters of Amazon EC2 instances. AWS Lambda is a serverless, event-driven compute service and supports a range of programming and deployment methods, including Docker images.
The purpose of this blog post is to walk you through on how to create Azure DevOps pipeline to build Docker container image and push it to Amazon ECR (Elastic Container Registry) and automate the deployment of container image to AWS Fargate environment.
We assume here you have already created Azure DevOps project called test, repository called test-services and imported your project code into the repository.
The following diagram illustrates the integration architecture of Azure DevOps and AWS Platform. Azure DevOps pipeline will push the image to Amazon ECR repository in AWS shared service account and then update the task definition with the new image in AWS Fargate environments in AWS dev, test and prod account.
The following guides through the steps for automating deployment of microservices to AWS Fargate using Azure DevOps.
- Create Amazon ECR Repository in AWS shared service account.
Amazon ECR repository needs to be created to push docker image. You can sign into the AWS Management Console, open the Amazon ECR console at https://console.aws.amazon.com/ecr/repositories and create a private repository. For instructions, see Creating a private repository in the Amazon ECR documentation.
2. Create an IAM User, Role and Policy in AWS.
To push the image and update the task definition in AWS cloud, Azure DevOps requires to access to AWS resources. You need to create an IAM user in shared service account with below policy to push image to Amazon ECR repository and to assume role from AWS development account, AWS test account and AWS production account. Azure DevOps can use AWS access keys to make calls to AWS.
You also need to create azure-devops-pipeline role in dev, test and prod account with the following policy.
3. Create script to run it in Azure DevOps pipeline to update AWS Fargate task definition.
This script creates AWS profiles using access keys and assuming IAM roles and updates the stack, which is created to provision AWS Fargate environment using AWS CloudFormation, with the new image.
The sample AWS CloudFormation templates used to create AWS Fargate environment is provided below in step 5.
Amazon ECR image URI is passed as parameter in AWS CloudFormation and this script will set the parameter value with new image URI and update the stack so that task definition will get updated with new image.
4. Create Azure DevOps Pipeline
Azure DevOps provides multi-stage YAML pipeline and release pipelines. Release pipeline enables to set up only CD process to automate the deployment, whereas multi-stage YAML pipeline allows to create entire pipeline from CI to CD as YAML code that can be included in version control along with the application code and changed continuously to improve the pipeline over time. Best practice is to stick with multi-stage YAML pipeline as release pipeline has limited capabilities.
Stages in the pipeline allows to segregate parts of the pipeline into groups to execute sequentially. A stage will contain a set of jobs which consists of one or more steps. A step is the smallest building block of a pipeline, and the step can be either task or scripts.
As a first stage in the pipeline, you can include building project code, publishing image to Amazon ECR and deploy to development account as below. The pipeline will get triggered when you push your code changes to feature/develop/master branches. DeployToDev will get executed if any change is pushed to develop branch only.
The best practice is to push your changes to feature branch and let others know about the changes you have pushed by creating pull request (PR) so that others can review your changes and merged into the develop branch.
Next stage in the pipeline is manual trigger and once you change is tested in dev, you can promote your changes to test and prod by starting the DeployToTestAndProd stage manually. Pipeline also contains approval job before prod deployment job in order to get the manual approval for prod deployment from authorised people.
Ensure the global variables are already created in Azure DevOps as below.
5. Create AWS Fargate environment to run container using the following sample AWS CloudFormation template in dev, test and prod accounts.
You need to provide the Amazon ECR image URI as parameter value when you create the template. Make sure Azure DevOps pipeline first stage is already executed and image is pushed to Amazon ECR repository.
This post explained in detail on how to continuously deploy microservices to to AWS Fargate environment using Azure DevOps pipeline in multi cloud environment. Furthermore, it provides sample pipeline, scripts, IAM policy and AWS CloudFormation template to provision AWS Fargate environment for your ease of reference.
