Many of us know that introducing large batches of changes into production is risky. However, because of complexity and many moving parts, it can also be risky when deploying changes in small batches – without the right techniques. One of the better ways of mitigating deployment risk is by gradually deploying small and frequent changes to production and testing the impact of these changes before deploying to a larger group of end users. One of these patterns is known as a canary deployment.

The AWS Serverless Application Model (SAM) provides serverless capabilities via a template framework that extends AWS CloudFormation. In this example, you will launch a CloudFormation stack that uses a SAM template. SAM templates use a shorthand syntax for defining serverless resources so that you get more done in less configuration code.

AWS CodeDeploy supports the capability of gradually deploying new versions of Lambda functions using techniques such as canaries. This approach reduces the risk of deploying all changes at once by ensuring the application has been properly tested for a small subset of users in production before being widely deployed to everyone. AWS refers to this as “safe deployments” as well. The SAM makes this even easier by providing safe deployment support as part of the  AWS::Serverless::Function resource of a SAM template.

In this post, you will learn the steps for launching canary deployments for a serverless application solution in your AWS account. Below, I have provided an accompanying video that walks through these steps.

In Figure 1, you see an illustration of some of the core services in this solution. As you will see next, you define configuration for the gradual deployments in the Serverless Function properties of the SAM template. In turn, the SAM stack uses the CodeDeploy service to gradually deploy a new version of the Lambda Function to end users.

 

Figure 1 – Canary Deployment Architecture for Serverless Web Application

In Listing 1, you see a snippet of a SAM template that has been configured to perform a canary deployment in which 10 percent of end users see the new Lambda function for the first five minutes and then it’s deployed to the remaining 90 percent of users five minutes later (LambdaCanary10Percent5Minutes).

Listing 1 – Canary Deployment Definition in a SAM Template

You can use the SAM to set the AutoPublishAlias and DeploymentPreference properties in the SAM template. Use the Type: to set the Lambda Deployment Type. As you can see, the SAM uses AWS CodeDeploy – which is a fully managed service for deploying code – to define this deployment. For a list of options, see Predefined Deployment Configurations for an AWS Lambda Compute Platform.

You may also notice that there are PreTraffic and PostTraffic hooks that allow you to run Lambda functions to perform tests in the first five minutes of the canary deployment before deploying to all users.

Launch the Solution

With this automated solution, you will run a single command to launch a CloudFormation Stack that provisions all of the resources in this solution. Then, you will go to your web browser to see the data returned from an Amazon DynamoDB table via Amazon API Gateway and AWS Lambda. After this, you will make changes to some Node.js code that runs via AWS Lambda to see how the canary deployments run as part of CodePipeline and CodeDeploy.

Prerequisites

  • AWS Account and permissions for the required AWS services.
  • AWS CLI installed and access keys configured.
  • While most IDEs should work, only AWS Cloud9 is supported for this solution. Git is automatically installed with Cloud9.

Launch CloudFormation Stack

To reduce variation, you can run the commands from an AWS Cloud9 IDE.  Regardless, you will need to ensure your environment has access credentials for running the AWS CLI.

Before running the commands to launch the stack, you should create a secret in AWS Secrets Manager by running the command below – replacing GITHUBTOKEN with your GitHub token you created at https://github.com/settings/tokens.

aws secretsmanager create-secret --name github/personal-access-token --description "GitHub Token" --secret-string "GITHUBTOKEN"

To launch the solution, run the commands from Listing 2. The bash script uses the AWS CLI to create a CloudFormation stack that provisions S3, IAM, CodeBuild, and CodePipeline resources.

It takes about 3 minutes to launch the stack and run through the pipeline.

Listing 2 – Commands to Launch Canary Serverless Deployments Solution

It will take a few minutes to initiate the launch of the stack and then another five minutes to launch all the resources. Once the cloudproviders-pmd-us-east-1 stack is complete, continue to the next step to get the data.

Run a Canary Deployment

To see a canary deployment in action, follow these steps:

  • Modify the Node.js function code by going to the webapp/src directory within your local cloudproviders Git repo. Next, modify the text for the index-get.js file (an example is shown in Figure 2). Then, commit and push those changes to GitHub. CodePipeline automatically starts a new pipeline revision.

Figure 2 – Updating Lambda Function Node.js Code

  • In the ExecuteChangeSet action of CodePipeline, CodeDeploy launches a deployment based on the Deployment Type in the SAM template. Once it gets to this stage in CodePipeline, go to the CloudFormation console and click on the cloudproviders-pmd-us-east-1 stack and then click on the CodeDeployUrl Output value. Select the cloudproviders-pmd-us-east-1 deployment that is running. You will see the traffic shifting occurring on the CodeDeploy dashboard – as shown in Figure 3.

Figure 3 – CodeDeploy Dashboard showing Canary Deployment Occurring

  • From the command line, run aws lambda list-aliases --function-name FUNCTION-NAME (get the function by going to LambdaFunctionUrl Output in the cloudproviders-pmd-us-east-1 CloudFormation stack) to get the latest version and what percentage of traffic is deploying which version to the Lambda alias for that function. In Figure 4, you see the FunctionVersion is 2 and the RoutingConfig->AdditionVersionWeights is set to .2 for “3”. This means that 20% of the users are seeing version 3 of the Lambda function while 80% are using version 2. Over a period of five minutes, all traffic is using version 3 of the Lambda function.

Figure 4 – Percentage of Traffic Going to Newer Version of Lambda Function

  • Click on the LambdaFunctionUrl Output value in the cloudproviders-pmd-us-east-1 CloudFormation stack to launch the Lambda function in the AWS console. Then, click the Test button to see which Lambda version gets called for the live alias. Continue to click this button to see which version gets pulled for the live alias. As CodeDeploy gradually deploys the function and points the function to the alias, the more times you will see the most recent Lambda version running. Once it completes the deployment, all invocations of the Lambda alias will be using the latest version (i.e., version 3 in my example) of the Lambda function. In Figure 5, you see a screenshot running either version 2 or version 3 for the Lambda function alias depending on the relative weighting of the version in the context of the canary deployment.

Figure 5 – CloudWatch Logs Showing Different Versions of Lambda Function Linked to “live” Alias

For more information on performing safe deployments using the SAM, see Deploying Serverless Applications Gradually.

Conclusion

In this post, you were able to launch a stack that deployed all of the resources for a serverless application on AWS using AWS CloudFormation, AWS CodePipeline, and AWS CodeBuild. You were able to perform a canary deployment for a Lambda function using an AWS SAM template that uses AWS CodeDeploy to deploy a new version. This approach helps reduce deployment risks by automatically testing new features for a small subset of end users in production before deploying to the rest of your user base.

All of the source code for this solution is located at https://github.com/PaulDuvall/cloudproviders/

Photo by David Clode on Unsplash It’s a Yellow Honeyeater not a canary but it looks pretty, pretty, pretty, cool!

Stelligent Amazon Pollycast
Voiced by Amazon Polly