DevOps in AWS Radio: Orchestrating Docker containers with AWS ECS, ECR and CodePipeline (Episode 4)

In this episode, Paul Duvall and Brian Jakovich from Stelligent cover recent DevOps in AWS news and speak about the AWS EC2 Container Service (ECS), AWS EC2 Container Registry (ECR), HashiCorp Consul, AWS CodePipeline, and other tools in providing Docker-based solutions for customers. Here are the show notes:

DevOps in AWS News

Episode Topics

  1. Benefits of using ECS, ECR, Docker, etc.
  2. Components of ECS, ECR and Service Discovery
  3. Orchestrating and automating the deployment pipeline using CloudFormation, CodePipeline, Jenkins, etc. 

Blog Posts

  1. Automating ECS: Provisioning in CloudFormation (Part 1)
  2. Automating ECS: Orchestrating in CodePipeline and CloudFormation (Part 2)

About DevOps in AWS Radio

On DevOps in AWS Radio, we cover topics around applying DevOps principles and practices such as Continuous Delivery in the Amazon Web Services cloud. This is what we do at Stelligent for our customers. We’ll bring listeners into our roundtables and speak with engineers who’ve recently published on our blog and we’ll also be reaching out to the wider DevOps in AWS community to get their thoughts and insights.

The overall vision of this podcast is to describe how listeners can create a one-click (or “no click”) implementation of their software systems and infrastructure in the Amazon Web Services cloud so that teams can deliver software to users whenever there’s a business need to do so. The podcast will delve into the cultural, process, tooling, and organizational changes that can make this possible including:

  • Automation of
    • Networks (e.g. VPC)
    • Compute (EC2, Containers, Serverless, etc.)
    • Storage (e.g. S3, EBS, etc.)
    • Database and Data (RDS, DynamoDB, etc.)
  • Organizational and Team Structures and Practices
  • Team and Organization Communication and Collaboration
  • Cultural Indicators
  • Version control systems and processes
  • Deployment Pipelines
    • Orchestration of software delivery workflows
    • Execution of these workflows
  • Application/service Architectures – e.g. Microservices
  • Automation of Build and deployment processes
  • Automation of testing and other verification approaches, tools and systems
  • Automation of security practices and approaches
  • Continuous Feedback systems
  • Many other Topics…

Stelligent Bookclub: “Building Microservices” by Sam Newman

At Stelligent, we put a strong focus on education and so I wanted to share some books that have been popular within our team. Today we explore the world of microservices with “Building Microservices” by Sam Newman.

Microservices are an approach to distributed systems that promotes the use of small independent services within a software solution. By adopting microservices, teams can achieve better scaling and gain autonomy, that allows teams to chose their technologies and iterate independently from other teams.

As a result, a change to one part of the system could unintentionally break a different part, which in turn might lead to hard-to-predict outages

Microservices are an alternative to the development of a monolithic codebase in many organizations – a codebase that contains your entire application and where new code piles on at alarming rates. Monoliths become difficult to work with as interdependencies within the code begin to develop.

As a result, a change to one part of the system could unintentionally break a different part, which in turn might lead to hard-to-predict outages. This is where Newman’s argument about the benefits of microservices really comes into play.

  • Reasons to split the monolith
    • Increase pace of change
    • Security
    • Smaller team structure
    • Adopt the proper technology for a problem
    • Remove tangled dependencies
    • Remove dependency on databases for integration
    • Less technical debt

By splitting monoliths at their seams, we can slowly transform a monolithic codebase into a group of microservices. Each service his loosely coupled and highly cohesive, as a result changes within a microservice do not change it’s function to other parts of the system. Each element works in a blackbox where only the inputs and outputs matter. When splitting a monolith, databases pose some of the greatest challenge; as a result, Newman devotes a significant chunk of the text/book to explaining various useful techniques to reduce these dependencies.

Ways to reduce dependencies

  • Clear well documented api
  • Loose coupling and high cohesion within a microservice
  • Enforce standards on how services can interact with each other

Though Newman’s argument for the adoption of microservices is spot-on, his explanation on continuous delivery and scaling micro-services is shallow. For anyone who has a background in CD or has read “Continuous Delivery” these sections do not deliver. For example, he takes the time to talk about machine images at great length but lightly brushes over build pipelines. The issue I ran into with scaling microservices is Newman suggests that ideally each microservice should ideally be put on its own instance where it exists independently of all other services. Though this is a possibility and it would be nice to have this would be highly unlikely to happen in a production environment where cost is a consideration. Though he does talk about using traditional virtualization, Vagrant, linux containers, and Docker to host multiple services on a single host he remains platform agnostic and general. As a result he misses out on the opportunity to talk about services like Amazon ECS, Kubernetes, or Docker Swarm. Combining these technologies with reserved cloud capacity would be a real world example that I feel would have added a lot to this section

Overall Newman’s presentation of microservices is a comprehensive introduction for IT professionals. Some of the concepts covered are basic but there are many nuggets of insight that are worth reading for. If you are looking to get a good idea about how microservices work, pick it up. If you’re looking to advance your microservice patterns or suggest some, feel free to comment below!

Interested in working someplace that gives all employees an impressive book expense budget? We’re hiring.

Automating and Orchestrating OpsWorks in CloudFormation and CodePipeline

pipeline_opsworks_consoleIn this post, you’ll learn how to provision, configure, and orchestrate a PHP application using the AWS OpsWorks application management service into a deployment pipeline using AWS CodePipeline that’s capable of deploying new infrastructure and code changes when developers commit changes to the AWS CodeCommit version-control repository. This way, team members can release new changes to users whenever they choose to do so: aka, Continuous Delivery.

Recently, AWS announced the integration of OpsWorks into AWS CodePipeline so I’ll be describing various components and services that support this solution including CodePipeline along with codifying the entire infrastructure in AWS CloudFormation. As part of the announcement, AWS provided a step-by-step tutorial of integrating OpsWorks with CodePipeline that I used as a reference in automating the entire infrastructure and workflow.

This post describes how to automate all the steps using CloudFormation so that you can click on a Launch Stack button to instantiate all of your infrastructure resources.

OpsWorks

“AWS OpsWorks is a configuration management service that helps you configure and operate applications of all shapes and sizes using Chef. You can define the application’s architecture and the specification of each component including package installation, software configuration and resources such as storage. Start from templates for common technologies like application servers and databases or build your own to perform any task that can be scripted. AWS OpsWorks includes automation to scale your application based on time or load and dynamic configuration to orchestrate changes as your environment scales.” [1]

OpsWorks provides a structured way to automate the operations of your AWS infrastructure and deployments with lifecycle events and the Chef configuration management tool. OpsWorks provides more flexibility than Elastic Beanstalk and more structure and constraints than CloudFormation. There are several key constructs that compose OpsWorks. They are:

  • Stack – An OpsWorks stack is the logical container defining OpsWorks layers, instances, apps and deployments.
  • Layer – There are built-in layers provided by OpsWorks such as Static Web Servers, Rails, Node.js, etc. But, you can also define your own custom layers as well.
  • Instances – These are EC2 instances on which the OpsWorks agent has been installed. There are only certain Linux and Windows operating systems supported by OpsWorks instances.
  • App – “Each application is represented by an app, which specifies the application type and contains the information that is needed to deploy the application from the repository to your instances.” [2]
  • Deployment – Runs Chef recipes to deploy the application onto instances based on the defined layer in the stack.

There are also lifecycle events that get executed for each deployment. Lifecycle events are linked to one or more Chef recipes. The five lifecycle events are setup, configure, deploy, undeploy, shutdown. Events get triggered based upon certain conditions. Some events can be triggered multiple times. They are described in more detail below:

  • setup – When an instance finishes booting as part of the initial setup
  • configure – When this event is run, it executes on all instances in all layers whenever a new instance comes in service, or an EIP changes, or an ELB is attached
  • deploy – When running a deployment on an instance, this event is run
  • undeploy – When an app gets deleted, this event is run
  • shutdown – Before an instance is terminated, this event is run

Solution Architecture and Components

In Figure 2, you see the deployment pipeline and infrastructure architecture for the OpsWorks/CodePipeline integration.

opsworks_pipeline_arch.jpg
Figure 2 – Deployment Pipeline Architecture for OpsWorks

Both OpsWorks and CodePipeline are defined in a single CloudFormation stack, which is described in more detail later in this post. Here are the key services and tools that make up the solution:

  • OpsWorks – In this stack, code configures operations of your infrastructure using lifecycle events and Chef
  • CodePipeline – Orchestrate all actions in your software delivery process. In this solution, I provision a CodePipeline pipeline with two stages and one action per stage in CloudFormation
  • CloudFormation – Automates the provisioning of all AWS resources. In this solution, I’m using CloudFormation to automate the provisioning for OpsWorks, CodePipeline,  IAM, and S3
  • CodeCommit – A Git repo used to host the sample application code from this solution
  • PHP – In this solution, I leverage AWS’ OpsWorks sample application written in PHP.
  • IAM – The CloudFormation stack defines an IAM Instance Profile and Roles for controlled access to AWS resources
  • EC2 – A single compute instance is launched as part of the configuration of the OpsWorks stack
  • S3 – Hosts the deployment artifacts used by CodePipeline.

Create and Connect to a CodeCommit Repository

While you can store your software code in any version-control repository, in this solution, I’ll be using the AWS CodeCommit Git repository. I’ll be integrating CodeCommit with CodePipeline. I’m basing the code off of the Amazon OpsWorks PHP Simple Demo App located at https://github.com/awslabs/opsworks-demo-php-simple-app.

To create your own CodeCommit repo, follow these instructions: Create and Connect to an AWS CodeCommit Repository. I called my CodeCommit repository opsworks-php-demo. You can call it the same but if you do name it something different, be sure to replace the samples with your repo name.

After you create your CodeCommit repo, copy the contents from the AWS PHP OpsWorks Demo app and commit all of the files.

Implementation

I created this sample solution by stitching together several available resources including the CloudFormation template provided by the Step-by-Step Tutorial from AWS on integrating OpsWorks with CodePipeline and existing templates we use at Stelligent for CodePipeline. Finally, I manually created the pipeline in CodePipeline using the same step-by-step tutorial and then obtained the configuration of the pipeline using the get-pipeline command as shown in the command snippet below.

aws codepipeline get-pipeline --name OpsWorksPipeline > pipeline.json

This section describes the various resources of the CloudFormation solution in greater detail including IAM Instance Profiles and Roles, the OpsWorks resources, and CodePipeline.

Security Group

Here, you see the CloudFormation definition for the security group that the OpsWorks instance uses. The definition restricts the ingress port to 80 so that only web traffic is accepted on the instance.

    "CPOpsDeploySecGroup":{
      "Type":"AWS::EC2::SecurityGroup",
      "Properties":{
        "GroupDescription":"Lets you manage OpsWorks instances deployed to by CodePipeline"
      }
    },
    "CPOpsDeploySecGroupIngressHTTP":{
      "Type":"AWS::EC2::SecurityGroupIngress",
      "Properties":{
        "IpProtocol":"tcp",
        "FromPort":"80",
        "ToPort":"80",
        "CidrIp":"0.0.0.0/0",
        "GroupId":{
          "Fn::GetAtt":[
            "CPOpsDeploySecGroup",
            "GroupId"
          ]
        }
      }
    },

IAM Role

Here, you see the CloudFormation definition for the OpsWorks instance role. In the same CloudFormation template, there’s a definition for an IAM service role and an instance profile. The instance profile refers to OpsWorksInstanceRole defined in the snippet below.

The roles, policies and profiles restrict the service and resources to the essential permissions it needs to perform its functions.

    "OpsWorksInstanceRole":{
      "Type":"AWS::IAM::Role",
      "Properties":{
        "AssumeRolePolicyDocument":{
          "Statement":[
            {
              "Effect":"Allow",
              "Principal":{
                "Service":[
                  {
                    "Fn::FindInMap":[
                      "Region2Principal",
                      {
                        "Ref":"AWS::Region"
                      },
                      "EC2Principal"
                    ]
                  }
                ]
              },
              "Action":[
                "sts:AssumeRole"
              ]
            }
          ]
        },
        "Path":"/",
        "Policies":[
          {
            "PolicyName":"s3-get",
            "PolicyDocument":{
              "Version":"2012-10-17",
              "Statement":[
                {
                  "Effect":"Allow",
                  "Action":[
                    "s3:GetObject"
                  ],
                  "Resource":"*"
                }
              ]
            }
          }
        ]
      }
    },

Stack

The snippet below shows the CloudFormation definition for the OpsWorks Stack. It makes references to the IAM service role and instance profile, using Chef 11.10 for its configuration, and using Amazon Linux 2016.03 for its operating system. This stack is used as the basis for defining the layer, app, instance, and deployment that are described later in this section.

    "MyStack":{
      "Type":"AWS::OpsWorks::Stack",
      "Properties":{
        "Name":{
          "Ref":"AWS::StackName"
        },
        "ServiceRoleArn":{
          "Fn::GetAtt":[
            "OpsWorksServiceRole",
            "Arn"
          ]
        },
        "ConfigurationManager":{
          "Name":"Chef",
          "Version":"11.10"
        },
        "DefaultOs":"Amazon Linux 2016.03",
        "DefaultInstanceProfileArn":{
          "Fn::GetAtt":[
            "OpsWorksInstanceProfile",
            "Arn"
          ]
        }
      }
    },

Layer

The OpsWorks PHP layer is described in the CloudFormation definition below. It references the OpsWorks stack that was previously created in the same template. It also uses the php-app layer type. For a list of valid types, see CreateLayer in the AWS API documentation. This resource also enables auto healing, assigns public IPs and references the previously-created security group.

    "MyLayer":{
      "Type":"AWS::OpsWorks::Layer",
      "Properties":{
        "StackId":{
          "Ref":"MyStack"
        },
        "Name":"MyLayer",
        "Type":"php-app",
        "Shortname":"mylayer",
        "EnableAutoHealing":"true",
        "AutoAssignElasticIps":"false",
        "AutoAssignPublicIps":"true",
        "CustomSecurityGroupIds":[
          {
            "Fn::GetAtt":[
              "CPOpsDeploySecGroup",
              "GroupId"
            ]
          }
        ]
      },
      "DependsOn":[
        "MyStack",
        "CPOpsDeploySecGroup"
      ]
    },

OpsWorks Instance

In the snippet below, you see the CloudFormation definition for the OpsWorks instance. It references the OpsWorks layer and stack that are created in the same template. It defines the instance type as c3.large and refers to the EC2 Key Pair that you will provide as an input parameter when launching the stack.

    "MyInstance":{
      "Type":"AWS::OpsWorks::Instance",
      "Properties":{
        "LayerIds":[
          {
            "Ref":"MyLayer"
          }
        ],
        "StackId":{
          "Ref":"MyStack"
        },
        "InstanceType":"c3.large",
        "SshKeyName":{
          "Ref":"KeyName"
        }
      }
    },

OpsWorks App

In the snippet below, you see the CloudFormation definition for the OpsWorks app. It refers to the previously created OpsWorks stack and uses the current stack name for the app name – making it unique. In the OpsWorks type, I’m using php. For other supported types, see CreateApp.

I’m using other for the AppSource type (OpsWorks doesn’t seem to make the documentation obvious in terms of the types that AppSource supports, so I resorted to using the OpsWorks console to determine the possibilities). I’m using other because my source type is CodeCommit, which isn’t currently an option in OpsWorks.

    "MyOpsWorksApp":{
      "Type":"AWS::OpsWorks::App",
      "Properties":{
        "StackId":{
          "Ref":"MyStack"
        },
        "Type":"php",
        "Shortname":"phptestapp",
        "Name":{
          "Ref":"AWS::StackName"
        },
        "AppSource":{
          "Type":"other"
        }
      }
    },

CodePipeline

In the snippet below, you see the CodePipeline definition for the Deploy stage and the DeployPHPApp action in CloudFormation. It takes MyApp as an Input Artifact – which is an Output Artifact of the Source stage and action that obtains code assets from CodeCommit.

The action uses a Deploy category and OpsWorks as the Provider. It takes four inputs for the configuration: StackId, AppId, DeploymentType, LayerId. With the exception of DeploymentType, these values are obtained as references from previously created AWS resources in this CloudFormation template.

For more information, see CodePipeline Concepts.

         {
            "Name":"Deploy",
            "Actions":[
              {
                "InputArtifacts":[
                  {
                    "Name":"MyApp"
                  }
                ],
                "Name":"DeployPHPApp",
                "ActionTypeId":{
                  "Category":"Deploy",
                  "Owner":"AWS",
                  "Version":"1",
                  "Provider":"OpsWorks"
                },
                "OutputArtifacts":[

                ],
                "Configuration":{
                  "StackId":{
                    "Ref":"MyStack"
                  },
                  "AppId":{
                    "Ref":"MyOpsWorksApp"
                  },
                  "DeploymentType":"deploy_app",
                  "LayerId":{
                    "Ref":"MyLayer"
                  }
                },
                "RunOrder":1
              }
            ]
          }

Launch the Stack

Click the button below to launch a CloudFormation stack that provisions the OpsWorks environment including all the resources previously described such as CodePipeline, OpsWorks, IAM Roles, etc.

When launching a stack, you’ll enter a value the KeyName parameter from the drop down. Optionally, you can enter values for your CodeCommit repository name and branch if they are different than the default values.

opsworks_pipeline_cfn
Figure 3- Parameters for Launching the CloudFormation Stack

You will charged for your AWS usage – particularly EC2, CodePipeline and S3.

To launch the same stack from your AWS CLI, type the following (while modifying the same parameter values described above):

aws cloudformation create-stack --stack-name OpsWorksPipelineStack --template-url https://s3.amazonaws.com/stelligent-training-public/public/codepipeline/codepipeline-opsworks.json --region us-east-1 --disable-rollback --capabilities="CAPABILITY_IAM" --parameters  ParameterKey=KeyName,ParameterValue=YOURKEYNAME

Outputs

Once the CloudFormation stack successfully launches, there’s an output for the CodePipelineURL. You can click on this value to launch the pipeline that’s running that’s getting the source assets from CodeCommit and launch an OpsWorks stack and associated resources. See the screenshot below.

cfn_opsworks_pipeline_outputs
Figure 4 – CloudFormation Outputs for CodePipeline/OpsWorks stack

Once the pipeline is complete, you can access the OpsWorks stack and click on the Public IP link for one of the instances to launch the PHP application that was deployed using OpsWorks as shown in Figures 5 and 6 below.

opsworks_public_ip.jpg
Figure 5 – Public IP for the OpsWorks instance

 

opsworks_app_before.jpg
Figure 6 – OpsWorks PHP app once initially deployed

Commit Changes to CodeCommit

Make some visual changes to the code (e.g. your local CodeCommit version of index.php) and commit these changes to your CodeCommit repository to see these software get deployed through your pipeline. You perform these actions from the directory where you cloned a local version of your CodeCommit repo (in the directory created by your git clone command). Some example command-line operations are shown below.

git commit -am "change color to rust orange"
git push

Once these changes have been committed, CodePipeline will discover the changes made to your CodeCommit repo and initiate a new pipeline. After the pipeline is successfully completed, follow the same instructions for launching the application from your browser – as shown in Figure 7.

opsworks_app_after.jpg
Figure 7 – Application after code changes committed to CodeCommit, orchestrated by CodePipeline and deployed by OpsWorks

Sample Code

The code for the examples demonstrated in this post are located at https://github.com/stelligent/cloudformation_templates/tree/master/labs/opsworks. Let us know if you have any comments or questions @stelligent or @paulduvall.

Stelligent is hiring! Do you enjoy working on complex problems like figuring out ways to automate all the things as part of a deployment pipeline? Do you believe in the “everything-as-code” mantra? If your skills and interests lie at the intersection of DevOps automation and the AWS cloud, check out the careers page on our website.

Useful Resources and References

OpsWorks Reference

Below, I’ve documented some additional information that might be useful on the OpsWorks service itself including its available integrations, supported versions and features.

  • OpsWorks supports three application source types: GitHub, S3, and HTTP.
  • You can store up to five versions of an OpsWorks application: the current revision plus four more for rollbacks.
  • When using the create-deployment method, you can target the OpsWorks stack, app, or instance
  • OpsWorks require internet access for the OpsWorks endpoint instance
  • Chef supports Windows in version 12
  • You cannot mix Windows and Linux instances in an OpsWorks stack
  • To change the default OS in OpsWorks, you need to change the OS and reprovision the instances
  • You cannot change the VPC for an OpsWorks instance
  • You can add ELB, EIPs, Volumes and RDS to an OpsWorks stack
  • OpsWorks autoheals at the layer level
  • You can assign multiple Chef recipes to an OpsWorks layer event
  • The three instance types in OpsWorks are: 24/7, time-based, load-based
  • To initiate a rollback in OpsWorks, you use create-deployment command
  • The following commands are available when using OpsWorks create-deployment along with possible use cases:
    • install_dependencies
    • update_dependencies – Patches to the Operating System. Not available after Chef 12.
    • update_custom_cookbooks – pulling down changes in your Chef cookbooks
    • execute_recipes – manually run specific Chef recipes that are defined in your layers
    • configure – service discovery or whenever endpoints change
    • setup
    • deploy
    • rollback
    • start
    • stop
    • restart
    • undeploy
  • To enable the use of multiple custom cookbook repositories in OpsWorks, you can enable custom cookbook at the stack and then create a cookbook that has a Berkshelf file with multiple sources. Before Chef 11.10, you couldn’t use multiple cookbook repositories.
  • You can define Chef databags in OpsWorks Users, Stacks, Layers, Apps and Instances
  • OpsWorks Auto Healing is triggered when an OpsWorks Agent detects loss of communication and stops, then restarts the instances. If it fails, it goes into manual intervention
  • OpsWorks will not auto heal an upgrade to the OS
  • OpsWorks does not auto heal by monitoring performance, only failures.

Acknowledgements

My colleague Casey Lee provided some of the background information on OpsWorks features. I also used several resources from AWS including the PHP sample app and the step-by-step tutorial on the OpsWorks/CodePipeline integration.

 

 

 

Automate CodeCommit and CodePipeline in AWS CloudFormation

Amazon Web Services (AWS) recently announced the integration of AWS CodeCommit with AWS CodePipeline. This means you can now use CodeCommit as a version-control repository as part of your pipelines! AWS describes how to manucodepipeline_codecommitally configure this integration at Simple Pipeline Walkthrough (AWS CodeCommit Repository).

One of the biggest benefits of using CodeCommit is its seamless integration with other AWS services including IAM.

After describing how to create and connect to a new CodeCommit repository, in this blog post, I’ll explain how to fully automate the provisioning of all of the AWS resources in CloudFormation to achieve Continuous Delivery. This includes CodeCommit, CodePipeline, CodeDeploy and IAM using a sample application provided by AWS.

Create a CodeCommit Repository

To create a new CodeCommit version-control repository, go to your AWS console and select CodeCommit under Developer Tools. Click the Create new repository button, enter a unique repository name and a description and click Create repository. Next, you will connect to the repository.

create_codecommit_repo

Connect to the CodeCommit Repository

There are a couple of ways to connect to an existing CodeCommit repository. One is via https and the other ssh. We’re going to focus on the connecting via SSH in this post. For additional information, see Connect to an AWS CodeCommit Repository. The instructions in this section are based on CodeCommit Setup from Andrew Puch.

The first thing you’ll do is create an IAM user. You do not need to create a password or access keys at this time. To do this, go to the AWS IAM console, then the Users section and create a new user. Once you’ve created the new user, open a terminal and type the following. NOTE: These instructions are designed for Mac/Linux environments.

cd $HOME/.ssh
ssh-keygen

When prompted after typing ssh-keygen, use a name like codecommit_rsa and leave all fields blank and just hit enter

cat codecommit_rsa.pub

Go to IAM, select the user you previously created, click on the Security Credentials tab and click the Upload SSH public key button. Copy the contents of the codecommit_rsa.pub to the text box in the Upload SSH public key section and save.

Go back to your terminal and type:

touch config
chmod 600 config
sudo vim config

The YOUR_SSH_KEY_ID_FROM_IAM variable below is the row value for the SSH Key Id that you created when uploading the public key. You will replace this placeholder and brackets with the value for the SSH Key Id from the Security Credentials tab for the IAM user you’d previously created.

Host git-codecommit.*.amazonaws.com
  User [YOUR_SSH_KEY_ID_FROM_IAM]
  IdentityFile ~/.ssh/codecommit_rsa

To verify your SSH connection works, type:

ssh git-codecommit.us-east-1.amazonaws.com

To clone a local copy of the CodeCommit repo you just created, type something like (your region and/or repo name may be different):

git clone ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/codecommit-demo

You’ll be using this local copy of your CodeCommit repo later.

Manually Integrate CodeCommit with CodePipeline

Follow the instructions for manually integrating CodeCommit with CodePipeline as described in Simple Pipeline Walkthrough (AWS CodeCommit Repository). Name the pipeline CodeCommitPipeline.

Create a CloudFormation Template

In this solution, the CloudFormation template is composed of several components. It includes launching EC2 instances and installing the CodeDeploy agent on the instances, provisioning a CodeDeploy application and DeploymentGroup, creating an IAM role that defines a policy for all resources used in the solution including CodeCommit, and provisioning the CodePipeline pipeline stages and action – including the inclusion of CodeCommit.

codepipeline_cc_arch

Launch EC2 instances and install CodeDeploy agent

To launch the EC2 instances that are used by CodeDeploy to deploy the sample application, I’m using a CloudFormation template provided by AWS as part of a nested stack. I’m passing in the name of my EC2 Key Pair along with the tag that CodeDeploy will use to run a deployment against EC2 instances labeled with this tag.

    "CodeDeployEC2InstancesStack":{
      "Type":"AWS::CloudFormation::Stack",
      "Properties":{
        "TemplateURL":"http://s3.amazonaws.com/aws-codedeploy-us-east-1/templates/latest/CodeDeploy_SampleCF_Template.json",
        "TimeoutInMinutes":"60",
        "Parameters":{
          "TagValue":{
            "Ref":"TagValue"
          },
          "KeyPairName":{
            "Ref":"EC2KeyPairName"
          }
        }
      }
    },

Create CodeDeploy Application and Deployment Group

In the snippet below, I’m creating a simple CodeDeploy application along with a DeploymentGroup. I’m passing in the location of the S3 bucket and key that hosts the sample application provided by AWS. You can find the sample application at AWS CodeDeploy Resource Kit. In my template, I’m entering aws-codedeploy-us-east-1 for the S3Bucket parameter and samples/latest/SampleApp_Linux.zip for the S3Key parameter. It translates to this location in S3: s3://aws-codedeploy-us-east-1/samples/latest/SampleApp_Linux.zip. Finally, the stack from the CloudFormation template that provisions the EC2 instances for CodeDeploy provides CodeDeployTrustRoleARN as an output that I use in defining the permissions for my CodeDeploy DeploymentGroup.

    "MyApplication":{
      "Type":"AWS::CodeDeploy::Application",
      "DependsOn":"CodeDeployEC2InstancesStack"
    },
    "MyDeploymentGroup":{
      "Type":"AWS::CodeDeploy::DeploymentGroup",
      "DependsOn":"MyApplication",
      "Properties":{
        "ApplicationName":{
          "Ref":"MyApplication"
        },
        "Deployment":{
          "Description":"First time",
          "IgnoreApplicationStopFailures":"true",
          "Revision":{
            "RevisionType":"S3",
            "S3Location":{
              "Bucket":{
                "Ref":"S3Bucket"
              },
              "BundleType":"Zip",
              "Key":{
                "Ref":"S3Key"
              }
            }
          }
        },
        "Ec2TagFilters":[
          {
            "Key":{
              "Ref":"TagKey"
            },
            "Value":{
              "Ref":"TagValue"
            },
            "Type":"KEY_AND_VALUE"
          }
        ],
        "ServiceRoleArn":{
          "Fn::GetAtt":[
            "CodeDeployEC2InstancesStack",
            "Outputs.CodeDeployTrustRoleARN"
          ]
        }
      }
    },

Create an IAM role and policy to include CodeCommit

In previous posts on CodePipeline, I’d relied on the fact that, by default, AWS has created an AWS-CodePipeline-Service role in IAM. This is, frankly, a lazy and error-prone way of getting the proper permissions assigned to my AWS resources. The reason it’s error prone is because anyone else using the template might have modified the permissions of this built-in IAM role. Because the CodeCommit integration is new, I needed to add the CodeCommit permissions to my IAM policy so I decided to create a new IAM role on the fly as part of my CloudFormation template. This provides the added benefit of assuming nothing else had been previously created in the user’s AWS account.

Below, I’ve included the relevant IAM policy snippet that provides CodePipeline access to certain CodeCommit actions.

"PolicyName":"codepipeline-service",
  "PolicyDocument":{
    "Statement":[
      {
        "Action":[
          "codecommit:GetBranch",
          "codecommit:GetCommit",
          "codecommit:UploadArchive",
          "codecommit:GetUploadArchiveStatus",
          "codecommit:CancelUploadArchive"
        ],
          "Resource":"*",
          "Effect":"Allow"
      },

 

Create a pipeline in CodePipeline using CloudFormation

To get the configuration from the pipeline you manually created in the “Manually Integrate CodeCommit with CodePipeline” step from above, go to your AWS CLI and type:

aws codepipeline get-pipeline --name CodeCommitPipeline > pipeline.json

You will use the contents of the pipeline.json in a latter step.

Below, you can see that I’m creating the initial part of the pipeline in CloudFormation. I’m referring to the IAM role that I created previously in the template. This uses the AWS::CodePipeline::Pipeline CloudFormation resource.

  "CodePipelineStack":{
      "Type":"AWS::CodePipeline::Pipeline",
      "Properties":{
        "RoleArn":{
          "Fn::Join":[
            "",
            [
              "arn:aws:iam::",
              {
                "Ref":"AWS::AccountId"
              },
              ":role/",
              {
                "Ref":"CodePipelineRole"
              }
            ]
          ]
        },

Source Stage for CodeCommit

I got the contents for the stages and actions by copying the contents from the pipeline.json that I’d created above and pasted them into my CloudFormation template in the CodePipeline resource section. After copying the contents, I updated the template to use title case vs. camel case for some of the attribute names in order to conform to the CloudFormation DSL.

For the CodePipeline Source stage and action of this CloudFormation template, I’m referring to the CodeCommit Provider as my Source category. There are no input artifacts and the OutputArtifacts is defined as MyApp. This is used in all downstream stages as part of each action’s InputArtifacts.

        "Stages":[
          {
            "Name":"Source",
            "Actions":[
              {
                "InputArtifacts":[

                ],
                "Name":"Source",
                "ActionTypeId":{
                  "Category":"Source",
                  "Owner":"AWS",
                  "Version":"1",
                  "Provider":"CodeCommit"
                },
                "OutputArtifacts":[
                  {
                    "Name":"MyApp"
                  }
                ],
                "Configuration":{
                  "BranchName":{
                    "Ref":"RepositoryBranch"
                  },
                  "RepositoryName":{
                    "Ref":"RepositoryName"
                  }
                },
                "RunOrder":1
              }
            ]
          },

Beta Stage for CodeDeploy

The beta stage refers to the CodeDeploy DeploymentGroup and Application that were created in previously-defined resources in this CloudFormation template. In the Configuration for this action, I’m referring to these previously-defined references using the Ref intrinsic function in CloudFormation.

          {
            "Name":"Beta",
            "Actions":[
              {
                "InputArtifacts":[
                  {
                    "Name":"MyApp"
                  }
                ],
                "Name":"DemoFleet",
                "ActionTypeId":{
                  "Category":"Deploy",
                  "Owner":"AWS",
                  "Version":"1",
                  "Provider":"CodeDeploy"
                },
                "OutputArtifacts":[

                ],
                "Configuration":{
                  "ApplicationName":{
                    "Ref":"MyApplication"
                  },
                  "DeploymentGroupName":{
                    "Ref":"MyDeploymentGroup"
                  }
                },
                "RunOrder":1
              }
            ]
          }

Store the Pipeline Artifact

You can store the artifact that’s transitioned through the actions in CodePipeline using any S3 bucket for which you have permissions. The template I provide as part of this sample solution dynamically generates a bucket name that should work for anyone’s AWS account as it uses the bucket that AWS CodePipeline defines which refers to the user’s current region and account id.

     ],
        "ArtifactStore":{
          "Type":"S3",
          "Location":{
            "Fn::Join":[
              "",
              [
                "codepipeline-",
                {
                  "Ref":"AWS::Region"
                },
                "-",
                {
                  "Ref":"AWS::AccountId"
                }
              ]
            ]
          }
        }
      }
    }

Launch the Stack

To launch the CloudFormation stack, simply click the button below to launch the template from the CloudFormation console in your AWS account. You’ll need to enter values for the following parameters: Stack name, EC2 KeyPair NameCodeCommit Repository Name, CodeCommit Repository Branch and, optionally, Tag Value for CodeDeploy EC2 instances. You can keep the default values for the other parameters.

codepipeline_cc_cfn

To launch the same stack from your AWS CLI, type the following (while modifying the same values described above):

aws cloudformation create-stack --stack-name CodePipelineCodeCommitStack 
--template-url https://s3.amazonaws.com/stelligent-training-public/public/codepipeline/codepipeline-codecommit.json 
--region us-east-1 --disable-rollback --capabilities="CAPABILITY_IAM" 
--parameters ParameterKey=EC2KeyPairName,ParameterValue=YOUREC2KEYPAIR 
ParameterKey=TagValue,ParameterValue=EC2TAG4CODEDEPLOY 
ParameterKey=RepositoryName,ParameterValue=codecommit-demo 
ParameterKey=RepositoryBranch,ParameterValue=master

Access the Application

Once the CloudFormation stacks have successfully completed, go to CodeDeploy and select Deployments. For example, if you’re in the us-east-1 region, the URL might look like: https://console.aws.amazon.com/codedeploy/home?region=us-east-1#/deployments. Click on the link for the Deployment Id of the deployment you just launched from CloudFormation. Then, click on the link for the Instance Id. From the EC2 instance, copy the Public DNS value and paste into your browser and hit enter. You should see a page like the one below.

codedeploy_deployment

Commit Changes to CodeCommit

Make some visual changes to the code and commit these changes to your CodeCommit repository to see these changes get deployed through your pipeline. You perform these actions from the directory where you cloned a local version of your CodeCommit repo (in the directory created by your git clone command). Some example command-line operations are shown below.

git commit -am "change color to dark orange"
git push

Once these changes have been committed, CodePipeline will discover the changes made to your CodeCommit repo and initiate a new pipeline. After the pipeline is successfully completed, follow the same instructions for launching the application from your browser.

codedeploy_deployment-new

Troubleshooting

In this section, I describe how to fix some of the common errors that you might experience.

After you’ve included CodeCommit as a Source provider in the Source action of a Source stage and run the pipeline, you might get an “Insufficient permissions” error in CodePipeline – like the one you see below.

codepipeline_codecommit_error

To fix this, make sure that the IAM role you’re using has the proper permissions to the appropriate codecommit.* actions in the IAM policy for the role. In the example, I’ve done this by defining the IAM role in CloudFormation and then assigning this role to the pipeline in CodePipeline.

Another error you might see when launching the example stack is when the S3 bucket does not exist or your user does not have the proper permissions to the bucket. Unfortunately, if this happens, all you’ll see is an “Internal Error” in the Source action/stage like the one below.

codepipeline_codecommit_s3_error

Lastly, if you use the default CodeCommit repository name and you’ve not created a repo with the same name or have not matched the CloudFormation parameter value with your CodeCommit repository name, you’ll see an error like this:

codepipeline_codecommit_repo_error

Sample Code

The code for the examples demonstrated in this post are located at https://github.com/stelligent/stelligent_commons/blob/master/cloudformation/codecommit/codepipeline-codecommit.json. Let us know if you have any comments or questions @stelligent or @paulduvall.

Stelligent is hiring! Do you enjoy working on complex problems like figuring out ways to automate all the things as part of a deployment pipeline? Do you believe in the “everything-as-code” mantra? If your skills and interests lie at the intersection of DevOps automation and the AWS cloud, check out the careers page on our website.

Resources

Serverless Delivery: Orchestrating the Pipeline (Part 3)

In the second post of this series, we focused on how to get our serverless application running with Lambda, API Gateway and S3. Our application is now able to run on a serverless platform, but we still have not applied the fundamentals of continuous delivery that we talked about in the first part of this series.

In this third and final part of this series on serverless delivery, we will implement a continuous delivery pipeline using serverless technology. Our pipeline will be orchestrated by CodePipeline with actions implemented in Lambda functions. The definition of the CodePipeline resource as well as the Lambda functions that support it are all defined in the same CloudFormation stack that we looked at last week.

Visualize The Goal

To help visualize what we are building, here is a picture of what the final pipeline looks like.

codepipeline

If you’re new to CodePipeline, let’s go over a few important terms:

  • Job – An execution of the pipeline.
  • Stage – A group of actions in the pipeline. Stages never run in parallel to each other and only one job can actively be running in a stage at a time. If a stage is currently running and a new job arrives at the stage it will block until the prior job completes. If multiple new jobs arrive, only the newest will be run while the rest will be dropped.
  • Action – A step to be performed. Actions can be in parallel or series to each other. In this pipeline, all our actions will be implemented by Lambda invocations.
  • Artifact – Each action can declare input and output artifacts that will be stored in an S3 bucket. These are objects that it will either expect to have before it runs, or objects that it will produce and make available after it runs.

The pipeline we have built for our application consists of the following four stages:

  • Source – The source stage has only one action to acquire the source code for later stages.
  • Commit – The commit stage has two actions that are responsible for:
    • Resolving project dependencies
    • Processing (e.g., compile, minify, uglify) the source code
    • Static analysis of the code
    • Unit testing of the application
    • Packaging the application for subsequent deployment
  • Acceptance – The acceptance stage has actions that will:
    • Update the Lambda function from latest source
    • Update S3 bucket with latest artifacts
    • Update API Gateway
    • End-to-end testing of the application
  • Production – The production stage performs the same steps as the Acceptance stage but against the production Lambda, S3 and API Gateway resources

Here is a more detailed picture of the pipeline. We will spend the rest of this post breaking down each step of the pipeline.

pipeline-overview

Start with Source Stage

Diagram Step: 1

The source stage only has one action in it, a 3rd party action provided by GitHub. The action will register a hook with the repo that you provide to kickoff a new job for the pipeline whenever code is pushed to the GitHub repository. Additionally, the action will pull the latest code from the branch you specified and zip it up into an object in an S3 bucket for later actions to reference.

{
  "Name": "Source",
  "Actions": [
    {
      "InputArtifacts": [],
      "Name": "Source",
      "ActionTypeId": {
        "Category": "Source",
        "Owner": "ThirdParty",
        "Version": "1",
        "Provider": "GitHub"
      },
      "Configuration": {
        "Owner": "stelligent",
        "Repo": "dromedary",
        "Branch": "serverless",
        "OAuthToken": "XXXXXX",
      },
      "OutputArtifacts": [
        {
          "Name": "SourceOutput"
        }
      ],
      "RunOrder": 1
    }
  ]
}

 

This approach helps solve a common challenge with source code management using Lambda. Obviously no one wants to upload code through the console, so many end up using CloudFormation to manage their Lambda functions. The challenge is that the CloudFormation Lambda resource expects your code to be zipped in an S3 bucket. This means you either need to use S3 as the “source of truth” for your source code, or have a process to keep it in sync fro the real “source of truth”. By building a pipeline, you can keep your source in GitHub and use the next actions that we are about to go through to deploy the Lambda function.

Build from Commit Stage

Diagram Steps: 2,3,4

The commit stage of the pipeline consists of two actions that are implemented with Lambda invocations. The first action is responsible for resolving the application dependencies via NPM. This can be an expensive operation taking many minutes, and is needed by many downstream actions, so the dependencies are zipped up and become an output artifact of this first action. Here are the details of the action:

  • Download & Unzip – Get the source artifact from S3 and unzip into a temp directory
  • Run NPM – Run npm install  the extracted source folder
  • Zip & Upload – Zip up the source folder with its dependencies in node_modules and upload the artifact to S3

Download the input artifact is accomplished with the following code:

var artifact = null;
jobDetails.data.inputArtifacts.forEach(function (a) {
  if (a.name == artifactName && a.location.type == 'S3') {
    artifact = a;
  }
});

if (artifact != null) {
  var params = {
    Bucket: artifact.location.s3Location.bucketName,
    Key: artifact.location.s3Location.objectKey
  };
  return getS3Object(params, destDirectory);
} else {
  return Promise.reject("Unknown Source Type:" + JSON.stringify(sourceOutput));
}

Likewise, the output artifact is uploaded with the following:

var artifact = null;
jobDetails.data.outputArtifacts.forEach(function (a) {
  if (a.name == artifactName && a.location.type == 'S3') {
    artifact = a;
  }
});

if (artifact != null) {
  var params = {
    Bucket: artifact.location.s3Location.bucketName,  
    Key: artifact.location.s3Location.objectKey
  };
  return putS3Object(params, zipfile);
} else {
  return Promise.reject("Unknown Source Type:" + JSON.stringify(sourceOutput));
}

 

Diagram Steps: 5,6,7

The second action in the commit stage is responsible for acquiring the source and dependencies, processing the source code, performing static analysis, running unit tests and packaging the output artifacts. This is accomplished by an Lambda action that invokes a Gulp task on the project. This allows the details of these steps to be defined in Gulp alongside the source code and able to change at a different pace than the pipeline. Here is the CloudFormation for this action:

{
  "InputArtifacts":[
    {
      "Name": "SourceInstalledOutput"
    }
  ],
  "Name":"TestAndPackage",
  "ActionTypeId":{
    "Category":"Invoke",
    "Owner":"AWS",
    "Version":"1",
    "Provider":"Lambda"
  },
  "Configuration":{
    "FunctionName":{
      "Ref":"CodePipelineGulpLambda"
    },
    "UserParameters": "task=package&DistSiteOutput=dist/site.zip&DistLambdaOutput=dist/lambda.zip”

  },
  "OutputArtifacts": [
    {
      "Name": "DistSiteOutput"
    },
    {
      "Name": "DistLambdaOutput"
    }
  ],
  "RunOrder":2
}

Notice the UserParameters  setting defined in the resource above. CodePipeline treats it as an opaque string that is passed into the Lambda function. I chose to use a query string format to pass multiple values into the Lambda function. The task  parameter defines what gulp task to run and the DistSiteOutput  and DistLambdaOutput  parameters tell the Lambda function where to expect to find artifacts to then upload to S3.

For more details on how to implement CodePipeline actions in Lambda, check out the entire source of these functions at index.js or read the post Mocking CodePipeline with Lambda.

Test in Acceptance Stage

Diagram Steps: 8,9,10,11

The Acceptance stage is responsible for acquiring the packaged application artifacts and deploying the application to a test environment and then running a Gulp task to execute the end-to-end tests against that environment. Let’s look at the details of each of these four actions in this stage:

  • Deploy App – The Lambda for the application is updated with the code from the Commit stage as a new version. Additionally, the test  alias is moved to this new version. As you may recall from part 2 this alias is used by the test stage of the API Gateway to determine which version of the Lambda function to invoke.

LambdaVersionAliases

  • Deploy API – At this point, this is a no-op. My goal is to have this action use a Swagger file in the source code to update the API Gateway resources, methods, and integrations. This will allow these API changes to affect the API Gateway on each build, where with this current solution would require an update of the CloudFormation stack outside the pipeline to change the API Gateway.
  • Deploy Site – This action publishes all static content (HTML, CSS, JavaScript and images) to a test S3 bucket. Additionally, it published a config.json file to the bucket that the application uses to determine the endpoint for the APIs. Here’s a sample of the file that is created:
{
  "apiBaseurl":"https://rue1bmchye.execute-api.us-west-2.amazonaws.com/test/",
  "version":"20160324-231829"
}
  • End-to-end Testing – This action invokes a Gulp action to run the functional tests. Additionally, it sets and an environment variable with the endpoint URL for the application for the Gulp process to test against.

Sidebar: Continuation Token

One challenge of using Lambda for actions is the current 300 second function execution timeout limit. If you have an action that will take longer than 300 seconds (e.g., launching a CloudFormation stack) you can utilize the continuation token. A continuation token is an opaque value that you can return to CodePipeline to indicate that you are not complete with your action yet. CodePipeline will then reinvoke your action, passing in the continuation token you provided in the prior invocation.

The following code uses the UserParameters  as a maximum number of attempts and uses continuationToken  as a number of attempts. If the action needs more time, it compares the maxAttempts  with the priorAttempts  and if there are still more attempts available, it calls into CodePipeline to signal success and passes a continuation token to indicate that the action needs to be reinvoked.

var jobData = event["CodePipeline.job"].data;
var maxAttempts = parseInt(jobData.actionConfiguration.configuration.UserParameters) || 0
var priorAttempts = parseInt(jobData.continuationToken) || 0;

if(priorAttempts < maxAttempts) {
    console.log("Retrying later.");

    var params = {
        jobId: event["CodePipeline.job"].id,
        continuationToken: (priorAttempts+1).toString()
    };
    codepipeline.putJobSuccessResult(params);

}

Deploy from Production Stage

The Production stage uses the same action definitions from the Acceptance stage to deploy and test the application. The only difference is that it passes in the production S3 bucket name and Lambda ARN to deploy to.

I spent time considering how to do a Blue/Green deployment with this environment. Blue/Green deployment is an approach to reduce deployment risk by launching a duplicate environment for code changes (green environment) and then cutting over traffic from the existing (blue environment) to the new environment. This also affords a safe and quick rollback by switching traffic back to the old (blue) environment.

I looked into doing a DNS based Blue/Green using Route53 Resource Records. This would be accomplished by creating a new API Gateway and Lambda function for each job and using weighted routing to move traffic over from the old API Gateway to the new API Gateway.

I’m not convinced this level of complexity would provide much value however, because given the way Lambda manages version and API Gateway manages deployments, you can easily roll changes back very quickly by moving the Lambda version alias. One limitation though is you cannot do a canary deployment with a single API Gateway and Lambda version aliases. I’m curious what your thoughts are on this, ping me on Twitter @Stelligent with #ServerlessDelivery.

Sidebar: Gulp + CloudFormation

You’ll also notice that there is a gulpfile.js in the dromedary-serverless repo to make it easier to launch and manage the CloudFormation stack. Specifically, you can run gulp pipeline:up  to launch the stack, gulp pipeline:wait  to wait for the pipeline creation to complete and gulp pipeline:status  to see the status of the stack and its outputs. This code has been factored out into its own repo named serverless-pipeline if you’d like to add this type of integration between Gulp and CloudFormation in your own project.

Try it out!

Want to experiment with this stack in your own account? The CloudFormation templates are available for you to run with the link below. First, you’ll want to fork the dromedary repository into your GitHub account. Then you’ll need to provide the following parameters to the stack:

  • Hosted Zone – you’ll need to setup a Route53 hosted zone in your account before launching the stack for the Route53 resource records to be created in.
  • Test DNS Name – a fully qualified hostname (within the Hosted Zone you created) for the test resources (e.g.test.example.com ).
  • Production DNS Name – a fully qualified hostname (within the Hosted Zone you created) for the production resources (e.g.prod.example.com ).
  • OAuth2 Token – your OAuth2 token (see here for details)
  • User Name – your GitHub username

stack-parameters

Conclusion

In this series, we have addressed how achieve the fundamentals of continuous delivery in a serverless environment. Let’s review those fundamentals and how we addressed them:

  • Continuous – We used CodePipeline to run a series of actions against every commit to our GitHub repository.
  • Quality – We built static analysis, unit tests and end-to-end tests into our pipeline and ran them for every commit.
  • Automated – The provisioning of the pipeline and the application was done from a single CloudFormation stack
  • Reproducible – Other than creation of a Route53 Hosted Zone, there were no prerequisites to running this CloudFormation stack
  • Serverless – All the tools chosen where AWS managed services, including Lambda, API Gateway, S3, Route53 and CodePipeline. No servers were harmed in the making of this series.

Please follow us on Twitter to be informed of future articles on Serverless Delivery and other exciting topics.  Also, keep your eye out for a new book set to be released later this year by Stelligent CTO, Paul Duvall, on Continuous Delivery in AWS – which will contain a chapter on serverless delivery.

 

Resources

Serverless Delivery: Bootstrapping the Pipeline (Part 2)

In the first of this three part series on Serverless Delivery, we took a look at the high level architecture of running a continuous delivery pipeline with CodePipeline + Lambda. Our objective is to run the Dromedary application in a serverless environment with a serverless continuous delivery pipeline.

Before we can build the pipeline, we need to have the platform in place to deploy our application to. In this second part of the series we will look at what changes need to be made to a Node.js Express application to run in Lambda and the CloudFormation templates needed to create the serverless resources to host our application.

Prepare Your Application

Lambdas are defined as a function that takes in an event object containing data elements mapped to it in the API Gateway Integration Request. An application using Express in Node.js however expects its request to be initiated from an HTTP request on a socket. In order to run your Express application as a Lambda function, you’ll need some code to mediate between the two frameworks.

Although the best approach would be to have your application natively support the Lambda event, this may not always be feasible. Therefore, I have created a small piece of code to serve as a mediator and put it outside of the Dromedary application in its own module named lambda-express for others to leverage.

lambda-express

Install the module with npm install –save lambda-express  and then use it in your Express application to define the Lambda handler:

var lambdaExpress = require('lambda-express');
var express = require('express');
var app = express();

// ... initialize the app as usual ...

// create a handler function that maps lambda inputs to express
exports.handler = lambdaExpress.appHandler(app);

In the dromedary application, this is available in a separate index.js file. You’ll also notice in the dromedary application, that it passes a callback function rather than the express app to the appHandler function. This allows it to use information on the event to configure the application, in this case via environment variables:

exports.handler = lambdaExpress.appHandler(function(event,context) {
process.env.DROMEDARY_DDB_TABLE_NAME = event.ddbTableName;

var app = require('./app.js');
return app;
});

You now have an Express application that will be able to respond to Lambda events that are generated from API Gateway. Now let’s look at what resources need to be defined in your CloudFormation templates. For the dromedary application, these templates are defined in a separate repository named dromedary-serverless in the pipeline/cfn directory.

Define Website Resources

Sample – site.json

Buckets need to be defined for test and production stages of the pipeline to host the static content of the application. This includes the HTML, CSS, images and any JavaScript that will run in the browser. Each bucket will need a resource like what you see below.

"TestBucket" : {
  "Type" : "AWS::S3::Bucket",
  "Properties" : {
    "AccessControl" : "PublicRead",
    "BucketName" : “www-test.mysite.com”,
    "WebsiteConfiguration" : {
      "IndexDocument" : "index.html"
    }
  }
}

Here are the important pieces to pay attention to:

  • AccessControl – set to to PublicRead  assuming this is a public website.
  • WebsiteConfiguration – add an IndexDocument  entry to define the home page for the bucket.
  • BucketName – needs to match exactly the Name  for the Route53 ResourceRecord you create. For example, if I’m going to setup a DNS record for www-test.mysite.com , then the bucket name should also be www-test.mysite.com .

We will also want Route53 resource records for each of the test and production buckets. Here is a sample record:

"TestSiteRecord": {
  "Type": "AWS::Route53::RecordSetGroup",
  "Properties": {
    "HostedZoneId": “Z00ABC123DEF”,
    "RecordSets": [{
      "Name": “www-test.mysite.com.”,
      "Type": "A",
      "AliasTarget": {
        "HostedZoneId": “Z3BJ6K6RIION7M”,
        "DNSName": “s3-website-us-west-2.amazonaws.com"
      }
    }]
  }
}

Make sure that your record does the following:

  • Name – must be the same as the bucket name above with a period at the end.
  • HostedZoneId – should be specific to your account and for the zone you are hosting (mysite.com in this example).
  • AliasTarget – will be reference the zone id and endpoint for S3 in the region you created the bucket. The zone ids and endpoints can be found in the AWS General Reference Guide.

Declare Lambda Functions

Sample – app.json

Lambda functions will need to be declared for test and production stages of the pipeline to serve the Express application. Each function will only be stubbed out in the CloudFormation template so the API Gateway resources can reference it. Each execution of a CodePipeline job will deploy the latest version of the code as a new version of the function. Here is a sample declaration of the Lambda function in CloudFormation:

"TestAppLambda": {
  "Type" : "AWS::Lambda::Function",
  "Properties" : {
    "Code" : {
      "ZipFile": { "Fn::Join": ["n", [
        "exports.handler = function(event, context) {",
        " context.fail(new Error(500));",
        "};"
      ]]}
    },
    "Description" : "serverless application",
    "Handler" : "index.handler",
    "MemorySize" : 384,
    "Timeout" : 10,
    "Role" : {“Ref”: “TestAppLambdaTrustRole”},
    "Runtime" : "nodejs"
  }
}

Notice the following about the Lambda resource:

  • ZipFile – the default implementation of the function is provided inline. Notice it just returns a 500 error. This will be replaced by real code when CodePipeline runs.
  • MemorySize – this is the only control you have over the system resources allocated to your function. CPU performance is determined by the amount of memory you allocate, so if you need more CPU, increase this number. Your cost is directly related to this number as is the duration of each invocation. There is a sweet spot you need to find where you get the shortest durations for the system resources.
  • Timeout – max time (in seconds) for a given invocation of the function to run before it is forcibly terminated. The maximum value for this is 300 seconds.
  • Role – reference the ARN of the IAM role that you want to assign to your function when it runs. You’ll want to have “Principal”:{“Service”:[“lambda.amazonaws.com”]} in the AssumePolicyDocument to grant the Lambda service access to the sts:AssumeRole action. You’ll also want to include in the policy access to CloudWatch Logs with “Action”: [“logs:CreateLogGroup”,”logs:CreateLogStream”,”logs:PutLogEvents”]

Define API Gateway and Stages

Sample – api-gateway.json

Our Lambda function requires something to receive HTTP requests and deliver them as Lambda events. Fortunately, the AWS API Gateway is a perfect solution for this need. A single API Gateway definition with two stages, one for test and one for production will be defined to provide the public access to your Lambda function defined above. Unfortunately, CloudFormation does not have support yet for API Gateway. However, Andrew Templeton has created a set of custom resources that do a great job of filling the gap. For each package, you will need to create a Lambda function in your CloudFormation template, for example:

"ApiGatewayRestApi": {
  "Type" : "AWS::Lambda::Function",
  "Properties" : {
    "Code" : {
      "S3Bucket": “dromedary-serverless-templates”,
      "S3Key": "cfn-api-gateway-restapi.zip"
    },
    "Description" : "Custom CFN Lambda",
    "Handler" : "index.handler",
    "MemorySize" : 128,
    "Timeout" : 30,
    "Role" : { "Ref": "ApiGatewayCfnLambdaRole" },
    "Runtime" : "nodejs"
  }
}

 

Make sure the role that you create and reference from the Lambda above contains policy statements to give it access to apigateway:*  actions, as well as granting the custom resource access to PassRole  the role defined for API Integration.

{
  "Effect": "Allow",
  "Resource": [
    { "Fn::GetAtt": [ "ApiIntegrationCredentialsRole", "Arn" ] }
  ],
  "Action": [
    "iam:PassRole"
  ]
}

 

Defining the API Gateway above consists of the following six resources. For each one, I will highlight only the important properties to be aware of as well as a picture from the console of what the CloudFormation resource creates:

  1. cfn-api-gateway-restapi – this is the top level API definition.
    apigw-api
    • Name – although not required, you ought to provide a unique name for the API
  2. cfn-api-gateway-resource – a resource (or path) for the API. In the reference application, I’ve created just one root resource that is a wildcard and captures all sub paths.  The sub path is then passed into lambda-express as a parameter and mapped into a path that Express can handle.
     apigw-resource
    • PathPart – define a specific path of your API, or {subpath}  to capture all paths as a variable named subpath
    • ParentId – This must reference the RootResourceId from the restapi resource
      "ParentId": { "Fn::GetAtt": [ "RestApi", "RootResourceId" ] }
  3. cfn-api-gateway-method – defines the contract for a request to an HTTP method (e.g., GET or POST) on the path created above.
    apigw-method
    • HttpMethod – the method to support (GET)
    • RequestParameters – a map of parameters on the request to expect and pass down to the integration
      "RequestParameters": {
        "method.request.path.subpath": true
      }
  4. cfn-api-gateway-method-response – defines the contract for the response from an HTTP method defined above.
     apigw-method-response
    • HttpMethod – the method to support
    • StatusCode – the code to return for this response (e.g., 200)
    • ResponseParameters – a map of the headers to include in the response
      "ResponseParameters": {
        "method.response.header.Access-Control-Allow-Origin": true,
        "method.response.header.Access-Control-Allow-Methods": true,
        "method.response.header.Content-Type": true
      }
  5. cfn-api-gateway-integration – defines where to send the request for a given method defined above.
    apigw-integration
    • Type – for Lambda function integration, choose AWS
    • IntegrationHttpMethod – for Lambda function integration, choose POST
    • Uri – the AWS service URI to integrate with. For Lambda use the example below. Notice that the function name and version are not in the URI, but rather there are variables in their place. This way we can allow the different stages (test and production) to control which Lambda function and which version of that function to call:
      arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2::function:${stageVariables.AppFunctionName}:${stageVariables.AppVersion}/invocations
    • Credentials – The role to run as when invoking the Lambda function
    • RequestParameters – The mapping of parameters from the request to integration request parameters
      "RequestParameters": {
        "integration.request.path.subpath": "method.request.path.subpath"
      }
    • RequestTemplates – The template for the JSON to pass to the Lambda function. This template captures all the context information from API Gateway that lambda-express will need to create the request that Express understands:
      "RequestTemplates": {
        "application/json": {
          "Fn::Join": ["n",[
            "{",
            " "stage": "$context.stage",",
            " "request-id": "$context.requestId",",
            " "api-id": "$context.apiId",",
            " "resource-path": "$context.resourcePath",",
            " "resource-id": "$context.resourceId",",
            " "http-method": "$context.httpMethod",",
            " "source-ip": "$context.identity.sourceIp",",
            " "user-agent": "$context.identity.userAgent",",
            " "account-id": "$context.identity.accountId",",
            " "api-key": "$context.identity.apiKey",",
            " "caller": "$context.identity.caller",",
            " "user": "$context.identity.user",",
            " "user-arn": "$context.identity.userArn",",
            " "queryString": "$input.params().querystring",",
            " "headers": "$input.params().header",",
            " "pathParams": "$input.params().path",",
            "}"
          ] ]
        }
      }
  6. cfn-api-gateway-integration-response – defines how to send the response back to the API client
     apigw-integration-response
    • ResponseParameters – a mapping from the integration response to the method response declared above. Notice the CORS headers that are necessary since the hostname for the API Gateway is different from the hostname provided in the Route53 resource record for the S3 bucket. Without these, the browser will deny AJAX request from the site to these APIs. You can read more about CORS in the API Gateway Developer Guide. Also notice that the response Content-Type is pulled from the contentType attribute in the JSON object returned from the Lambda function:
      "ResponseParameters": {
        "method.response.header.Access-Control-Allow-Origin": "'*'",
        "method.response.header.Access-Control-Allow-Methods": "'GET, OPTIONS'",
        "method.response.header.Content-Type": "integration.response.body.contentType"
      }
    • ResponseTemplates – a template of how to create the response from the Lambda invocation. In the reference application, the Lambda function returns a JSON object with a payload attribute containing a Base64 encoding of the response payload:
      "ResponseTemplates": {
        "application/json": "$util.base64Decode( $input.path('$.payload') )"
      }

       

Additionally, there are two deployment resources created, one for test and one for production. Here is an example of one:

  • cfn-api-gateway-deployment – a deployment has a name that will be the prefix for all resources defined.
     apigw-deployment
    • StageName – “test” or “prod”
    • Variables – a list of variables for the stage. This is where the function name and version are defined for the integration URI defined above:
      "Variables": {
        "AppFunctionName": "MyFunctionName”,
        "AppVersion": "prod"
      }

       

Deploy via Swagger

I would prefer to replace most of the above API Gateway CloudFormation with a Swagger file in the application source repository and have the pipeline use the import tool to create API Gateway deployments from the Swagger. There are a few challenges with this approach. First, the creation of the Swagger requires including the AWS extensions which has a bit of a learning curve. This challenge is made easier by the fact that you can create the API Gateway via the console and then export Swagger. The other challenge is that the import tool is a Java based application that requires Maven to run. This may be difficult to get working in a Lambda invocation from CodePipeline especially given the 300 second timeout.  I will however be spending some time researching this option and will blog about the results.

Stay Tuned!

Now that we have all the resources in place to deploy our application to, we can build a serverless continuous delivery pipeline with CodePipeline and Lambda. Next week we conclude this series with the third and final part looking at the details of the CloudFormation template for the pipeline and each stage of the pipeline as well as the Lambda functions that support them. Be sure to check it out!

Resources

Serverless Delivery: Architecture (Part 1)

If your application tech stack doesn’t need servers, why should your continuous delivery pipeline? Serverless applications deserve serverless delivery!

The software development discipline of continuous delivery has had a tremendous impact on decreasing the cost and risk of delivering changes while simultaneously increasing code quality by ensuring that software systems are always in a releasable state. However, when applying the tools and techniques that exist for this practice to serverless application frameworks and platforms, sometimes existing toolsets do not align well with these new approaches. This post is the first in a three-part series that looks at how to implement the same fundamental tenets of continuous delivery while utilizing tools and techniques that complement the serverless architecture in Amazon Web Services (AWS).

Here are the requirements of the serverless delivery pipeline:

  • Continuous – The pipeline must be capable of taking any commit on master that passes all test cases to production
  • Quality – The pipeline must include unit testing, static code analysis and functional testing of the application
  • Automated – The provisioning of the pipeline and the application must be done from a single CloudFormation command
  • Reproducible – The CloudFormation template should be able to run on a new AWS account with no additional setup other than creation of a Route53 Hosted Zone
  • Serverless – All layers of the application must run on platforms that meet the definition of serverless as described in the next section

What is Serverless?

What exactly is a serverless platform? Obviously, there is still hardware at some layer in the stack to host the application. However, what Amazon has provided with Lambda is a platform where developers no longer need to think about the following:

  • Operating System  – no need to select, secure, configure, administer or patch the OS
  • Servers– no cost risk of over-provisioning and no performance risk of under-provisioning
  • Capacity – no need to monitor utilization and scale capacity based on load
  • High Availability– compute resources are available across multiple AZs

In summary, a serverless platform is one in which an application can be deployed on top of without having to provision or administer any of the resources within the platform. Just show up with some code and the platform handles all the ‘ilities’.

app-overview

The diagram above highlights the components that are used in this serverless application. Let’s look at each one individually:

  • Node.js + Express – In the demo application for this blog series, we will be deploying a Node.js JavaScript application that leverages the Express framework into Lambda. There is a small bit of “glue” code that we will highlight later in the series to adapt the Lambda contract to the Express contract.
  • Lambda – This is where your application logic runs. You deploy your code here but do not need to specify the number of servers or size of the servers to run the code on. You only pay for the number of requests and amount of time those requests take to execute.
  • API Gateway – The API Gateway exposes your Lambda function at an HTTP endpoint. It provides capabilities such as authorization, policy enforcement, rate limiting and data transformation as a service that is entirely managed by Amazon.
  • DynamoDB – Dynamic data is stored in a DynamoDB table. DynamoDB is a NoSQL datastore that has both infinitely scalable storage capacity and throughput capacity which is entirely managed by Amazon.
  • S3 – All static content including HTML, CSS, images and client-side JavaScript is stored in an S3 bucket and served as a static website directly from S3.

The diagram below compares the pricing for running a Node.js application with Lambda and API Gateway versus a pair of EC2 instances and an ELB. Notice that for the m4.large, the break even is around two million requests per day. It is important to mention that 98.8% of the cost of the serverless deployment is the API Gateway. The cost of running the application out of Lambda is insignificant relative to the costs in using API Gateway.

serverless-app-cost

This cost analysis shows that applications/environments with low transaction volume can realize cost savings by running on Lambda + API Gateway, but the cost of API Gateway will become cost prohibitive at higher scale.

What is Serverless Delivery?

Serverless delivery is just the application of serverless platforms to achieve continuous delivery fundamentals. Specifically, a serverless delivery pipeline does not include tools such as Jenkins or resources such as EC2, Autoscaling Groups, and ELBs.

pipeline

The diagram above shows the technology used to accomplish serverless delivery for the sample application. Let’s look at what each component provides:

  • AWS CodePipeline – Orchestrates various Lambda tasks to move code that was checked into GitHub forward towards production.
  • AWS S3 Artifact Bucket – Each action in the pipeline can create a new artifact. The artifact becomes an output from the action in the pipeline and is stored in an S3 bucket to become an input for the next action.
  • AWS Lambda – You create Lambda functions to do the work of individual actions in the pipeline. For example, running a gulp task on a repository is handled by a Lambda function.
  • NPM and Gulp – NPM is used for resolving all the dependencies of a given repository. Gulp is used for defining the tasks of the repository, such as running unit tests and packaging artifacts.
  • Testing toolsJSHint is used for performing static analysis of the code, Mocha is a unit test framework for JavaScript and Chai is a BDD assertion library for describing the unit tests.
  • AWS CloudFormation – CFN templates are used to create and update all these resources in a serverless stack.

Serverless delivery for traditional architectures?

Although this serverless delivery architecture could be applied to more traditional application architectures (e.g., a Java application on EC2 and ELB resources) the challenge might be having pipeline actions complete within the 300 second Lambda maximum. For example, running Maven phases within a single Lambda function invocation, including the resolving of dependencies, compilation, unit testing and packaging would likely be difficult. There may be opportunities to split up the goals into multiple invocations and persist state to S3, but that is beyond the scope of this series.

Pricing

The pricing model for Lambda is favorable for applications that have an idle time and the cost grows linearly with the number of executions. The diagram below compares the pricing for running the pipeline with Lambda and CodePipeline against a Jenkins server running on an EC2 instance. For best performance, the Jenkins server ought to run on an m4.large instance, but just to highlight the savings, m3.medium and t2.micro instances were evaluated as well. Notice that for the m4.large, the break even happens after you are doing over 600 builds per day and even with a t2.micro, the break even doesn’t happen until well over 100 builds per day.

serverless-delivery-cost
In conclusion, running a continuous delivery pipeline with CodePipeline + Lambda is very attractive based on the cost efficiency of utility pricing, the simplicity of using a managed services environment, and the tech stack parity of using Node.js for both the application and the pipeline.

Stay Tuned!

Next week we will dive into part two of this series, looking at what changes need to be made for an Express application to run in Lambda and the CloudFormation templates needed to create a serverless delivery pipeline. Finally, we will conclude with part three going into the details of each stage of the pipeline and the Lambda functions that support them.

Resources

Running AWS Lambda Functions in AWS CodePipeline using CloudFormation

Recently, AWS announced that they’ve added support for triggering AWS Lambda functions into AWS CodePipeline – AWS’ Continuous Delivery service. They also provided some great step-by-step documentation to describe the process for configuring a new stage in CodePipeline to run a Lambda function. In this article, I’ll describe how I codified the provisioning of all of the AWS resources in the documentation using CloudFormation.

aws_code_pipeline_lambda

This announcement is really big news as it opens up a whole realm of possibilities about what can be run from CodePipeline. Now, you can run event-driven functions any time you want from your pipelines. With this addition, CodePipeline added a new Invoke action category that adds to the list of other actions such as Build, Deploy, Test and Source.

NOTE: All of the CloudFormation examples in this article are defined in the codepipeline-lambda.json file.

tl;dr

If you’d rather not read the detailed explanation of the resources and code snippets of this solution, just click on the CloudFormation Launch Stack button below to automatically provision the AWS resources described herein. You will be charged for your AWS usage. 
Launch Stack

CloudFormation

I went through the 20+ pages of instructions which were easy to follow but, as I often do when going through this kind of documentation, I thought about how I’d make it easier for me and others to run it again without copying/pasting, clicking multiple buttons and so on. In other words, I’m lazy and don’t enjoy repeatedly going over the same thing again and again and I figured this would be something I’d (and others) like to use often in the future. Of course, this leads me to writing a template in CloudFormation since I can define everything in code and type a single command or click a button to reliably and repeatedly provision all the necessary resources to run Invoke actions within a Lambda stage in CodePipeline.

There are six core services that compose this infrastructure architecture. They are CloudFormation, CodePipeline, Lambda, IAM, EC2 and CodeDeploy.

To launch the infrastructure stacks that make up this solution, type the following from the command line. The command will only work if you’ve installed the AWS CLI.

Command for launching CodePipeline Lambda stacks

aws cloudformation create-stack 
--stack-name CodePipelineLambdaStack  
--template-body https://raw.githubusercontent.com/stelligent/stelligent_commons/master/cloudformation/codepipeline-lambda.json 
--region us-east-1 
--disable-rollback --capabilities="CAPABILITY_IAM" 
--parameters ParameterKey=KeyName,ParameterValue=YOUREC2KEYPAIRNAME

EC2

From my CloudFormation template, I launched a single EC2 instance that installed a CodeDeploy agent onto it. I used the sample provided by AWS at http://s3.amazonaws.com/aws-codedeploy-us-east-1/templates/latest/CodeDeploy_SampleCF_Template.json and added one small modification to return the PublicIp of the EC2 instance after it’s launched as a CloudFormation Output. Because of this modification, I created a new template based on AWS’ sample.
CloudFormation JSON to define EC2 instance used by CodeDeploy

    "CodeDeployEC2InstancesStack":{
      "Type":"AWS::CloudFormation::Stack",
      "Properties":{
        "TemplateURL":"https://s3.amazonaws.com/stelligent-training-public/public/codedeploy/codedeploy-ec2.json",
        "TimeoutInMinutes":"60",
        "Parameters":{
          "TagValue":{
            "Ref":"AWS::StackName"
          },
          "KeyPairName":{
            "Ref":"KeyName"
          }
        }
      }
    },

When the stack is complete, you’ll see that one EC2 instance has been launched and automatically tagged with the name you entered when naming your CloudFormation stack. This name is used to run CodeDeploy operations on instance(s) with this tag.
codepipeline_lambda_ec2

CodeDeploy

AWS CodeDeploy automates code deployments to any instance. Previously, I had automated the steps of the Simple Pipeline Walkthrough which included the provisioning of AWS CodeDeploy resources as well so I used this CloudFormation template as a starting point. I uploaded the sample Linux app provided by CodePipeline in the walkthrough to Amazon S3 and used S3 as the Source action in the Source stage in my pipeline in CodePipeline. Below, you see a snippet of defining the CodeDeploy stack from the codepipeline-lambda.json. The nested stack defined in the TemplateURL property defines the CodeDeploy application and the deployment group.

CloudFormation JSON to define EC2 instance used by CodeDeploy

    "CodeDeploySimpleStack":{
      "Type":"AWS::CloudFormation::Stack",
      "DependsOn":"CodeDeployEC2InstancesStack",
      "Properties":{
        "TemplateURL":"https://s3.amazonaws.com/stelligent-training-public/public/codedeploy/codedeploy-deployment.json",
        "TimeoutInMinutes":"60",
        "Parameters":{
          "TagValue":{
            "Ref":"AWS::StackName"
          },
          "RoleArn":{
            "Fn::GetAtt":[
              "CodeDeployEC2InstancesStack",
              "Outputs.CodeDeployTrustRoleARN"
            ]
          },
          "Bucket":{
            "Ref":"S3Bucket"
          },
          "Key":{
            "Ref":"S3Key"
          }
        }
      }
    },

The screenshot below is that of a CodeDeploy deployment that was generated from the CloudFormation stack launch.
codepipeline_lambda_codedeploy

The CodeDeploy provisioning of this is described in more detail in my article on this topic: Automating AWS CodeDeploy Provisioning in CloudFormation.

CodePipeline

I took a CodePipeline example that I’d written in CloudFormation that defines a simple three-stage pipeline (based on the Simple Pipeline Walkthrough)  and added a new stage in the CloudFormation Resource block to invoke the Lambda function. If I were manually adding this stage, I’d go to my specific pipeline in AWS CodePipeline, click add Stage and then add an action to the stage. Below, you see of a screenshot of what you’d do if you were manually defining this configuration within an AWS CodePipeline action. This is also what got generated from the CloudFormation stack.

codepipeline_lambda_stage

AWS::CodePipeline::Pipeline

At the beginning of the snippet below, you see the use of the AWS::CodePipeline::Pipeline CloudFormation resource type. It has dependencies on the CodeDeploySimpleStack and CodePipelineLambdaTest resources. One of the reasons for this is that there needs to be an EC2 instance type defined already so that I can get access to the PublicIp that I use to run a Lambda function later when verifying the application is up and running. The other is that we need to set the FunctionName property of the Configuration of the Lambda stage in CodePipeline. This function name is generated by the AWS::Lambda::Function resource type that I’ll describe later. By using this approach, you don’t need to know the name of the Lambda function when defining the CloudFormation template.
CloudFormation JSON to define IAM Role for Lambda function execution

    "GenericPipeline":{
      "Type":"AWS::CodePipeline::Pipeline",
      "DependsOn":[
        "CodeDeploySimpleStack",
        "CodePipelineLambdaTest"
      ],
      "Properties":{
        "DisableInboundStageTransitions":[
          {
            "Reason":"Demonstration",
            "StageName":"Production"
          }
        ],
        "RoleArn":{
          "Fn::Join":[
            "",
            [
              "arn:aws:iam::",
              {
                "Ref":"AWS::AccountId"
              },
              ":role/AWS-CodePipeline-Service"
            ]
          ]
        },
        "Stages":[
...
          {
            "Name":"LambdaStage",
            "Actions":[
              {
                "InputArtifacts":[

                ],
                "Name":"MyLambdaAction",
                "ActionTypeId":{
                  "Category":"Invoke",
                  "Owner":"AWS",
                  "Version":"1",
                  "Provider":"Lambda"
                },
                "OutputArtifacts":[

                ],
                "Configuration":{
                  "FunctionName":{
                    "Ref":"CodePipelineLambdaTest"
                  },
                  "UserParameters":{
                    "Fn::Join":[
                      "",
                      [
                        "http://",
                        {
                          "Fn::GetAtt":[
                            "CodeDeployEC2InstancesStack",
                            "Outputs.PublicIp"
                          ]
                        }
                      ]
                    ]
                  }
                },
                "RunOrder":1
              }
            ]
          },...

Lambda

AWS Lambda lets you run event-based functions without provisioning or managing servers. That said, there’s still a decent amount of configuration you’ll need to define in running your Lambda functions. In the example provided by AWS, the Lambda function tests whether it can access a website without receiving an error. If it succeeds, the CodePipeline action and stage succeed, turn to green, and it automatically transitions to the next stage or completes the pipeline. If it fails, that pipeline instance fails, turns red, and ceases any further actions from occurring. It’s a very typical test you’d run to be sure your application was successfully deployed. In the example, AWS has you manually enter the URL for the application. Since this requires manual intervention, I needed to figure out a way to get this URL dynamically. I did this by setting the PublicIp of the EC2 instance that was launched earlier in the stack as an Output of the nested stack. Then I used this PublicIp as an input to the UserParameters property of the Invoke action within the Lambda stage that I defined in CloudFormation for my CodePipeline pipeline.

Once the function has been generated by the stack, you’ll be able to go to a list of Lambda functions in your AWS Console and see the function that was created from the stack.

codepipeline_lambda_function

AWS::IAM::Role

In the CloudFormation code snippet you see below, I’m defining an IAM role that’s capable of calling Lambda functions.
CloudFormation JSON to define IAM Role for Lambda function execution

    "CodePipelineLambdaRole":{
      "Type":"AWS::IAM::Role",
      "Properties":{
        "AssumeRolePolicyDocument":{
          "Version":"2012-10-17",
          "Statement":[
            {
              "Effect":"Allow",
              "Principal":{
                "Service":[
                  "lambda.amazonaws.com"
                ]
              },
              "Action":[
                "sts:AssumeRole"
              ]
            }
          ]
        },
        "Path":"/"
      }
    },

AWS::IAM::Policy

The code snippet below depends on the creation of the IAM role I showed in the example above. The IAM policy that’s attached to the IAM role provides access to the AWS logs and the CodePipeline results so that it can signal success or failure to the CodePipeline action that I defined earlier.
CloudFormation JSON to define IAM Policy for IAM Role for Lambda function execution

    "LambdaCodePipelineExecutionPolicy":{
      "DependsOn":[
        "CodePipelineLambdaRole"
      ],
      "Type":"AWS::IAM::Policy",
      "Properties":{
        "PolicyName":"LambdaRolePolicy",
        "Roles":[
          {
            "Ref":"CodePipelineLambdaRole"
          }
        ],
        "PolicyDocument":{
          "Version":"2012-10-17",
          "Statement":[
            {
              "Effect":"Allow",
              "Action":[
                "logs:*"
              ],
              "Resource":[
                "arn:aws:logs:*:*:*"
              ]
            },
            {
              "Effect":"Allow",
              "Action":[
                "codepipeline:PutJobSuccessResult",
                "codepipeline:PutJobFailureResult"
              ],
              "Resource":[
                "*"
              ]
            }
          ]
        }
      }
    },

AWS::Lambda::Function

In the code snippet below, you see how I’m defining the Lambda function in CloudFormation. There are several things to point out here. I uploaded some JavaScript (Node.js) code to S3 with the name Archive.zip into a bucket specified by the S3Bucket parameter that I set when I launched the CloudFormation stack. This S3 bucket needs to have S3 Versioning enabled on it. Moreover, the Archive.zip file needs to have the .js file used by Lambda in the root of the Archive.zip. Keep in mind that I can call the .zip file whatever I want, but once I name the file and upload it then my CloudFormation template needs to refer to the correct name of the file.

Also, you see that I’ve defined a Handler named validateurl.handler. This means that the JavaScript file in the Archive.zip that hosts the file(s) that Lambda runs must be named validateurl.js. If I want to use a different name, I must change both the JavaScript filename and the CloudFormation template that references it.

CloudFormation JSON to define Lambda function execution

    "CodePipelineLambdaTest":{
      "Type":"AWS::Lambda::Function",
      "DependsOn":[
        "CodePipelineLambdaRole",
        "LambdaCodePipelineExecutionPolicy"
      ],
      "Properties":{
        "Code":{
          "S3Bucket":{
            "Ref":"S3Bucket"
          },
          "S3Key":"Archive.zip"
        },
        "Role":{
          "Fn::GetAtt":[
            "CodePipelineLambdaRole",
            "Arn"
          ]
        },
        "Description":"Validate a website URL",
        "Timeout":20,
        "Handler":"validateurl.handler",
        "Runtime":"nodejs",
        "MemorySize":128
      }
    },

Lambda Test Function

With all of this configuration to get something to run, sometimes it’s easy to overlook that we’re actually executing something useful and not just configuring the support infrastructure. The snippet below is the actual test that gets run as part of the Lambda action in the Lambda stage that I defined in the CloudFormation template for CodePipeline. This code is taken directly from the Integrate AWS Lambda Functions into Pipelines in AWS CodePipeline instructions from AWS. This JavaScript code verifies that it can access the supplied website URL of the deployed application. If it fails, it signals for CodePipeline to cease any further actions in the pipeline.

JavaScript to test access to a website

var assert = require('assert');
var AWS = require('aws-sdk');
var http = require('http');

exports.handler = function(event, context) {

    var codepipeline = new AWS.CodePipeline();

    // Retrieve the Job ID from the Lambda action
    var jobId = event["CodePipeline.job"].id;

    // Retrieve the value of UserParameters from the Lambda action configuration in AWS CodePipeline, in this case a URL which will be
    // health checked by this function.
    var url = event["CodePipeline.job"].data.actionConfiguration.configuration.UserParameters;

    // Notify AWS CodePipeline of a successful job
    var putJobSuccess = function(message) {
        var params = {
            jobId: jobId
        };
        codepipeline.putJobSuccessResult(params, function(err, data) {
            if(err) {
                context.fail(err);
            } else {
                context.succeed(message);
            }
        });
    };

    // Notify AWS CodePipeline of a failed job
    var putJobFailure = function(message) {
        var params = {
            jobId: jobId,
            failureDetails: {
                message: JSON.stringify(message),
                type: 'JobFailed',
                externalExecutionId: context.invokeid
            }
        };
        codepipeline.putJobFailureResult(params, function(err, data) {
            context.fail(message);
        });
    };

    // Validate the URL passed in UserParameters
    if(!url || url.indexOf('http://') === -1) {
        putJobFailure('The UserParameters field must contain a valid URL address to test, including http:// or https://');
        return;
    }

    // Helper function to make a HTTP GET request to the page.
    // The helper will test the response and succeed or fail the job accordingly
    var getPage = function(url, callback) {
        var pageObject = {
            body: '',
            statusCode: 0,
            contains: function(search) {
                return this.body.indexOf(search) > -1;
            }
        };
        http.get(url, function(response) {
            pageObject.body = '';
            pageObject.statusCode = response.statusCode;

            response.on('data', function (chunk) {
                pageObject.body += chunk;
            });

            response.on('end', function () {
                callback(pageObject);
            });

            response.resume();
        }).on('error', function(error) {
            // Fail the job if our request failed
            putJobFailure(error);
        });
    };

    getPage(url, function(returnedPage) {
        try {
            // Check if the HTTP response has a 200 status
            assert(returnedPage.statusCode === 200);
            // Check if the page contains the text "Congratulations"
            // You can change this to check for different text, or add other tests as required
            assert(returnedPage.contains('Congratulations'));

            // Succeed the job
            putJobSuccess("Tests passed.");
        } catch (ex) {
            // If any of the assertions failed then fail the job
            putJobFailure(ex);
        }
    });
};

Post-Commit Git Hook for Archiving and Uploading to S3

I’m in the process of figuring out how to add a post-commit hook that moves files committed to a specific directory in a Git repository, zip up the necessary artifacts and upload to a pre-defined directory in S3 so that I can remove this manual activity as well.

Summary

By adding the ability to invoke Lambda functions directly from CodePipeline, AWS has opened a whole new world of what can be orchestrated into our software delivery processes in AWS. You learned how to automate the provisioning of not just the Lambda configuration, but dependent AWS resources including the automated provisioning of your pipelines in AWS CodePipeline. If you have any questions, reach out to us on Twitter @stelligent or @paulduvall.

Resources

Automating AWS CodeDeploy Provisioning in CloudFormation

Over the past few weeks, I’ve been describing how I’m automating the provisioning of several of the AWS Code Services including CodePipeline and Custom CodePipeline Actions. This time, I’m describing a way of provisioning AWS CodeDeploy in CloudFormation. For now, I’m doing the automation against a generic application provided by AWS. I’ll apply it to our Dromedary demo application later and publish it in a future post.

DeploymentManagement_CodeDeploy

   About AWS CodeDeploy

As AWS describes: “AWS CodeDeploy is a service that automates code deployments to any instance, including Amazon EC2 instances and instances running on-premises. AWS CodeDeploy makes it easier for you to rapidly release new features, helps you avoid downtime during application deployment, and handles the complexity of updating your applications. You can use AWS CodeDeploy to automate software deployments, eliminating the need for error-prone manual operations, and the service scales with your infrastructure so you can easily deploy to one instance or thousands.” Below, I describe the core components that make up CodeDeploy.

  • AppSpec – The appspec.yml must be in the root directory of an AWS CodeDeploy deployment bundle. Here’s an example appspec.yml file from AWS.
  • Application – The application is the container for the deployment and deployment configuration. The application uses the appspec.yml to define how the lifecycle events behave.
  • Deployment Group – Define how the deployment behaves. In the deployment group, you set the previously-defined Application, Deployment Configuration, on which EC2 instances to run the deployment and any AutoScaling configuration.
  • Deployment Configuration – There are three default configurations provided by CodeDeploy: CodeDeployDefault.OneAtATimeCodeDeployDefault.AllAtOnce and CodeDeployDefault.HalfAtATime. You can describe custom configurations as well.
  • Lifecycle Events – Each deployment triggers certain events that run in a pre-defined sequence. The actual code that is run is defined in the appspec.yml using hooks section of the file. Here are the events that get triggered as part of a typical CodeDeploy deployment.
    • ApplicationStop
    • DownloadBundle
    • BeforeInstall
    • Install
    • AfterInstall
    • ApplicationStart
    • ValidateService

In order for CodeDeploy to work, you need to install the CodeDeploy agent on each EC2 instance on which you’re running CodeDeploy.  CodeDeploy communicates with the agents via HTTPS over port 443. The agent contains code that has CodeDeploy domain-specific knowledge and uses the defined configuration to run through its lifecycle events. 

ManagementTools_CloudFormation  Executing the CloudFormation Template

Below, you see an example of running the template that I’ve defined. You’ll need to define your own stack name and EC2KeyPairName. From your AWS CLI, run a command to launch the CloudFormation stack. You can also launch the same using the CloudFormation console.

aws cloudformation create-stack --stack-name MyStackName --template-url https://s3.amazonaws.com/stelligent-training-public/public/codedeploy/codedeploy-master.json
--region us-east-1 --disable-rollback --capabilities="CAPABILITY_IAM"
--parameters ParameterKey=S3Bucket,ParameterValue=aws-codedeploy-us-east-1
ParameterKey=S3Key,ParameterValue=public/Sample_Linux_App.zip
ParameterKey=EC2TagValue,ParameterValue=CodeDeployEC2Tag
ParameterKey=EC2KeyPairName,ParameterValue=MyEC2KeyPairName

It’ll take about 10-15 minutes to launch the stacks that launch the EC2 instance, install the CodeDeploy agent, configure and run a deployment of the application. You can visually verify the application works by going to the CodeDeploy console (shown in Figure 1), select an application, then a deployment and click on the link under the Instance ID column. From the EC2 console, find the Public IP and prepend http:// to it from your web browser. codedeploy-deployment

Figure 1: AWS CodeDeploy deployment status

Architecture

There are two CloudFormation templates to define the EC2 instances, S3 Distribution Location, IAM and CodeDeploy resources along with its overall orchestration via nested stacks. They are:

  • codedeploy-master.json – The master template orchestrates the execution of the other CloudFormation templates using nested CloudFormation stacks and the DependsOn attribute. It uses the Outputs from one stack as input parameters to the next calling template. Using the DependsOn attribute ensures that the resources have been provisioned prior to calling the next template.
  • codedeploy-deployment.json – Automates the provisioning of AWS CodeDeploy resources including the CodeDeploy Application and Deployment. It points to an S3 bucket and key where the sample application zip file is stored.

The resulting infrastructure is illustrated in Figure 2. codedeploy-arch

Figure 2: Infrastructure Architecture Diagram

Implementation

There are several parts to automating the process of downloading the sample application, provisioning EC2 instances with the CodeDeploy agent, provisioning CodeDeploy to create an application and to run a deployment for that application using CodeDeploy. The first example snippet below can be found in the codedeploy-master.json CloudFormation template. This launches a new stack from the AWS CloudFormation template that provisions EC2 instances and installs the CodeDeploy agent.

    "CodeDeployEC2InstancesStack":{
      "Type":"AWS::CloudFormation::Stack",
      "Properties":{
        "TemplateURL":"http://s3.amazonaws.com/aws-codedeploy-us-east-1/templates/latest/CodeDeploy_SampleCF_Template.json",
        "TimeoutInMinutes":"60",
        "Parameters":{
          "TagValue":{
            "Ref":"EC2TagValue"
          },
          "KeyPairName":{
            "Ref":"EC2KeyPairName"
          }
        }
      }
    },

The next snippet from the same codedeploy-master.json template uses the EC2 tag value that was set in the first stack as a way for this template to determine on which EC2 instances it will run the CodeDeploy deployment. It also uses the DependsOn attribute to ensure that the stack has been created before attempting to launch this stack – since it requires resources from the first stack.

    "CodeDeploySimpleStack":{
      "Type":"AWS::CloudFormation::Stack",
      "DependsOn":"CodeDeployEC2InstancesStack",
      "Properties":{
        "TemplateURL":"https://s3.amazonaws.com/stelligent-training-public/public/codedeploy/codedeploy-deployment.json",
        "TimeoutInMinutes":"60",
        "Parameters":{
          "TagValue":{
            "Ref":"EC2TagValue"
          },
          "RoleArn":{
            "Fn::GetAtt":[
              "CodeDeployEC2InstancesStack",
              "Outputs.CodeDeployTrustRoleARN"
            ]
          },
          "Bucket":{
            "Ref":"S3Bucket"
          },
          "Key":{
            "Ref":"S3Key"
          }
        }
      }
    }

In the codedeploy-deployment.json template snippet below, I’m defining the CodeDeploy application. This generates a unique identifier that’s used in the next resource definition in the template.

    "MyApplication":{
      "Type":"AWS::CodeDeploy::Application"
    },

In the snippet below (also from codedeploy-deployment.json), I’m defining how my deployment will behave using the CodeDeploy Deployment Group. Once again, I use the DependsOn attribute to ensure that the CodeDeploy application exists prior to provisioning the deployment group since it needs to already exist. Then, I find the application bundle in S3 using the S3Location property. From the command-line execution snippet described earlier, you’ll see that I’m passing aws-codedeploy-us-east-1 as the S3 bucket parameter and samples/latest/SampleApp_Linux.zip as the S3 Key. This resolves to https://s3.amazonaws.com/aws-codedeploy-us-east-1/samples/latest/SampleApp_Linux.zip.

    "MyDeploymentGroup":{
      "Type":"AWS::CodeDeploy::DeploymentGroup",
      "DependsOn":"MyApplication",
      "Properties":{
        "ApplicationName":{
          "Ref":"MyApplication"
        },
        "Deployment":{
          "Description":"First time",
          "IgnoreApplicationStopFailures":"true",
          "Revision":{
            "RevisionType":"S3",
            "S3Location":{
              "Bucket":{
                "Ref":"Bucket"
              },
              "BundleType":"Zip",
              "Key":{
                "Ref":"Key"
              }
            }
          }
        },
        "Ec2TagFilters":[
          {
            "Key":{
              "Ref":"TagKey"
            },
            "Value":{
              "Ref":"TagValue"
            },
            "Type":"KEY_AND_VALUE"
          }
        ],
        "ServiceRoleArn":{
          "Ref":"RoleArn"
        }
      }

Troubleshooting

As you’re experimenting with CodeDeploy and automating the provisioning of CodeDeploy in CloudFormation, you’ll likely experience a few problems along the way. There were a couple of useful suggestions provided to me by AWS Support.

      • Get the logs – SSH into the EC2 instances on which the CodeDeploy agent was installed and view the contents of the /opt/codedeploy-agent/deployment-root and /var/log/ directories. You might choose to zip up the directories and download them for easy viewing as demonstrated below.

zip -r varlog.zip /var/log

zip -r deployment-root.zip /opt/codedeploy-agent/deployment-root

      • View in CloudWatch – I haven’t tried this yet, but there’s an interesting article on the AWS DevOps blog about viewing the CodeDeploy logs in CloudWatch: View AWS CodeDeploy logs in Amazon CloudWatch console
      • Turn off Auto Rollback in CloudFormation – When you’re using the AWS CloudFormation Console and you accept all the default options, it will auto rollback all the stacks. This makes it more difficult to troubleshoot when something does go wrong. Instead, you might choose to call from the command line as shown in the example below. You can still turn off auto rollback using the console, but it’s easier to forget.
aws cloudformation create-stack --stack-name MyStackName --template-body file://mystack.json --region us-east-1 --disable-rollback

Feature Backlog

There’s some additional functionality that I plan to implement in the coming weeks as described below.

      • Use Elastic Load Balancing and AutoScaling so that it’s a more highly available solution with a single endpoint
      • Create a deployment pipeline using AWS CodePipeline and automate the provisioning of the pipeline in AWS CloudFormation
      • Implement CodeDeploy in CloudFormation as part of the Dromedary Demo pipeline.
      • Include re-deployment to the same EC2 instances when an application change occurs

Useful Resources

 

Provisioning AWS CodePipeline with CloudFormation

Since our entire focus at Stelligent is to help our customers apply Continuous Delivery in Amazon Web Services (AWS), we were really excited when we learned that AWS CloudFormation added support for AWS CodePipeline. I spent some time upgrading our Dromedary demo scripts to incorporate this new functionality. This article describes the process we applied in making the necessary changes along with some bonus functionality we were able to implement a more fully automated solution – thanks to Corey at AWS Support.

When the AWS CodePipeline team released its Continuous Delivery service in July 2015, they provided a custom JSON-based DSL for it which we used to automate the provisioning of CodePipeline itself but we didn’t get it into the 100% automated state for which we’d been hoping. It was close, but were “chomping at the bit” for the time when AWS would begin providing CloudFormation support. Thankfully, now’s the time.

A couple of Stelligent’s engineers wrote the first version of the demo that we used at the “Infrastructure as Code” breakout session as part of AWS re:Invent 2015 so they’d already been through some of the things I learned while adding and updating our scripts.

Below, you’ll find the steps for getting this demo running on your own AWS account. All of the code is freely available and open source.

Running from CloudFormation

Click on the Launch Stack button below.


Enter the required parameters. In particular: Stack nameDDBTableNameGitHubToken, KeyName and ProdHostedZone. See the figure below and complete the rest of the steps to launch the stack. It’ll take approximately 20 minutes to launch the stack and then launch a CodePipeline pipeline instance that launches an application stack. You can see the README for other ways to run this template from CloudFormation.

dromedary_master

Once all the required AWS resources are bootstrapped, it automatically launches a pipeline instance in CodePipeline as shown below.

codepipeline_cfn

Running from the Command Line

You’ll need to run the command below from a computer on which the AWS CLI has been installed.

Launching CloudFormation stack from command line

aws cloudformation create-stack 
--stack-name DromedaryStack  
--template-body https://raw.githubusercontent.com/stelligent/dromedary/master/pipeline/cfn/dromedary-master.json 
--region us-east-1 
--disable-rollback --capabilities="CAPABILITY_IAM" 
--parameters ParameterKey=KeyName,ParameterValue=YOURKEYPAIR 
    ParameterKey=Branch,ParameterValue=master 
    ParameterKey=BaseTemplateURL,ParameterValue=https://s3.amazonaws.com/stelligent-training-public/master/ 
    ParameterKey=GitHubUser,ParameterValue=YOURGITHUBUSER 
    ParameterKey=GitHubToken,ParameterValue=YOURGITHUBTOKEN 
    ParameterKey=DDBTableName,ParameterValue=YOURUNIQUEDDBTABLENAME 
    ParameterKey=ProdHostedZone,ParameterValue=.YOURHOSTEDZONE

For more information, you can see the README at https://github.com/stelligent/dromedary.

The steps I went through to create this CloudFormation template were fairly straightforward. First, I started with a CloudFormation template that I had implemented for another effort and removed most everything except for the core structure including the AWSTemplateFormatVersion, an empty Parameters block and an empty Resources block. The core Resources Type for CodePipeline is – as you might’ve guessed – AWS::CodePipeline::Pipeline. After this, I got the name of an existing pipeline that we’d created using the AWS CodePipeline JSON DSL from our AWS CodePipeline and ran this command:

aws codepipeline get-pipeline --name YOUR_PIPELINE_NAME > pipeline.json

This provided a JSON structure with all the stages and actions already defined in code so I was able to copy and paste within my AWS::CodePipeline::Pipeline Resources construct in the CloudFormation template. Since the CodePipeline DSL produces a slightly different case than CloudFormation, I needed to update the letter case for several properties to conform to the CloudFormation standard.

Then, I added several parameters to the template including GitHubToken and GitHubUser. Finally, I ran through several testing scenarios. Be sure to run through all the steps in README as well since it’s important that you enable security for Jenkins.

Summary

Essentially, until AWS implements a service in CloudFormation, we don’t really consider it a service we can use in the manner we like so we’re really happy that CodePipeline can now be provisioned in CloudFormation. Moreover, when implementing the CodePipeline provisioning in CloudFormation, we configured it so that all of CodePipeline including the Github configuration is automated so there are no manual steps anymore (except for enabling Jenkins Global Security). If you have any questions or comments, you can contact us at info@stelligent.com or consult the contact information at https://github.com/stelligent/dromedary.

Stelligent is an expert in deploying continuous delivery pipelines in AWS. If you are looking for help moving your applications into CodePipeline, or another type of continuous delivery pipeline, we can definitely help you with that. Reach out to us here.

Useful Resources