Automating ECS: Orchestrating in CodePipeline and CloudFormation (Part 2)
In my first post on automating the EC2 Container Service (ECS), I described how I automated the provisioning of ECS in AWS CloudFormation using its JSON-based DSL.
In this second and last part of the series, I will demonstrate how to create a deployment pipeline in AWS CodePipeline to deploy changes to ECS Docker images in the EC2 Container Registry (ECR).
In doing this, you’ll not only see how to automate the creation of the infrastructure but also automate the deployment of the application and its infrastructure via Docker containers. This way you can commit infrastructure, application and deployment changes as code to your version-control repository and have these changes automatically deployed to production or production-like environments.
The benefit is the customer responsiveness this embodies: you can deploy new features or fixes to users in minutes, not days or weeks.
Pipeline Architecture
In the figure below, you see the high-level architecture for the deployment pipeline
Deployment Pipeline Architecture for ECS
CodePipeline is a Continuous Delivery service that enables you to orchestrate every step of your software delivery process in a workflow that consists of a series of stages and actions. These actions perform the steps of your software delivery process.
In CodePipeline, I’ve defined two stages: Source and Build. The Source stage retrieves code artifacts via a CodeCommit repository whenever someone commits a new change. This initiates the pipeline. CodePipeline is integrated with the Jenkins Continuous Integration server. The Build stage updates the ECS Docker image (which runs a small PHP web application) within ECR and makes the new application available through an ELB endpoint.
Jenkins is installed and configured on an Amazon EC2 instance within an Amazon Virtual Private Cloud (VPC). The CloudFormation template runs commands to install and configure the Jenkins server, install and configure Docker, install and configure the CodePipeline plugin and configure the job that’s run as part of the CodePipeline build action. The Jenkins job is configured to run a bash script that’s committed to the CodeCommit repository. This bash script updates the ECS service and task definition by running a Docker build, tag and push to the ECR repository. I describe the implementation of this architecture in more detail in this post.
Jenkins
In this example, CodePipeline manages the orchestration of the software delivery workflow. Since CodePipeline doesn’t actually execute the actions, you need to integrate it with an execution platform. To perform the execution of the actions, I’m using the Jenkins Continuous Integration server. I’ll configure a CodePipeline plugin for Jenkins so that Jenkins executes certain CodePipeline actions.
In particular, I have an action to update an ECS service. I do this by running a CloudFormation update on the stack. CloudFormation looks for any differences in the templates and applies those changes to the existing stack.
To orchestrate and execute this CloudFormation update, I configure a CodePipeline custom action that calls a Jenkins job. In this Jenkins job, I call a shell script passing several arguments.
Provision Jenkins in CloudFormation
In the CloudFormation template, I create an EC2 instance on which I will install and configure the Jenkins server. This CloudFormation script is based on the CodePipeline starter kit.
To launch a Jenkins server in CloudFormation, you will use the AWS::EC2::Instance resource. Before doing this, you’ll be creating an IAM role and an EC2 security group to the already provisioned VPC (the VPC provisioning is part of the CloudFormation script).
Within the Metadata attribute of the resource (i.e. the EC2 instance on which Jenkins will run), you use the AWS::CloudFormation::Init resource to define the user data configuration. To apply your changes, you call cfn-init to run commands on the EC2 instance like this:
"/opt/aws/bin/cfn-init -v -s ",
Then, you can install and configure Docker:
"# Install Docker\n", "cd /tmp/\n", "yum install -y docker\n",
On this same instance, you will install and configure the Jenkins server:
"# Install Jenkins\n", ... "yum install -y jenkins-1.658-1.1\n", "service jenkins start\n",
And, apply the dynamic Jenkins configuration for the job so that it updates the CloudFormation stack based on arguments passed to the shell script.
"/bin/sed -i \"s/MY_STACK/",
{
"Ref":"AWS::StackName"
},
"/g\" /tmp/config-template.xml\n",
In the config-template.xml, I added tokens that get replaced as part of the commands run from the CloudFormation template. You can see a snippet of this below in which the command for the Jenkins job makes a call to the configure-ecs.sh bash script with some tokenized parameters.
<command>bash ./configure-ecs.sh MY_STACK MY_ACCTID MY_ECR</command>
All of the commands for installing and configuring the Jenkins Server, Docker, the CodePipeline plugin and Jenkins jobs are described in the CloudFormation template that is hosted in the version-control repository.
Jenkins Job Configuration Template
In the previous code snippets from CloudFormation, you see that I’m using sed to update a file called config-template.xml. This is a Jenkins job configuration file for which I’m updating some token variables with dynamic information that gets passed to it from CloudFormation. This information is used to run a bash script to update the CloudFormation stack – which is described in the next section.
ECS Service Script to Update CloudFormation Stack
The code snippet below shows how the bash script captures that arguments that are passed by the Jenkins job into bash variables. Later in the script, it uses these bash variables to make a call the update-stack command in the CloudFormation API to apply a new ECS Docker image to the endpoint.
MY_STACK=$1 MY_ACCTID=$2 MY_ECR=$3 uuid=$(date +%s) awsacctid="$MY_ACCTID" ecr_repo="$MY_ECR" ecs_stack_name="$MY_STACK" ecs_template_url="$MY_URL"
In the code snippet below of the configure-ecs.sh script, I’m building, tagging and pushing to the Docker repository in my EC2 Container Registry repository using the dynamic values passed to this script from Jenkins (which were initially passed from the parameters and resources of my CloudFormation script).
In doing this, it creates a new Docker image for each commit and tags it with a unique id based on date and time. Finally, it uses the AWS CLI to call the update-stack command of the CloudFormation API using the variable information.
eval $(aws --region us-east-1 ecr get-login) # Build, Tag and Deploy Docker docker build -t $ecr_repo:$uuid . docker tag $ecr_repo:$uuid $awsacctid.dkr.ecr.us-east-1.amazonaws.com/$ecr_repo:$uuid docker push $awsacctid.dkr.ecr.us-east-1.amazonaws.com/$ecr_repo:$uuid aws cloudformation update-stack --stack-name $ecs_stack_name \ --template-url $ecs_template_url --region us-east-1 \ --capabilities="CAPABILITY_IAM" --parameters \ ParameterKey=AppName,UsePreviousValue=true \ ParameterKey=ECSRepoName,UsePreviousValue=true \ ParameterKey=DesiredCapacity,UsePreviousValue=true \ ParameterKey=KeyName,UsePreviousValue=true \ ParameterKey=RepositoryBranch,UsePreviousValue=true \ ParameterKey=RepositoryName,UsePreviousValue=true \ ParameterKey=InstanceType,UsePreviousValue=true \ ParameterKey=MaxSize,UsePreviousValue=true \ ParameterKey=S3ArtifactBucket,UsePreviousValue=true \ ParameterKey=S3ArtifactObject,UsePreviousValue=true \ ParameterKey=SSHLocation,UsePreviousValue=true \ ParameterKey=YourIP,UsePreviousValue=true \ ParameterKey=ImageTag,ParameterValue=$uuid
Now that you see the basics of install and configuring Jenkins in CloudFormation and what happens when the Jenkins is run through the CodePipeline orchestration, let’s look at the steps for configuring the CodePipeline part of the CodePipeline/Jenkins configuration.
Create a Pipeline using AWS CodePipeline
Before I create a working pipeline, I prefer to model the stages and actions in CodePipeline using Lambda so that I can think through the workflow. To do this I refer to my blog post on Mocking AWS CodePipeline pipelines with Lambda. I’m going to create a two-stage pipeline consisting of a Source and a Build stage. These stages and the actions in these stages are described in more detail below.
Define a Custom Action
There are five types of action categories in CodePipeline: Source, Build, Deploy, Invoke and Test. Each action has four attributes: category, owner, provider and version. There are 
three types of action owners: AWS, ThirdParty and Custom. AWS refers to built-in actions provided by AWS. Currently, there are four built-in action providers from AWS: S3, CodeCommit, CodeDeploy and ElasticBeanstalk. Examples of ThirdParty action providers include RunScope and GitHub. If none of the action providers suit your needs, you can define custom actions in CodePipeline. In my case, I wanted to run a script from a Jenkins job so I used the CloudFormation sample configuration from the CodePipeline starter kit for the configuration of the custom build action that I use to integrate Jenkins with CodePipeline. See the snippet below.
"CustomJenkinsActionType":{
"Type":"AWS::CodePipeline::CustomActionType",
"DependsOn":"JenkinsHostWaitCondition",
"Properties":{
"Category":"Build",
"Provider":{
"Fn::Join":[
"",
[
{
"Ref":"AppName"
},
"-Jenkins"
]
]
},
"Version":"1",
"ConfigurationProperties":[
{
"Key":"true",
"Name":"ProjectName",
"Queryable":"true",
"Required":"true",
"Secret":"false",
"Type":"String"
}
],
"InputArtifactDetails":{
"MaximumCount":5,
"MinimumCount":0
},
"OutputArtifactDetails":{
"MaximumCount":5,
"MinimumCount":0
},
"Settings":{
"EntityUrlTemplate":{
"Fn::Join":[
"",
[
"http://",
{
"Fn::GetAtt":[
"JenkinsServer",
"PublicIp"
]
},
"/job/{Config:ProjectName}"
]
]
},
"ExecutionUrlTemplate":{
"Fn::Join":[
"",
[
"http://",
{
"Fn::GetAtt":[
"JenkinsServer",
"PublicIp"
]
},
"/job/{Config:ProjectName}/{ExternalExecutionId}"
]
]
}
}
}
},
The example pipeline that I’ve defined in CodePipeline (and described as code in CloudFormation) uses the above custom action in the Build stage of the pipeline, which is described in more detail in the Build Stage section later.
Source Stage
The Source stage has a single action to look for any changes to a CodeCommit repository. If it discovers any new commits, it retrieves the the artifacts from the CodeCommit and stores them in an encrypted form in an S3 bucket. If it’s successful, it transitions to the next stage: Build. A snippet from the CodePipeline resource definition for the Source stage in CloudFormation is shown below.
"Stages":[
{
"Name":"Source",
"Actions":[
{
"InputArtifacts":[
],
"Name":"Source",
"ActionTypeId":{
"Category":"Source",
"Owner":"AWS",
"Version":"1",
"Provider":"CodeCommit"
},
"OutputArtifacts":[
{
"Name":{
"Fn::Join":[
"",
[
{
"Ref":"AWS::StackName"
},
"-SourceArtifact"
]
]
}
}
],
"Configuration":{
"BranchName":{
"Ref":"RepositoryBranch"
},
"RepositoryName":{
"Ref":"RepositoryName"
}
},
"RunOrder":1
}
]
},
Build Stage
The Build stage invokes actions to create a new ECS repository if one doesn’t exist, builds and tags a Docker image and makes a call to a CloudFormation template to launch the rest of the ECS environment – including creating an ECS cluster, task definition, ECS services, ELB, Security Groups and IAM resources. It does this using the custom CodePipeline action for Jenkins that I described earlier. A snippet from the CodePipeline resource definition in CloudFormation for the Build stage is shown below.
{
"Name":"Build",
"Actions":[
{
"Name":"DeployPHPApp",
"InputArtifacts":[
{
"Name":{
"Fn::Join":[
"",
[
{
"Ref":"AWS::StackName"
},
"-SourceArtifact"
]
]
}
}
],
"ActionTypeId":{
"Category":"Build",
"Owner":"Custom",
"Version":"1",
"Provider":{
"Fn::Join":[
"",
[
{
"Ref":"AWS::StackName"
},
"-Jenkins"
]
]
}
},
"OutputArtifacts":[
{
"Name":{
"Fn::Join":[
"",
[
{
"Ref":"AWS::StackName"
},
"-BuiltArtifact"
]
]
}
}
],
"Configuration":{
"ProjectName":{
"Ref":"AWS::StackName"
}
},
"RunOrder":1
}
]
}
The custom action for Jenkins (via the CodePipeline plugin) is looking for work from CodePipeline. When it finds work, it performs the task associated with the CodePipeline action. In this case, it runs the Jenkins job that calls the configure-ecs.sh script. This bash script makes a update-stack call to the original CloudFormation template passing in the new image via the ImageTag parameter which is the new tag generated for the Docker image created as part of this script.
CloudFormation seeks to run the minimum necessary changes to the infrastructure based on the stack update. In this case, I’m only providing a new image tag but this results in creating a new ECS task definition for the service. In your CloudFormation events console, you’ll see a message similar to the one below:
AWS::ECS::TaskDefinition Requested update requires the creation of a new physical resource; hence creating one.
As I mentioned in part 1 of this series, I defined a DeploymentConfiguration type with a MinimumHealthyPercent property of 0 since I’m only using one EC2 instance as running through the earlier stages of the pipeline. This means the application experiences a few seconds of downtime during the update. Like most applications/services these days, if I need to continual uptime, I’d increase the number of instances in my Auto Scaling Group and increase the MinimumHealthyPercent property.
Other Stages
In the example I provided, I stop at the Build stage. If you were to take this to production, you might include other stages as well. Perhaps you might have a “Staging” stage in which you might include actions to deploy the application to the ECS containers using a production-like configuration which might include more instances in the Auto Scaling Group.
Once Staging is complete, the pipeline would automatically transition to the Production stage where it might make Lambda calls to test the application running in ECS containers. If everything looks ok, it switches the Route 53 hosted zone endpoint to the new container.
Launch the ECS Stack and Pipeline
In this section, you’ll launch the CloudFormation stack that creates the ECS and Pipeline resources.
Prerequisites
You need to have already created an ECR repository and a CodeCommit repository to successfully launch this stack. For instructions on creating an ECR repository, see part 1 of this series (or to directly launch the CloudFormation stack to create this ECR repository, click this button: 
.) For creating a CodeCommit repository, you can either see part 1 or use the instructions described at: Create and Connect to an AWS CodeCommit Repository.
Launch the Stack
Click the button below to launch a CloudFormation stack that provisions the ECS environment including all the resources previously described such as CodePipeline, ECS Cluster, ECS Task Definition, ECS Service, ELB, VPC resources, IAM Roles, etc.
You’ll enter values for the following parameters: RepositoryName, YourIP, KeyName, and ECSRepoName.
To launch the same stack from your AWS CLI, type the following (while modifying the same parameter values described above):
aws cloudformation create-stack --stack-name ecs-stack-1648 --template-url https://s3.amazonaws.com/stelligent-training-public/public/codepipeline/ecs-pipeline.json --region us-east-1 --disable-rollback --capabilities="CAPABILITY_IAM" --parameters ParameterKey=RepositoryName,ParameterValue=YOURCCREPO ParameterKey=RepositoryBranch,ParameterValue=master ParameterKey=KeyName,ParameterValue=YOUREC2KEYPAIR ParameterKey=YourIP,ParameterValue=YOURIP/32 ParameterKey=ECSRepoName,ParameterValue=YOURECRREPO ParameterKey=ECSCFNURL,ParameterValue=NOURL ParameterKey=AppName,ParameterValue=app-name-1648
Outputs
Once the CloudFormation stack successfully launches, there are several outputs but the two most relevant are AppURL and CodePipelineURL. You can click on the AppURL value to launch the PHP application running on ECS from the ELB endpoint. The CodePipelineURL output value launches the generated pipeline from the CodePipeline console. See the screenshot below.
Access the Application
Once the stack successfully completes, go to the Outputs tab for the CloudFormation stack and click on the AppURL value to launch the application.
Commit Changes to CodeCommit
Make some visual changes to the code and commit these changes to your CodeCommit repository to see these changes get deployed through your pipeline. You perform these actions from the directory where you cloned a local version of your CodeCommit repo (in the directory created by your git clone command). Some example command-line operations are shown below.
git commit -am "change color to pink" git push
Once these changes have been committed, CodePipeline will discover the changes made to your CodeCommit repo and initiate a new pipeline. After the pipeline is successfully completed, follow the same instructions for launching the application from your browser.
Making Modifications
While the solution can work “straight out of the box”, if you’d like to make some changes, I’ve included a few sections of the code that you’ll need to modify.
configure-ecs.sh
The purpose of the configure-ecs.sh Bash script is to run the Docker commands to build, tag and push the image along with updating the existing CloudFormation stack to update the ECS service and task. The source for this bash script is here: https://github.com/stelligent/cloudformation_templates/blob/master/labs/ecs/configure-ecs.sh. I hard coded the ecs_template_url variable to a specific S3 location. You can either download the source file from one of these two locations: GitHub or S3 to make your desired modifications and then modify the ecs_template_url variable to the new location (presumably in S3).
config-template.xml
The purpose of the config-template.xml file is the Jenkins job configuration for the update ECS action. This XML file contains tokens that get replaced from the ecs-pipeline.json CloudFormation template with dynamic information like the CloudFormation stack name, account id, etc. This XML file is obtained via a wget command from within the template. The file is stored in S3 at https://s3.amazonaws.com/stelligent-training-public/public/jenkins/config-template.xml so you can modify the S3 location to your account while updating the CloudFormation template to point to the new location. In doing this, you can modify any of the behavior of the updates to the file when used by Jenkins.
Summary
In this series, you learned how to use CloudFormation to fully automate the provisioning of the Elastic Container Service along with a CodePipeline pipeline that uses CodeCommit as its version-control repository so that whenever a change is made to the Git repo, the changes are automatically applied to a PHP application hosted on ECS images.
By modeling your pipeline in CodePipeline you can apply even more stages and actions as part of your Continuous Delivery process so that it runs through all the tests and other checks enabling you to deliver changes to the production whenever there’s a business need to do so.
Sample Code
The code for the examples demonstrated in this post are located at https://github.com/stelligent/cloudformation_templates/tree/master/labs/ecs. Let us know if you have any comments or questions @stelligent or @paulduvall.
Stelligent is hiring! Do you enjoy working on complex problems like figuring out ways to automate all the things as part of a deployment pipeline? Do you believe in the “everything-as-code” mantra? If your skills and interests lie at the intersection of DevOps automation and the AWS cloud, check out the careers page on our website.
Notes
The sample solution currently only works in the us-east-1 AWS region. You will be charged for your AWS usage – including EC2, S3, CodePipeline and other services.
Resources
Here’s a list of some of the resources described or were influenced in this post:
- Explore Continuous Delivery in AWS with the Pipeline Starter Kit
- Getting Started with Amazon ECS
- Set up a build pipeline with Jenkins and Amazon ECS
- Docker Basics – Instructions for installing Docker on Amazon Linux including uploading to the ECS Repository (ECR)
- ECS Repositories in US-EAST-1
- Deleting an ECS Image in ECR
- CloudFormation resources covered in this post
|
Stelligent Amazon Pollycast
 |