Testing Nix Packages in Docker

In this blog post, we will cover developing nix packages and testing them in docker.  We will set up a container with the proper environment to build nix packages, and then we will test build existing packages from the nixpkgs repo.  This lays the foundation for using docker to test your own nix packages.

First, a quick introduction to Nix. Nix is a purely functional package manager. Nix expressions, a simple functional language, declaratively define each package.  A Nix expression describes everything that goes into a package build action, known as a derivation.  Because it’s a functional language, it’s easy to support building variants of a package: turn the Nix expression into a function and call it any number of times with the appropriate arguments. Due to the hashing scheme, variants don’t conflict with each other in the Nix store.  More details regarding Nix packages and NixOS can be found here.

To begin, we must write the Dockerfile that will be used to build the development environment.  We need to declare the baseimage to be used (centos7), install a few dependencies in order to install nix and clone nixpkgs repo.  We also need to set up the nix user and permissions:

FROM centos:latest

MAINTAINER “Fernando J Pando” <nando@********.com>

RUN yum -y install bzip2 perl-Digest-SHA git

RUN adduser nixuser && groupadd nixbld && usermod -aG nixbld nixuser

RUN mkdir -m 0755 /nix && chown nixuser /nix

USER nixuser

WORKDIR /home/nixuser

We clone the nixpkgs github repo (this can be set to any fork/branch for testing):

RUN git clone https://github.com/nixos/nixpkgs.git

We download the nix installer (latest stable 1.11.4):

RUN curl https://nixos.org/nix/install -o nix.install.sh

We are now ready to set up the environment that will allow us to build nix packages in this container.  There are several environment variables that need to be set for this to work, and we can pull them from the environment script provided by nix installation (~/.nix-profile/etc/profile.d/nix.sh).  For easy reference, here is that file:

# Set the default profile.
if ! [ -L &amp;amp;quot;$NIX_LINK&amp;amp;quot; ]; then
echo &amp;amp;quot;creating $NIX_LINK&amp;amp;quot; &amp;amp;gt;&amp;amp;amp;2
/nix/store/xmlp6pyxi6hi3vazw9821nlhhiap6z63-coreutils-8.24/bin/ln -s &amp;amp;quot;$_NIX_DEF_LINK&amp;amp;quot; &amp;amp;quot;$NIX_LINK&amp;amp;quot;

export PATH=$NIX_LINK/bin:$NIX_LINK/sbin:$PATH

# Subscribe the user to the Nixpkgs channel by default.
if [ ! -e $HOME/.nix-channels ]; then
echo &amp;amp;quot;https://nixos.org/channels/nixpkgs-unstable nixpkgs&amp;amp;quot; &amp;amp;gt; $HOME/.nix-channels

# Append ~/.nix-defexpr/channels/nixpkgs to $NIX_PATH so that
# &amp;amp;lt;nixpkgs&amp;amp;gt; paths work when the user has fetched the Nixpkgs
# channel.
export NIX_PATH=${NIX_PATH:+$NIX_PATH:}nixpkgs=$HOME/.nix-defexpr/channels/nixpkgs

# Set $SSL_CERT_FILE so that Nixpkgs applications like curl work.
if [ -e /etc/ssl/certs/ca-certificates.crt ]; then # NixOS, Ubuntu, Debian, Gentoo, Arch
export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
elif [ -e /etc/ssl/certs/ca-bundle.crt ]; then # Old NixOS
export SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt
elif [ -e /etc/pki/tls/certs/ca-bundle.crt ]; then # Fedora, CentOS
export SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt
elif [ -e &amp;amp;quot;$NIX_LINK/etc/ssl/certs/ca-bundle.crt&amp;amp;quot; ]; then # fall back to cacert in Nix profile
export SSL_CERT_FILE=&amp;amp;quot;$NIX_LINK/etc/ssl/certs/ca-bundle.crt&amp;amp;quot;
elif [ -e &amp;amp;quot;$NIX_LINK/etc/ca-bundle.crt&amp;amp;quot; ]; then # old cacert in Nix profile
export SSL_CERT_FILE=&amp;amp;quot;$NIX_LINK/etc/ca-bundle.crt&amp;amp;quot;

We pull out the relevant environment given a Docker base Centos7 image:

ENV USER=nixuser

ENV HOME=/home/nixuser

ENV _NIX_DEF_LINK=/nix/var/nix/profiles/default

ENV SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt

ENV NIX_LINK=$HOME/.nix-profile


ENV NIX_PATH=${NIX_PATH:+$NIX_PATH:}nixpkgs=$HOME/.nix-defexpr/channels/nixpkgs

RUN echo “https://nixos.org/channels/nixpkgs-unstable nixpkgs” > $HOME/.nix-channels


Now that the environment has been set up, we are ready to install nix:

RUN bash /home/nixuser/nix.install.sh

The last step is to set the working directory to the cloned github nixpkgs directory, so testing will execute from there when the container is run:

WORKDIR /home/nixuser/nixpkgs

At this point, we are ready to build the container:

docker build . -t nand0p/nixpkgs-devel

Conversely, you can pull this container image from docker hub:

docker pull nand0p/nixpkgs-devel

(NOTE: The docker hub image will contain a version of the nixpkgs repo from the time the container image was built.   If you are testing on the bleeding edge, always build the container fresh before testing.)

Once the container is ready, we can now begin test building an existing nix package:

docker run -ti nand0p/nixpkgs-devel nix-build -A nginx

The above command will test build the nginx package inside your new docker container. It will also build all the dependencies.  In order to test the package interactively to ensure to resulting binary works as expected, the container can be launched with bash:

docker run -ti nand0p/nixpkgs-devel bash
[nixuser@aa2c8e29c5e9 nixpkgs]$

Once you have shell inside the container, you can build and test run the package:

[nixuser@aa2c8e29c5e9 nixpkgs]$ nix-build -A nginx

[nixuser@aa2c8e29c5e9 nixpkgs]$ /nix/store/7axviwfzbsqy50zznfxb7jzfvmg9pmwx-nginx-1.10.1/bin/nginx -V
nginx version: nginx/1.10.1
built by gcc 5.4.0 (GCC)
built with OpenSSL 1.0.2h 3 May 2016
TLS SNI support enabled
configure arguments: –prefix=/nix/store/7axviwfzbsqy50zznfxb7jzfvmg9pmwx-nginx-1.10.1 –with-http_ssl_module –with-http_v2_module –with-http_realip_module –with-http_addition_module –with-http_xslt_module –with-http_image_filter_module –with-http_geoip_module –with-http_sub_module –with-http_dav_module –with-http_flv_module –with-http_mp4_module –with-http_gunzip_module –with-http_gzip_static_module –with-http_auth_request_module –with-http_random_index_module –with-http_secure_link_module –with-http_degradation_module –with-http_stub_status_module –with-ipv6 –with-file-aio –add-module=/nix/store/vbfb8z3hgymbaz59wa54qf33yl84jii7-nginx-rtmp-module-v1.1.7-src –add-module=/nix/store/ps5s8q9v91l03972gw0h4l9iazx062km-nginx-dav-ext-module-v0.0.3-src –add-module=/nix/store/46sjsnbkx7rgwgn3pgigcfaygrs2cx30-headers-more-nginx-module-v0.26-src –with-cc-opt=’-fPIE -fstack-protector-all –param ssp-buffer-size=4 -O2 -D_FORTIFY_SOURCE=2′ –with-ld-opt=’-pie -Wl,-z,relro,-z,now’

Thanks for reading, and stay tuned for part II, where we will modify a nix package for our custom needs, and test it against our own fork of the nixpkgs repo.

Stelligent Bookclub: “Building Microservices” by Sam Newman

At Stelligent, we put a strong focus on education and so I wanted to share some books that have been popular within our team. Today we explore the world of microservices with “Building Microservices” by Sam Newman.

Microservices are an approach to distributed systems that promotes the use of small independent services within a software solution. By adopting microservices, teams can achieve better scaling and gain autonomy, that allows teams to chose their technologies and iterate independently from other teams.

As a result, a change to one part of the system could unintentionally break a different part, which in turn might lead to hard-to-predict outages

Microservices are an alternative to the development of a monolithic codebase in many organizations – a codebase that contains your entire application and where new code piles on at alarming rates. Monoliths become difficult to work with as interdependencies within the code begin to develop.

As a result, a change to one part of the system could unintentionally break a different part, which in turn might lead to hard-to-predict outages. This is where Newman’s argument about the benefits of microservices really comes into play.

  • Reasons to split the monolith
    • Increase pace of change
    • Security
    • Smaller team structure
    • Adopt the proper technology for a problem
    • Remove tangled dependencies
    • Remove dependency on databases for integration
    • Less technical debt

By splitting monoliths at their seams, we can slowly transform a monolithic codebase into a group of microservices. Each service his loosely coupled and highly cohesive, as a result changes within a microservice do not change it’s function to other parts of the system. Each element works in a blackbox where only the inputs and outputs matter. When splitting a monolith, databases pose some of the greatest challenge; as a result, Newman devotes a significant chunk of the text/book to explaining various useful techniques to reduce these dependencies.

Ways to reduce dependencies

  • Clear well documented api
  • Loose coupling and high cohesion within a microservice
  • Enforce standards on how services can interact with each other

Though Newman’s argument for the adoption of microservices is spot-on, his explanation on continuous delivery and scaling micro-services is shallow. For anyone who has a background in CD or has read “Continuous Delivery” these sections do not deliver. For example, he takes the time to talk about machine images at great length but lightly brushes over build pipelines. The issue I ran into with scaling microservices is Newman suggests that ideally each microservice should ideally be put on its own instance where it exists independently of all other services. Though this is a possibility and it would be nice to have this would be highly unlikely to happen in a production environment where cost is a consideration. Though he does talk about using traditional virtualization, Vagrant, linux containers, and Docker to host multiple services on a single host he remains platform agnostic and general. As a result he misses out on the opportunity to talk about services like Amazon ECS, Kubernetes, or Docker Swarm. Combining these technologies with reserved cloud capacity would be a real world example that I feel would have added a lot to this section

Overall Newman’s presentation of microservices is a comprehensive introduction for IT professionals. Some of the concepts covered are basic but there are many nuggets of insight that are worth reading for. If you are looking to get a good idea about how microservices work, pick it up. If you’re looking to advance your microservice patterns or suggest some, feel free to comment below!

Interested in working someplace that gives all employees an impressive book expense budget? We’re hiring.

Beyond Continuous Deployment

If I were to whittle the principle behind all modern software development approaches into one word, that word would be: feedback. By “modern approaches”, I’m referring to DevOps, Continuous Integration (CI), Continuous Delivery (CD), Continuous Deployment, Microservices, and so on. For definitions of these terms, see the Stelligent Glossary.

It’s not just feedback: it’s fast and effective feedback. It’s incorporating that feedback into subsequent behavior. It’s amplifying this feedback back into the development process to affect future work as soon as possible.

In this post, I describe how we can move beyond continuous deployment by focusing on the principle of feedback.


I think it’s important to define all of this as one word and one principle because it’s so easy to get lost in the weeds of tools, processes, patterns, and practices. When I’m presented with a new concept, a problem, or an approach in my work, I often ask myself: “How will this affect feedback? Will it increase fast, effective feedback or decrease it?” These questions often help guide my decision making.

Amazon Web Services (AWS) has a great talk on how Amazon embraced DevOps in their organization (well before it was called “DevOps”). As part of the talk, they show an illustration similar to the one you see below in which they describe the feedback loop between developers and customers.

Deployment Pipeline Feedback Loop (Based on: https://aws.amazon.com/devops/what-is-devops/)

They go on to describe two key points with this feedback loop:

  1. The faster you’re able to get through the feedback loop determines your customer responsiveness and your ability to innovate.
  2. In the eyes of your customers, you’re only delivering value when you’re spending time on the left side – developing new features.

The key, as AWS describes, is that any time you spend on building the pipeline itself or hand-holding changes through this pipeline, you’re not delivering value – at least in the eyes of your customer. So, you want to maximize the time you’re spending on the left side (developing new features) and minimize the time you’re spending in the middle – while delivering high-quality software that meets the needs of your customers.

They go on to describe DevOps as anything that helps increase these feedback loops, which might include changes to the organization, process, tooling or culture.

The Vision

There’s a vision on feedback that I’ve discussed with a few people and only recently realized that I hadn’t shared with the wider software community. In many ways, I still feel like it’s “Day 1” when it comes to software delivery. As mentioned, there’s been the introduction of some awesome tools, approaches, and practices in the past few years like Cloud, Continuous Delivery, Microservices, and Serverless but we’ll be considering all of this ancient times several years from now.

Martin Fowler is fond of saying that software integration should be a “non event”. I wholeheartedly agree but the reality is that even in the best CI/CD environments, there are still lots of events in the form of interruptions and wait time associated with the less creative side of software development (i.e. the delivery of the software to users).

The vision I describe below is inspired by an event on Continuous Integration that I attended in 2008 that Andy Glover describes on The Disco Blog. I’m still working on the precise language of this vision, but by focusing on fast and effective feedback, it led me to what I describe here:

Beyond Continuous Deployment

Using smart algorithms, code is automatically integrated and pushed to production in nanoseconds as a developer continues to work as long as it passes myriad validation and verification checks in the deployment pipeline. The developer is notified of success or failure in nanoseconds passively through their work environment.

I’m sure there are some physics majors who might not share my “nanoseconds” view on this, but sometimes approaching problems devoid of present day limitations can lead to better future outcomes. Of course, I don’t think people will complain if it’s “seconds” instead of “nanoseconds” as it moves closer toward the vision.

This vision goes well beyond the idea of today’s notion of “Continuous Deployment” which relies on developers to commit code to a version-control repository according to the individual developer’s idiosyncrasies and schedule. In this case, smart algorithms would determine when and how often code is “committed” to a version-control repository and, moreover, these same algorithms are responsible for orchestrating it into the pipeline on its way to production.

These smart algorithms could be applied when a code block is complete or some other logical heuristic. It’d likely use some type of machine learning algorithm to determine these logical intervals, but it might equate to hundreds of what we call “commits” per developer, per day. As a developer, you’d continue writing code as these smart algorithms automatically determine the best time to integrate your changes with the rest of the code base. In your work environment, you might see some passive indicators of success or failure as you continue writing code (e.g. color changes and/or other passive notifiers). The difference is that your work environment is informing you not just based on some simple compilation, but based upon the full creation and verification of the system as part of a pipeline – resulting in an ultra fast and effective feedback cycle.

The “developer’s work environment” that I describe in the vision could be anything from what we think of as an Integrated Development Environment (IDE) to a code editor, to a developer’s environment managed in the cloud. It doesn’t really matter because the pipeline runs based on the canonical source repository as defined and integrated through the smart algorithms and orchestrated through a canonical pipeline.

Some deployment pipelines today effectively use parallel actions to increase the throughput of the system change. But, even in the most effective pipelines, there’s still a somewhat linear process in which some set of actions relies upon preceding actions to succeed in order to initiate its downstream action(s). The pipelines that would enable this vision would need to rethink the approach to parallelization in order to provide feedback as fast I’m suggesting.

This approach will also likely require more granular microservices architectures as a means of decreasing the time it takes to provide fast and effective feedback.

In this vision, you’d continue to separate releases from deployments whereas deployments will regularly occur in this cycle, but releases would be associated with more business-related concerns. For example, you might have thousands of deployments for a single application/service in a week, but maybe only a single release during that same time.

If a deployment were to fail, it wouldn’t deploy the failure to production. It only deploys to production if it passes all of the defined checks that are partially built from machine learning and other autonomic system techniques.


By focusing on the principle of feedback, you can eliminate a lot of the “noise” when it comes to making effective decisions on behalf of your customers and your teams. Your teams need fast and effective feedback to be more responsive to customers. I shared how you can often arise at better decisions by focusing on principles over practices. Finally, this vision goes well beyond today’s notion of Continuous Deployment to enable even more effective engineer and customer responsiveness.


Stelligent is hiring! Do you enjoy working on complex problems like figuring out ways to automate all the things as part of a deployment pipeline? Do you believe in the “everything-as-code” mantra? If your skills and interests lie at the intersection of DevOps automation and the AWS cloud, check out the careers page on our website.

Automating and Orchestrating OpsWorks in CloudFormation and CodePipeline

pipeline_opsworks_consoleIn this post, you’ll learn how to provision, configure, and orchestrate a PHP application using the AWS OpsWorks application management service into a deployment pipeline using AWS CodePipeline that’s capable of deploying new infrastructure and code changes when developers commit changes to the AWS CodeCommit version-control repository. This way, team members can release new changes to users whenever they choose to do so: aka, Continuous Delivery.

Recently, AWS announced the integration of OpsWorks into AWS CodePipeline so I’ll be describing various components and services that support this solution including CodePipeline along with codifying the entire infrastructure in AWS CloudFormation. As part of the announcement, AWS provided a step-by-step tutorial of integrating OpsWorks with CodePipeline that I used as a reference in automating the entire infrastructure and workflow.

This post describes how to automate all the steps using CloudFormation so that you can click on a Launch Stack button to instantiate all of your infrastructure resources.


“AWS OpsWorks is a configuration management service that helps you configure and operate applications of all shapes and sizes using Chef. You can define the application’s architecture and the specification of each component including package installation, software configuration and resources such as storage. Start from templates for common technologies like application servers and databases or build your own to perform any task that can be scripted. AWS OpsWorks includes automation to scale your application based on time or load and dynamic configuration to orchestrate changes as your environment scales.” [1]

OpsWorks provides a structured way to automate the operations of your AWS infrastructure and deployments with lifecycle events and the Chef configuration management tool. OpsWorks provides more flexibility than Elastic Beanstalk and more structure and constraints than CloudFormation. There are several key constructs that compose OpsWorks. They are:

  • Stack – An OpsWorks stack is the logical container defining OpsWorks layers, instances, apps and deployments.
  • Layer – There are built-in layers provided by OpsWorks such as Static Web Servers, Rails, Node.js, etc. But, you can also define your own custom layers as well.
  • Instances – These are EC2 instances on which the OpsWorks agent has been installed. There are only certain Linux and Windows operating systems supported by OpsWorks instances.
  • App – “Each application is represented by an app, which specifies the application type and contains the information that is needed to deploy the application from the repository to your instances.” [2]
  • Deployment – Runs Chef recipes to deploy the application onto instances based on the defined layer in the stack.

There are also lifecycle events that get executed for each deployment. Lifecycle events are linked to one or more Chef recipes. The five lifecycle events are setup, configure, deploy, undeploy, shutdown. Events get triggered based upon certain conditions. Some events can be triggered multiple times. They are described in more detail below:

  • setup – When an instance finishes booting as part of the initial setup
  • configure – When this event is run, it executes on all instances in all layers whenever a new instance comes in service, or an EIP changes, or an ELB is attached
  • deploy – When running a deployment on an instance, this event is run
  • undeploy – When an app gets deleted, this event is run
  • shutdown – Before an instance is terminated, this event is run

Solution Architecture and Components

In Figure 2, you see the deployment pipeline and infrastructure architecture for the OpsWorks/CodePipeline integration.

Figure 2 – Deployment Pipeline Architecture for OpsWorks

Both OpsWorks and CodePipeline are defined in a single CloudFormation stack, which is described in more detail later in this post. Here are the key services and tools that make up the solution:

  • OpsWorks – In this stack, code configures operations of your infrastructure using lifecycle events and Chef
  • CodePipeline – Orchestrate all actions in your software delivery process. In this solution, I provision a CodePipeline pipeline with two stages and one action per stage in CloudFormation
  • CloudFormation – Automates the provisioning of all AWS resources. In this solution, I’m using CloudFormation to automate the provisioning for OpsWorks, CodePipeline,  IAM, and S3
  • CodeCommit – A Git repo used to host the sample application code from this solution
  • PHP – In this solution, I leverage AWS’ OpsWorks sample application written in PHP.
  • IAM – The CloudFormation stack defines an IAM Instance Profile and Roles for controlled access to AWS resources
  • EC2 – A single compute instance is launched as part of the configuration of the OpsWorks stack
  • S3 – Hosts the deployment artifacts used by CodePipeline.

Create and Connect to a CodeCommit Repository

While you can store your software code in any version-control repository, in this solution, I’ll be using the AWS CodeCommit Git repository. I’ll be integrating CodeCommit with CodePipeline. I’m basing the code off of the Amazon OpsWorks PHP Simple Demo App located at https://github.com/awslabs/opsworks-demo-php-simple-app.

To create your own CodeCommit repo, follow these instructions: Create and Connect to an AWS CodeCommit Repository. I called my CodeCommit repository opsworks-php-demo. You can call it the same but if you do name it something different, be sure to replace the samples with your repo name.

After you create your CodeCommit repo, copy the contents from the AWS PHP OpsWorks Demo app and commit all of the files.


I created this sample solution by stitching together several available resources including the CloudFormation template provided by the Step-by-Step Tutorial from AWS on integrating OpsWorks with CodePipeline and existing templates we use at Stelligent for CodePipeline. Finally, I manually created the pipeline in CodePipeline using the same step-by-step tutorial and then obtained the configuration of the pipeline using the get-pipeline command as shown in the command snippet below.

aws codepipeline get-pipeline --name OpsWorksPipeline > pipeline.json

This section describes the various resources of the CloudFormation solution in greater detail including IAM Instance Profiles and Roles, the OpsWorks resources, and CodePipeline.

Security Group

Here, you see the CloudFormation definition for the security group that the OpsWorks instance uses. The definition restricts the ingress port to 80 so that only web traffic is accepted on the instance.

        "GroupDescription":"Lets you manage OpsWorks instances deployed to by CodePipeline"

IAM Role

Here, you see the CloudFormation definition for the OpsWorks instance role. In the same CloudFormation template, there’s a definition for an IAM service role and an instance profile. The instance profile refers to OpsWorksInstanceRole defined in the snippet below.

The roles, policies and profiles restrict the service and resources to the essential permissions it needs to perform its functions.



The snippet below shows the CloudFormation definition for the OpsWorks Stack. It makes references to the IAM service role and instance profile, using Chef 11.10 for its configuration, and using Amazon Linux 2016.03 for its operating system. This stack is used as the basis for defining the layer, app, instance, and deployment that are described later in this section.

        "DefaultOs":"Amazon Linux 2016.03",


The OpsWorks PHP layer is described in the CloudFormation definition below. It references the OpsWorks stack that was previously created in the same template. It also uses the php-app layer type. For a list of valid types, see CreateLayer in the AWS API documentation. This resource also enables auto healing, assigns public IPs and references the previously-created security group.


OpsWorks Instance

In the snippet below, you see the CloudFormation definition for the OpsWorks instance. It references the OpsWorks layer and stack that are created in the same template. It defines the instance type as c3.large and refers to the EC2 Key Pair that you will provide as an input parameter when launching the stack.


OpsWorks App

In the snippet below, you see the CloudFormation definition for the OpsWorks app. It refers to the previously created OpsWorks stack and uses the current stack name for the app name – making it unique. In the OpsWorks type, I’m using php. For other supported types, see CreateApp.

I’m using other for the AppSource type (OpsWorks doesn’t seem to make the documentation obvious in terms of the types that AppSource supports, so I resorted to using the OpsWorks console to determine the possibilities). I’m using other because my source type is CodeCommit, which isn’t currently an option in OpsWorks.



In the snippet below, you see the CodePipeline definition for the Deploy stage and the DeployPHPApp action in CloudFormation. It takes MyApp as an Input Artifact – which is an Output Artifact of the Source stage and action that obtains code assets from CodeCommit.

The action uses a Deploy category and OpsWorks as the Provider. It takes four inputs for the configuration: StackId, AppId, DeploymentType, LayerId. With the exception of DeploymentType, these values are obtained as references from previously created AWS resources in this CloudFormation template.

For more information, see CodePipeline Concepts.



Launch the Stack

Click the button below to launch a CloudFormation stack that provisions the OpsWorks environment including all the resources previously described such as CodePipeline, OpsWorks, IAM Roles, etc.

When launching a stack, you’ll enter a value the KeyName parameter from the drop down. Optionally, you can enter values for your CodeCommit repository name and branch if they are different than the default values.

Figure 3- Parameters for Launching the CloudFormation Stack

You will charged for your AWS usage – particularly EC2, CodePipeline and S3.

To launch the same stack from your AWS CLI, type the following (while modifying the same parameter values described above):

aws cloudformation create-stack --stack-name OpsWorksPipelineStack --template-url https://s3.amazonaws.com/stelligent-training-public/public/codepipeline/codepipeline-opsworks.json --region us-east-1 --disable-rollback --capabilities="CAPABILITY_IAM" --parameters  ParameterKey=KeyName,ParameterValue=YOURKEYNAME


Once the CloudFormation stack successfully launches, there’s an output for the CodePipelineURL. You can click on this value to launch the pipeline that’s running that’s getting the source assets from CodeCommit and launch an OpsWorks stack and associated resources. See the screenshot below.

Figure 4 – CloudFormation Outputs for CodePipeline/OpsWorks stack

Once the pipeline is complete, you can access the OpsWorks stack and click on the Public IP link for one of the instances to launch the PHP application that was deployed using OpsWorks as shown in Figures 5 and 6 below.

Figure 5 – Public IP for the OpsWorks instance


Figure 6 – OpsWorks PHP app once initially deployed

Commit Changes to CodeCommit

Make some visual changes to the code (e.g. your local CodeCommit version of index.php) and commit these changes to your CodeCommit repository to see these software get deployed through your pipeline. You perform these actions from the directory where you cloned a local version of your CodeCommit repo (in the directory created by your git clone command). Some example command-line operations are shown below.

git commit -am "change color to rust orange"
git push

Once these changes have been committed, CodePipeline will discover the changes made to your CodeCommit repo and initiate a new pipeline. After the pipeline is successfully completed, follow the same instructions for launching the application from your browser – as shown in Figure 7.

Figure 7 – Application after code changes committed to CodeCommit, orchestrated by CodePipeline and deployed by OpsWorks

Sample Code

The code for the examples demonstrated in this post are located at https://github.com/stelligent/cloudformation_templates/tree/master/labs/opsworks. Let us know if you have any comments or questions @stelligent or @paulduvall.

Stelligent is hiring! Do you enjoy working on complex problems like figuring out ways to automate all the things as part of a deployment pipeline? Do you believe in the “everything-as-code” mantra? If your skills and interests lie at the intersection of DevOps automation and the AWS cloud, check out the careers page on our website.

Useful Resources and References

OpsWorks Reference

Below, I’ve documented some additional information that might be useful on the OpsWorks service itself including its available integrations, supported versions and features.

  • OpsWorks supports three application source types: GitHub, S3, and HTTP.
  • You can store up to five versions of an OpsWorks application: the current revision plus four more for rollbacks.
  • When using the create-deployment method, you can target the OpsWorks stack, app, or instance
  • OpsWorks require internet access for the OpsWorks endpoint instance
  • Chef supports Windows in version 12
  • You cannot mix Windows and Linux instances in an OpsWorks stack
  • To change the default OS in OpsWorks, you need to change the OS and reprovision the instances
  • You cannot change the VPC for an OpsWorks instance
  • You can add ELB, EIPs, Volumes and RDS to an OpsWorks stack
  • OpsWorks autoheals at the layer level
  • You can assign multiple Chef recipes to an OpsWorks layer event
  • The three instance types in OpsWorks are: 24/7, time-based, load-based
  • To initiate a rollback in OpsWorks, you use create-deployment command
  • The following commands are available when using OpsWorks create-deployment along with possible use cases:
    • install_dependencies
    • update_dependencies – Patches to the Operating System. Not available after Chef 12.
    • update_custom_cookbooks – pulling down changes in your Chef cookbooks
    • execute_recipes – manually run specific Chef recipes that are defined in your layers
    • configure – service discovery or whenever endpoints change
    • setup
    • deploy
    • rollback
    • start
    • stop
    • restart
    • undeploy
  • To enable the use of multiple custom cookbook repositories in OpsWorks, you can enable custom cookbook at the stack and then create a cookbook that has a Berkshelf file with multiple sources. Before Chef 11.10, you couldn’t use multiple cookbook repositories.
  • You can define Chef databags in OpsWorks Users, Stacks, Layers, Apps and Instances
  • OpsWorks Auto Healing is triggered when an OpsWorks Agent detects loss of communication and stops, then restarts the instances. If it fails, it goes into manual intervention
  • OpsWorks will not auto heal an upgrade to the OS
  • OpsWorks does not auto heal by monitoring performance, only failures.


My colleague Casey Lee provided some of the background information on OpsWorks features. I also used several resources from AWS including the PHP sample app and the step-by-step tutorial on the OpsWorks/CodePipeline integration.




Automating Penetration Testing in a CI/CD Pipeline (Part 2)

Continuous Security: Security in the Continuous Delivery Pipeline is a series of articles addressing security concerns and testing in the Continuous Delivery pipeline. This is the sixth article in the series.

In the first post, we discussed what OWASP ZAP is, how it’s installed and automating that installation process with Ansible. This second article of three will drill down into how to use the ZAP server, created in Part 1 for penetration testing your web-based application.

Penetration Test Script

If you recall the flow diagram (below) from the first post, we will need a way to talk to ZAP so that it can trigger a test against our application. To do this we’ll use the available ZAP API and wrap up the API in a Python script. The script will allow us to specify our ZAP server, target application server, trigger each phase of the penetration test and report our results.

ZAP-Basic-CI_CD-Flow - New Page (1)

The core of the ZAP API is to open our proxy, access the target application, spider the application, run an automated scan against it and fetch the results. This can be accomplished with just a handful of commands; however, our goal is to eventually get this bound into a CI/CD environment, so the script will have to be more versatile than a handful of commands.

The Python ZAP API can be easily installed via pip:

pip install python-owasp-zap-v2.4

We’ll start by breaking down what was outlined in the above paragraph. For learning purposes, these can be easily ran from the Python command line.

from zapv2 import ZAPv2

target = "http://" % target_application_url
zap = ZAPv2(proxies={'http': "http://%s" %zap_hostname_or_ip,
                     'https': "https://%s" %zap_hostname_or_ip}
# when status is >= 100, the spider has completed and we can run our scan
# when status is >= 100, the scan has completed and we can fetch results
print zap.core.alerts()

This snippet will print our results straight to STDOUT in a mostly human readable format. To wrap all this up so that we can easily integrate this into an automated environment we can easily change our output to JSON, accept incoming parameters for our ZAP host names and target url. The following script takes the above commands and adds the features just mentioned.

The script can be called as follows:

./pen-test-app.py --zap-host zap_host.example.com:8080 --target app.example.com

Take note, the server that is launching our penetration test does not need to run ZAP itself, nor does it need to run the application we wish to run our pen test against.

Lets set up a very simple web-based application that we can use to test against. This isn’t a real-world example but it works well for the scope of this article. We’ll utilize Flask, a simple Python-based http server and allow it run a basic application that will simply display what was typed into the form field once submitted. The script can be downloaded here.

First Flask needs to be installed and the server started with the following:

pip install flask
python simple_server.py

The server will run on port 5000 over http. Using the example command above, we’ll run our ZAP penetration test against it as so:

/pen-test-app.py --zap-host --target
Spider completed
Info: Scan completed; writing results.

Please note that the ZAP host is simply a url and a port, while the target must specify the protocol, either ‘http’ or ‘https’.

The ‘pen-test-app.py’ script is just an example of one of the many ways OWASP ZAP can be used in an automated manner. Tests can also be written to integrate FireFox (with ZAP as its proxy) and Selenium to mimic user interaction with your application. This could also be ran from the same script in addition to the existing tests.

Scan and Report the Results

The ZAP API will return results to the ‘pen-test-app.py’ script which in turns will write them to a JSON file, ‘results.json’. These results could be easily scanned for risk severities such as “grep -ie ‘high’ -e ‘medium’ results.json”. This does not give us much granularity in determining which tests are reporting errors nor if they critical enough to fail an entire build pipeline.

This is where a tool called Behave comes into play. Behave is a Gerkin-based language that allows the user to write test scenarios in a very human readable format.

Behave can be easily installed with pip:

pip install behave

Once installed our test scenarios are placed into a feature file. For this example we can create a file called ‘pen_test.feature’ and create a scenario.

Feature: Pen test the Application
  Scenario: The application should not contain Cross Domain Scripting vulnerabilities
    Given we have valid json alert output
    When there is a cross domain source inclusion vulnerability
    Then none of these risk levels should be present
      | risk |
      | Medium |
      | High |

The above scenario gets broken down into steps. The ‘Given’, ‘When’ and ‘Then’ will each correlate to a portion of Python code that will test each statement. The ‘risk’ portion is a table, that will be passed to our ‘Then’ statement. This can be read as “If the scanner produced valid JSON, succeed if there are no CSX vulnerabilities or only ones with ‘Low’ severity.

With the feature file in place, each step must now be written. A directory must be created called ‘steps’. Inside the ‘steps’ directory we create a file with the same name as the feature file but with a ‘.py’ extension instead of a ‘.feature’ extension. The following example contains the code for each step above to produce a valid test scenario.

import json
import re
import sys

from behave import *

results_file = 'results.json'

@given('we have valid json alert output')
def step_impl(context):
    with open(results_file, 'r') as f:
            context.alerts = json.load(f)
        except Exception as e:
            sys.stdout.write('Error: Invalid JSON in %s: %s\n' %
                             (results_file, e))
            assert False

@when('there is a cross domain source inclusion vulnerability')
def step_impl(context):
    pattern = re.compile(r'cross(?:-|\s+)(?:domain|site)', re.IGNORECASE)
    matches = list()

    for alert in context.alerts:
        if pattern.match(alert['alert']) is not None:
    context.matches = matches
    assert True

@then('none of these risk levels should be present')
def step_impl(context):
    high_risks = list()

    risk_list = list()
    for row in context.table:

    for alert in context.matches:
         if alert['risk'] in risk_list:
             if not any(n['alert'] == alert['alert'] for n in high_risks):
                 high_risks.append(dict({'alert': alert['alert'],
                                          'risk': alert['risk']}))

    if len(high_risks) > 0:
        sys.stderr.write("The following alerts failed:\n")
    for risk in high_risks:
        sys.stderr.write("\t%-5s: %s\n" % (risk['alert'], risk['risk']))
        assert False

    assert True

To run the above test simply type ‘behave’ from the command line.

Feature: Pen test the Application # pen_test.feature:1

  Scenario: The application should not contain Cross Domain Scripting vulnerabilities # pen_test.feature:7
    Given we have valid json alert output # steps/pen_test.py:14 0.001s
    When there is a cross domain source inclusion vulnerability # steps/pen_test.py:25 0.000s
    Then none of these risk levels should be present # steps/pen_test.py:67 0.000s
      | risk |
      | Medium |
      | High |

1 feature passed, 0 failed, 0 skipped
1 scenario passed, 0 failed, 0 skipped
3 steps passed, 0 failed, 0 skipped, 0 undefined
Took 0m0.001s

We can clearly see what was ran and each result. If this was ran from a Jenkins server, the return code will be read and the job will succeed. If a step fails, behave will return non-zero, triggering Jenkins to fail the job. If the job fails, it’s up to the developer to investigate the pipeline, find the point it failed, login to the Jenkins server and view the console output to see which test failed. This may not be the most ideal method. We can tell behave that we want our output in JSON so that another script can consume the JSON, reformat it into something an existing reporting mechanism could use and upload it to a central location.

To change behave’s behavior to dump JSON:

behave --no-summary --format json.pretty > behave_results.json

A reporting script can either read the behave_results, json file or read the STDIN pipe directly from behave. We’ll discuss more regarding this in the followup post.


If you’ve been following along since the first post, we have learned how to set up our own ZAP service, have the ZAP service penetration test a target web application and examine the results. This may be a suitable scenario for many systems. However, integrating this into a full CI/CD pipeline would be the optimal and most efficient use of this.

In part three we will delve into how to fully integrate ZAP so that not only will your application involve user, acceptance and capacity testing, it will now pass through security testing before reaching your end users.

Stelligent is hiring! Do you enjoy working on complex problems like security in the CD pipeline? Do you believe in the “everything-as-code” mantra? If your skills and interests lie at the intersection of DevOps automation and the AWS cloud, check out the careers page on our website.

Automating Penetration Testing in a CI/CD Pipeline

Continuous Security: Security in the Continuous Delivery Pipeline is a series of articles addressing security concerns and testing in the Continuous Delivery pipeline. This is the first article in the series.

As the internet has matured and grown the need for stronger security has exceeded that pace. Penetration testing has become a large part of the development process. However, most developers do not know how to properly test their application for security or believe their operations team will handle this tedious task. Additionally, finding a severe security issue should raise a red flag and prevent that application release from seeing deployment.


The concept of testing an application for security vulnerabilities sounds quite basic it can be a laborious task. Thankfully a number of tools exist that can make a good portion of this process automatic. While this may not replace a actual person physical attempting to crack your application it will reduce that person’s time by handling on the tedious day-to-day type vulnerability attacks.

What we’re going to look at is implementing these tools into a full CI/CD pipeline, so that our pipeline will fail if our application has any severe vulnerabilities. We will be discussing using OWASP ZAP for our pentest. For this first post we’ll delve into how to install ZAP manually as well as creating an automated method for provisioning ZAP using Ansible.

The Open Web Application Security Project (OWASP) Foundation is a not-for-profit organization that fosters open-source development of trusted tools. Zed Attack Proxy (ZAP) is an OWASP Foundation open-source project designed for web application security scanning.

While ZAP can be used manually, we will utilize its automated functionality for use in a CI/CD environment. ZAP does not need to run on the same server as the application or the script that will interact with ZAP for the penetration test. Here is an example interaction of these services:

ZAP-Basic-CI_CD-Flow - New Page (1)

Installing OWASP ZAP

ZAP can be run in a handful of different modes, from an intercepting proxy, to a spider and an automated scanner, among others. ZAP will require a persistent system to run from that is accessable from your CI/CD pipeline, a jenkins server or it’s own ec2 instance for example. These instructions assume there is a Linux server available for you to use.

ZAP requires Java 7 or higher, refer to your distribution’s instructions for installing Java or follow the Oracle provided instructions for installation.

With Java installed we will install ZAP into the ‘/opt’ directory from the archive available from the OWASP website <link: https://github.com/zaproxy/zaproxy/wiki/Downloads&gt;

wget -q -O - https://github.com/zaproxy/zaproxy/releases/download/2.4.3/ZAP_2.4.3_Linux.tar.gz | tar zxf - -C /opt
ln -s /opt/ZAP_2.4.3 /opt/zap

OWASP ZAP does not include an init script, one can be downloaded and installed; please run the following commands as the root user:

wget -q -O /etc/init.d/zap https://raw.githubusercontent.com/stelligent/zap/master/packer/roles/zap/files/zap-init.sh
chmod 755 /etc/init.d/zap

The init script has a few default settings that may want to be changed prior to running. By default the ZAP daemon is configured to listen on all available network interfaces, on port 8080 and send it’s logs to ‘/var/log/zap.log’. These are variables defined at towards the top of the zap init, downloaded earlier. With everything in order, start ZAP:

/etc/init.d/zap start

If everything started successfully you will be able to hit the url ‘http://localhost:8080‘ if running locally; otherwise, replace ‘localhost’ with the IP or DNS address of your server. This will display the “Welcome to the OWASP Zed Attack Proxy (ZAP)” title page. If this does not display or ZAP failed to start from the init script earlier, check the logs in ‘/var/log/zap.log’ for details.

Lastly, we can configure ZAP to run at startup. This step is optional.

for rl in {2..5}; do ln -s /etc/init.d/zap /etc/rc${rl}.d/S93zap;done
for rl in 0 6; do ln -s /etc/init.d/zap /etc/rc${rl}.d/K5zap;done

Infrastructure as Code

If one of our goals is to achieve a full infrastructure as code, this can be accomplished with a configuration management system of your choice, such as Chef, Puppet or Ansible to provision ZAP. Additionally, if AWS is utilized, this can be baked into an AMI with Packer, which will be covered in a later post, allowing for fully automated provisioning.

Using Ansible in this example, we can provision ZAP to a remote server. If you’re unfamiliar with Ansible, it’s a python-based, open source platform for configuring and managing your systems. Ansible works by creating a “play” that runs on a your system. A series of plays make up a “playbook”, which in turn make up a “role”.

A basic Ansible role can be created to manage the ZAP provisioning, we’ll call it ‘zap’. A playbook will be created for this role that will install ZAP, add the init script created earlier, install Java and register our ZAP service.

Start by creating the following directory structure:

mkdir -p zap/roles/zap{tasks,defaults,files}

We’re now going to create the main playbook. This playbook will manage downloading ZAP and its init script as well as installing Java on the system. Create a new file, ‘zap/roles/zap/tasks/main.yml’ with the following content:


- name: Download ZAP
  unarchive: >
    src={{ zap_url }}

- name: Link zap version
  file: >
    src=/opt/ZAP_{{ zap_version }}

- name: Install init script
  copy: >

- name: Install Java
  yum: >

- name: Register and start zap service
  service: >

The playbook above references the ‘zap-init.sh’ script, this must exist in the ‘files’ directory of our ‘zap’ role.

wget -q -O zap/roles/zap/files/zap-init.sh https://raw.githubusercontent.com/stelligent/zap/master/packer/roles/zap/files/zap-init.sh

Additionally, there are a couple of variables for the ZAP version and a URL for downloading the version of ZAP we’ve been using. It’s best to store these in ‘zap/roles/zap/default/main.yml’ inside our role as follows:

zap_version: "2.4.3"
zap_url: "https://github.com/zaproxy/zaproxy/releases/download/{{ zap_version }}/ZAP_{{ zap_version }}_Linux.tar.gz"

Lastly, a simple playbook that calls the ‘zap’ role is required. Edit ‘zap/install_zap.yml’ and add the following:

- name: Install OWASP ZAP
  hosts: localhost
  sudo: True
    - zap

Ansible can now be called to install ZAP on your target server. From the ‘zap/’ directory, run the following command:

ansible-playbook -l , install_zap.yml

The Ansible role for this can be found in the ‘packer’ directory of the Stelligent Zap repository. Also included in the repository are methods for creating an AWS AMI image as well as a Vagrant system via Test Kitchen for local development and testing. We’ll cover packer more in another post.


We have now covered how to install ZAP manually on to a Linux system as well as providing an automated provisioning method, via Ansible, to help lay the ground work for a future CI/CD environment.

In the next post we’ll dig into how to penetration test your web-based application, manage the results of the scan and integrate everything into your CI/CD environment.


Stelligent is hiring! Do you enjoy working on complex problems like security in the CD pipeline? Do you believe in the “everything-as-code” mantra? If your skills and interests lie at the intersection of DevOps automation and the AWS cloud, check out the careers page on our website.

Serverless Delivery: Architecture (Part 1)

If your application tech stack doesn’t need servers, why should your continuous delivery pipeline? Serverless applications deserve serverless delivery!

The software development discipline of continuous delivery has had a tremendous impact on decreasing the cost and risk of delivering changes while simultaneously increasing code quality by ensuring that software systems are always in a releasable state. However, when applying the tools and techniques that exist for this practice to serverless application frameworks and platforms, sometimes existing toolsets do not align well with these new approaches. This post is the first in a three-part series that looks at how to implement the same fundamental tenets of continuous delivery while utilizing tools and techniques that complement the serverless architecture in Amazon Web Services (AWS).

Here are the requirements of the serverless delivery pipeline:

  • Continuous – The pipeline must be capable of taking any commit on master that passes all test cases to production
  • Quality – The pipeline must include unit testing, static code analysis and functional testing of the application
  • Automated – The provisioning of the pipeline and the application must be done from a single CloudFormation command
  • Reproducible – The CloudFormation template should be able to run on a new AWS account with no additional setup other than creation of a Route53 Hosted Zone
  • Serverless – All layers of the application must run on platforms that meet the definition of serverless as described in the next section

What is Serverless?

What exactly is a serverless platform? Obviously, there is still hardware at some layer in the stack to host the application. However, what Amazon has provided with Lambda is a platform where developers no longer need to think about the following:

  • Operating System  – no need to select, secure, configure, administer or patch the OS
  • Servers– no cost risk of over-provisioning and no performance risk of under-provisioning
  • Capacity – no need to monitor utilization and scale capacity based on load
  • High Availability– compute resources are available across multiple AZs

In summary, a serverless platform is one in which an application can be deployed on top of without having to provision or administer any of the resources within the platform. Just show up with some code and the platform handles all the ‘ilities’.


The diagram above highlights the components that are used in this serverless application. Let’s look at each one individually:

  • Node.js + Express – In the demo application for this blog series, we will be deploying a Node.js JavaScript application that leverages the Express framework into Lambda. There is a small bit of “glue” code that we will highlight later in the series to adapt the Lambda contract to the Express contract.
  • Lambda – This is where your application logic runs. You deploy your code here but do not need to specify the number of servers or size of the servers to run the code on. You only pay for the number of requests and amount of time those requests take to execute.
  • API Gateway – The API Gateway exposes your Lambda function at an HTTP endpoint. It provides capabilities such as authorization, policy enforcement, rate limiting and data transformation as a service that is entirely managed by Amazon.
  • DynamoDB – Dynamic data is stored in a DynamoDB table. DynamoDB is a NoSQL datastore that has both infinitely scalable storage capacity and throughput capacity which is entirely managed by Amazon.
  • S3 – All static content including HTML, CSS, images and client-side JavaScript is stored in an S3 bucket and served as a static website directly from S3.

The diagram below compares the pricing for running a Node.js application with Lambda and API Gateway versus a pair of EC2 instances and an ELB. Notice that for the m4.large, the break even is around two million requests per day. It is important to mention that 98.8% of the cost of the serverless deployment is the API Gateway. The cost of running the application out of Lambda is insignificant relative to the costs in using API Gateway.


This cost analysis shows that applications/environments with low transaction volume can realize cost savings by running on Lambda + API Gateway, but the cost of API Gateway will become cost prohibitive at higher scale.

What is Serverless Delivery?

Serverless delivery is just the application of serverless platforms to achieve continuous delivery fundamentals. Specifically, a serverless delivery pipeline does not include tools such as Jenkins or resources such as EC2, Autoscaling Groups, and ELBs.


The diagram above shows the technology used to accomplish serverless delivery for the sample application. Let’s look at what each component provides:

  • AWS CodePipeline – Orchestrates various Lambda tasks to move code that was checked into GitHub forward towards production.
  • AWS S3 Artifact Bucket – Each action in the pipeline can create a new artifact. The artifact becomes an output from the action in the pipeline and is stored in an S3 bucket to become an input for the next action.
  • AWS Lambda – You create Lambda functions to do the work of individual actions in the pipeline. For example, running a gulp task on a repository is handled by a Lambda function.
  • NPM and Gulp – NPM is used for resolving all the dependencies of a given repository. Gulp is used for defining the tasks of the repository, such as running unit tests and packaging artifacts.
  • Testing toolsJSHint is used for performing static analysis of the code, Mocha is a unit test framework for JavaScript and Chai is a BDD assertion library for describing the unit tests.
  • AWS CloudFormation – CFN templates are used to create and update all these resources in a serverless stack.

Serverless delivery for traditional architectures?

Although this serverless delivery architecture could be applied to more traditional application architectures (e.g., a Java application on EC2 and ELB resources) the challenge might be having pipeline actions complete within the 300 second Lambda maximum. For example, running Maven phases within a single Lambda function invocation, including the resolving of dependencies, compilation, unit testing and packaging would likely be difficult. There may be opportunities to split up the goals into multiple invocations and persist state to S3, but that is beyond the scope of this series.


The pricing model for Lambda is favorable for applications that have an idle time and the cost grows linearly with the number of executions. The diagram below compares the pricing for running the pipeline with Lambda and CodePipeline against a Jenkins server running on an EC2 instance. For best performance, the Jenkins server ought to run on an m4.large instance, but just to highlight the savings, m3.medium and t2.micro instances were evaluated as well. Notice that for the m4.large, the break even happens after you are doing over 600 builds per day and even with a t2.micro, the break even doesn’t happen until well over 100 builds per day.

In conclusion, running a continuous delivery pipeline with CodePipeline + Lambda is very attractive based on the cost efficiency of utility pricing, the simplicity of using a managed services environment, and the tech stack parity of using Node.js for both the application and the pipeline.

Stay Tuned!

Next week we will dive into part two of this series, looking at what changes need to be made for an Express application to run in Lambda and the CloudFormation templates needed to create a serverless delivery pipeline. Finally, we will conclude with part three going into the details of each stage of the pipeline and the Lambda functions that support them.


Provisioning Custom CodePipeline Actions in CloudFormation

A few weeks ago, I’d described the process I went through for provisioning CodePipeline using CloudFormation. As part of the provisioning of AWS CodePipeline, we were using the AWS CodePipeline CLI to provision a couple of custom actions. In my initial refactoring of the CodePipeline provisioning, I didn’t include the provisioning of the CodePipeline custom actions in CloudFormation. Since then, I’ve updated our Dromedary demo scripts to include this provisioning in CloudFormation so I’ll describe the process I went through in making these changes.


There are five types of action categories in CodePipeline: Source, Build, Deploy, Invoke and Test. Each action has four attributes: category, owner, provider and version. There are three types of action owners: AWS, ThirdParty and Custom. AWS refers to built-in actions provided by AWS. Currently, there are three built-in action providers from AWS: S3, CodeDeploy and ElasticBeanstalk. Examples of ThirdParty action providers include RunScope and Github. If none of the action providers suit your needs, you can define custom actions in CodePipeline. As mentioned before, you can do this using the CodePipeline CLI and – now – using CloudFormation.


The reason we needed to define custom actions in the first place was because we’re running custom behavior not provided “out of the box” by CodePipeline to launch environments, run static code analysis, automated tests and so on. For more information, see the StaticCodeAnalysis, CreateEnvironment, AcceptanceTest, InfrastructureTest and PromoteEnvironment action names in codepipeline-cfn.json.

The Resource Type we’re using in CloudFormation is the AWS::CodePipeline::CustomActionType. In the CloudFormation template, we set the Category, Provider, VersionConfigurationProperties, InputArtifactDetails, OutputArtifactDetails, EntityUrlTemplate and ExecutionUrlTemplate for the custom action. This is explained in greater detail below.

  • Category: For the first action, I’m using Build and for the second, I’m using Test.
  • Provider: The custom-generated name for the Jenkins provider we created
  • Version: For the Custom Action. I just used 1.
  • ConfigurationProperties: Whether or not the custom action is required, queryable, etc.
  • InputArtifactDetails: I’m providing the Maximum and Minimum counts for the input artifacts that get supplied to the custom action
  • OutputArtifactDetails: I’m providing the Maximum and Minimum counts for the output artifacts that get supplied to the custom action
  • EntityUrlTemplate: This is the Jenkins URL along with some Jenkins job-specific configuration
  • ExecutionUrlTemplate: This is the Jenkins URL along with the last successfully executed Jenkins job-specific configuration.

I’ve defined two custom actions in this CloudFormation template. One uses the Build action category and the other, the Test category. Otherwise, they’re the same configuration. Ideally, and later, I’ll need to iterate on this using a CloudFormation DSL so that I’m not just copying/pasting the configuration.

You can find the custom actions CloudFormation template at codepipeline-custom-actions.json. This CloudFormation template is integrated with the rest of the CloudFormation templates that provision and configure this infrastructure so it’s not intended to be run on its own. For example, it needs for Jenkins to already be provisioned so that it can reference its unique provider configuration such as URL, name, etc. For instructions on running the Dromedary demo from CloudFormation, see CloudFormation Bootstrapping.

Troubleshooting Custom CodePipeline Actions for CloudFormation

It can be challenging to figure out what may have gone wrong when defining a custom action and using it in a pipeline. While the CodePipeline API will return errors, it’s not always easy to determine what you need to fix. Below, I’ve documented a checklist that you might use in troubleshooting any custom action errors.

  • The Category property of AWS::CodePipeline::CustomActionType must be of one of the following values:
    • Source
    • Build
    • Test
    • Deploy
    • Invoke
  • The Name property in ConfigurationProperties must be a valid configuration property for the Provider (e.g. Jenkins, etc.) you’re defining. For example, in codepipeline-custom-actions.json, ProjectName is a valid configuration property for the Jenkins CodePipeline job worker.
  • When defining EntityUrlTemplate or ExecutionUrlTemplate in AWS::CodePipeline::CustomActionType, the ConfigNameHere (this is not the actual property name) property in {Config:ConfigNameHere} must match one of the property names in ConfigurationProperties of the same custom action
  • Ensure the endpoints for EntityUrlTemplate and ExecutionUrlTemplate actually exist.
  • With the exception of the Source stage, when using the custom action in a pipeline stage in CloudFormation (i.e the Actions property of the Stages property of a AWS::CodePipeline::Pipeline type), InputArtifacts must equal the OutputArtifacts value of a previously executed action. See CodePipeline Concepts for a visualization of this requirement.
  • Run the following from your AWS CLI to see how your custom actions have been defined.
    aws codepipeline list-action-types --query actionTypes[*].id --output table
  • Ensure you’re using valid Provider, Category, Owner and Provider property values. The same properties should be defined in your AWS::CodePipeline::CustomActionType that you’re using in the Actions property of the Stages property of your AWS::CodePipeline::Pipeline type
  • Ensure the job worker for your provider has been configured correctly. See Create a Job Worker for Your Custom Action for more information.

Stelligent AWS Continuous Delivery Demo Screencast

See the YouTube screencast and transcript below of a Continuous Deployment pipeline demonstration for a Node.js application using AWS services such as EC2, DynamoDB, Route 53, ENI and VPC and tools such as AWS CodePipeline, Jenkins, Chef and AWS CloudFormation. Open source code is available at https://github.com/stelligent/dromedary

In this screencast, you’ll see a live demonstration of a system that uses Continuous Deployment of features based on a recent code commit to Github.

You can access this live demo right now by going to demo.stelligent.com. All of the code is available in open source form by going to https://github.com/stelligent/dromedary.

This is a simple Node.js application in which you can click on any of the colors to “vote” for that color and see the results of your votes and others in real time. While it’s a simple application, it uses many of the types of services and tools that you might define in your enterprise systems such as EC2, Route 53, DynamoDB, VPC, Elastic IP and ENI, and it’s built, deployed, tested and released using CloudFormation, Chef, CodePipeline and Jenkins – among other tools.

So, I want you to imagine there’s several engineers on a team that have a dashboard like this on a large monitor showing AWS CodePipeline. CodePipeline is a service released by AWS in July 2015. With CodePipeline, you can model your Continuous Delivery and Continuous Deployment workflows – from the point at which someone commits new code to a version-control repository until it gets released to production.

You can see it shows there’s a failure associated with the infrastructure test. So, let’s take a look at the failure in Jenkins.

Many of you may already be familiar with Jenkins as it’s a Continuous Integration server. In the context of this demo, we’re using CodePipeline to orchestrate our Continuous Deployment workflow and Jenkins to perform the execution of all the code that creates the software system.

Now, the reason for this failure is because we’ve written a test to prevent just anyone from SSH’ing into the EC2 instance from a random host. So, I’m going to put my infrastructure developer hat on and look at the test that failed.

This is an RSpec test to test whether port 22 is accessible to any host. This test gets run every time someone commits any code to the version-control repository.

Based on this test failure, I’m going to look at the CloudFormation template that provisions the application infrastructure. The app_instance.json CloudFormation template defines the parameters, conditions, mappings and resources for the application infrastructure. With CloudFormation there are over 100 built-in AWS resources we define in code. Here, I’m looking at the resource in this template that defines the security group for the EC2 instance that’s hosting the application.

I’m going to update the CIDR block to a specific host using the /32 notation so that only I can access the instance that hosts the application.

Normally, I might wait for the changes made by the other infrastructure developer to successfully pass through the pipeline, but in this case, I’m just going to make my application code changes as well.

So, putting my application developer hat on, I’m going to make an application code change by adding a new color to the pie chart that gets displayed in our application. So, I’ll add the color orange to this pie chart. I’m also going to update my unit and functional tests so that they pass.

Now, you might’ve noticed something in the application. I’m not using SSL. So you go to http, instead of https. That’s insecure. So, I’m going to make some code changes to enable SSL in my application.

Ok. So, I’ve committed my changes to Git and it’s going to run through all the stages and actions in CodePipeline.

…and commit it to Git

So, CodePipeline is polling Github looking for any changes. Since I just committed some new changes, CodePipeline will discover those changes and begin the execution of the defined jobs in Jenkins.

Now, you’ll notice that CodePipeline picked up these changes – the security/infrastructure, SSL, changes along with the application code changes will be built, tested, deployed as part of this deployment pipeline.

You’ll see that there are a number of different stages here, each consisting of different actions. A stage is simply a logical grouping of actions, and is largely dictated by your application or service. The actions themselves call out to other services. There are four types of built-in actions within CodePipeline: source, build, test and deploy. You can also define custom actions in CodePipeline

Ok, now CodePipeline has gone through all of its stages successfully.

You can see that I added the new color, orange, and all my unit and infrastructure tests passed.

It spun up a new infrastructure and used the Blue/Green Deployment pattern to switch to the new environment using Route 53. With a blue-green deployment, we’ve got a production environment (we’ll call this “blue”) and a pre-production environment that looks exactly like production (and we’ll call this “green”). We commit the changes to Github and CodePipeline orchestrates these changes by spinning up the green environment and then uses Route 53 to move all traffic to green. Or, another way of putting this is that you’re switching between production and pre-production environments. Anyway, with this approach you can continue serving users without them experiencing any downtime. You can also potentially rollback to that blue environment can become production again if anything goes wrong with the deployment.

So, let’s summarize what happened…I made application, infrastructure and security changes as code and committed them to Git. It automatically went through a deployment pipeline using CodePipeline and Jenkins – which was also defined in code – It built, ran unit and functional tests, and stored the distro, launched an environment from code and deployed the application – using CloudFormation and Chef. As part of this pipeline, it also ran infrastructure tests as code and deployed to production without anyone lifting another finger. This way you get feedback as soon as an error occurs and only release to production when it passes all these checks. You can do the same thing in your enterprises. From commit to production in minutes or hours instead of days or weeks. Just think about what’s possible when you’re releases become a non-event like this!

AWS CodePipeline released, and there was much rejoicing

AWS can be a big tease. Last November at re:Invent, they announced AWS CodePipeline, an orchestration engine for builds. As they described it, I got more and more excited, up until the point they said it wouldn’t be released in mid-2015. Well, today might as well be Christmas, because CodePipeline has just been released and I am already figuring out how I can use it all over the place.


AWS CodePipeline is part of the Amazon Web Services suite of DevOps tools, and provides orchestration of continuous delivery pipelines in the cloud. It fills the void between where your traditional continuous integration solution ends and your deployments begin (which if you’ve caught me at a conference or meetup in the past few years, you’ve heard me complaining about). It handles orchestrating every step of your build, test, and deploy process, from source code to server deployments.

CodePipeline uses four core building blocks, called providers, to put together your pipeline. The first is your Source Provider, which allows you to define the location of the source code for your application. At this time, CodePipeline supports two source providers: artifacts zipped up and uploaded to an S3 bucket, and repositories hosted on GitHub.

Another type of provider is the Deployment Provider. This provider will deploy your application to your servers. CodePipeline supports two AWS deployment providers: AWS CodeDeploy and AWS Elastic Beanstalk. Either will need to be configured independently, and then hooked in as a CodePipeline stage.

The final types of provider are the most interesting: the Build and Test Providers. With these providers you can execute the logic for building and testing your application. As of right now, CodePipeline supports these providers in a couple different ways. For build providers, there are Jenkins and CloudBees Plugins. Once installed on your Jenkins server, you can create jobs that will poll CodePipeline and execute actions that are part of the pipeline.

If you don’t currently have a Jenkins server, and don’t particularly feel like setting one up, CodePipeline also supports custom actions so you can plug into whatever service you are working with.

For testing, you can still delegate to Jenkins or a customer provider, but CodePipeline comes with built in support for the following test frameworks: Apica and Blazemeter for load testing, Ghost Inspector for UI testing, and Runscope for API monitoring.

To get CodePipeline configured and running, you’ll need to set up a few things. First, you’ll need to install the aforementioned CodePipeline Jenkins Plugin on your Jenkins server if you’re going to be using that. This can be done from your Jenkins Manage Plugins screen, or you can upload the plugin manually following these steps.

You’ll also need an AWS IAM Role configured for CodePipeline to execute AWS actions on your behalf (mostly messing around with S3 buckets). An example of the role you’ll need to available here. When you create your pipeline, you’ll need to reference that role in its configuration.

In addition to those, if you want CodePipeline to orchestrate deployments for you, you’ll need to configure your application to be deployed via CodeDeploy or Elastic Beanstalk. Which one you pick decides on the complexity of your app: simpler applications will do very will with Elastic Beanstalk, but for applications that require a bit of configuration to get going will be better served by using CodeDeploy.

With all those parts in place, setting up your first CodePipeline is as simple as using the wizard to glue all those parts together. Alternatively, CodePipeline is supported by the AWS CLI and the AWS API, and the SDKs have already started rolling out support for it.  Now all I have left to do is sit patiently until it’s supported by CloudFormation.🙂

Stelligent is an expert in deploying continuous delivery pipelines in AWS. If you are looking for help moving your application into CodePipeline, or another type of continuous delivery pipeline, we can definitely help you with that. Reach out to us here.