Continuous Security: Security in the Continuous Delivery Pipeline

Continuous Security is the addressing of security concerns and testing in the Continuous Delivery pipeline, and is as much a part of continuous delivery as operations, testing, or security is a part of the DevOps culture. This article is the first in a series which talks about ways of integrating security testing/validation of both software and infrastructure into the Continuous Delivery pipeline.
DevOps is a culture that emphasizes the collaboration and communication of all IT roles necessary to develop, deliver and support software and infrastructure through automation while eliminating silos and delays caused by organizational boundaries. Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time [1].
Continuous Delivery is used to build, test and deploy code rapidly, ideally with the automated ability for any given change to flow into production with the click of a button. It relies on the complete automation of the deployment process, such that bottlenecks, handoffs, and human intervention are eliminated. You are capable of deploying any code change immediately to production, and are not shackled by an onerous and manual release process. Only business drivers influence the decision to deploy; uncertainty over the coordination and execution of a labor intensive deployment never enters the equation.
Automated testing is fundamental to the Continuous Delivery process. Testing in the pipeline both eliminates delays from manual verification of code, as well as ensuring consistent verification of functionality. Various types of testing are performed in all of the stages of the pipeline in order to provide feedback to the developer as rapidly as possible. A typical pipeline is depicted in the image below.

What does this all have to do with security? Everything!
In the cloud, we talk often about infrastructure-as-code, spinning up compute resources on demand. But really, we are in an environment that should be treated as everything-as-code. Sure, we create instances from code, but we also codify networks, security, monitoring, operations and feedback mechanisms. The same Continuous Delivery process that allows you to test, verify and deploy application changes applies equally to networking, security monitoring, etc., now that those functions are represented as code.
And while Continuous Delivery is typically looked upon as an accelerator for developers, allowing business value to be deliver much more rapidly, an interesting transformation takes place when security is made a core focus of the pipeline. Change occurs more rapidly, but at the same time governance and security are drastically improved. This is due to the iterative testing of incremental change and the capturing of security requirements as code, rather than being manually, and often inconsistently, applied.
Commit Stage
What might it look like to validate your security posture in the Continuous Delivery pipeline? Referring back to the pipeline graphic, the first stage is Commit. This stage provides very rapid feedback to the developer, as the code is analyzed before building. For security, this means making a decision about the resources and changes that would be triggered by executing the code, before actually triggering resource creation/modification. Not only does this provide rapid feedback (literally seconds), but has powerful security implications in that infrastructure code that would otherwise expose a vulnerability can be vetted and prevented from being provisioned.
In order to bring this type of commit stage testing to CloudFormation, we have created a tool called cfn-nag, which builds a model of what your templates would do, and analyzes it against a set of user-controllable rules. Only if the template passes all tests, will it be subsequently executed.
Acceptance Stage
Once code passes the Commit stage, it is actually built and acceptance testing is performed. In terms of infrastructure code, this means AWS resources are created. Whenever you create AWS resources, you are creating or modifying your security posture, whether you want to or not. AWS has released a powerful tool for analyzing resources against security considerations, called Config Rules, perfect for resource verification in the Acceptance stage. Implementing Config Rules in the Continuous Delivery pipeline presents some interesting challenges, which will be covered in detail in an upcoming article.
Capacity Stage
After making it past the Acceptance stage gauntlet, We move on to the Capacity stage. Capacity is where we build and validate an environment capable of going into production. Since you anticipate this environment could go live, you want some in-depth security analysis here. One of our favorite security tools for evaluating web applications is the OWASP Zed Attack Proxy (ZAP). ZAP is typically used manually through a GUI, which doesn’t work well with our “automate everything” mindset. Utilizing Behave and some custom Python, we are able to parse the JSON output of ZAP and assess the results via an english-like DSL.
Exploratory and Production Stages
The last two stages are Exploratory and Production. Production is the stage that promotes a candidate environment that has passed all previous stages into production. Exploratory is really an out-of-band stage that provides a self-service model for creating clones of the production environment for experimentation. We’ll focus on Production stage and leave Exploratory for another day.
Production is a very interesting stage. In typical software development CD pipeline, Production is the end of the line. Hit this stage and you are done. But not so much with security. While your application is deployed and immutable, your security posture is still subject to change. You may have firm control of your CloudFormation templates through the pipeline, but resources can be added, changed and deleted independent of your templates.
For this reason, we strongly recommend that the Production stage provide periodic, ongoing testing of the security of the live environment. Ensuring that an environment remains in conformance with your rules is compliance, and can take the form of internal compliance with your own best practices, or compliance with industry/governmental requirements.
An interesting tool we are currently integrating into the CD pipeline is Inspector, an AWS security assessment service. This tool is still in preview mode, so more from us on integrating it as it hits general availability. We are also experimenting with integrating other open source tools into our pipeline, such as OpenSCAP.
Continuous Delivery is a discipline that provides rapid feedback and consistent, repeatable validation of software and infrastructure. There are a wealth of tools that provide valuable insight into your security posture. Leveraging these tools within your CD pipeline helps promote security to a first class citizen of the development process. Please join us for the ensuing articles that dive much deeper into the Continuous Security pipeline, and how to integrate security testing the Stelligent way.
Stelligent is hiring! Do you enjoy working on complex problems like security in the CD pipeline? Do you believe in the “everything-as-code” mantra? If your skills and interests lie at the intersection of DevOps automation and the AWS cloud, ping me on Twitter, LinkedIn or check out the careers page on our website.
[1] – http://martinfowler.com/bliki/ContinuousDelivery.html

Stelligent Amazon Pollycast