Security Integration Testing (Part 3): Integrating with a Continuous Delivery pipeline
Continuous Security: Security in the Continuous Delivery Pipeline is a series of articles addressing security concerns and testing in the Continuous Delivery pipeline. This is the seventh article in the series.
Introduction
The purpose of this blog series is to show how AWS Config and Lambda can be used to add Security Integration tests to a Continuous Delivery pipeline. Part 1 covered setting up the Config service and creating AWS-managed Config Rules. Part 2 stepped through the process of running Stelligent’s Config-Rule-Status tool to create and deploy Lambda-backed Config Rules. Here in Part 3 I will expand on that topic and show how to run local functional tests for the Lambda-backed rules and how to manage versioning and deployment to AWS. Finally, I will wrap up the series by describing how to use the “Tester” Lambda function to add a security integration test to a Continuous Delivery pipeline.
Before diving into the technical details, here’s a refresher on what this is all about. Config-Rule-Status sets up the Config service and Config Rule monitoring. Then, as represented in the image below, a CD pipeline step will call out to the Tester lambda function to get the security compliance status of the infrastructure. If the status is non-compliant then the pipeline stops.
Config-Rule-Status…continued
In Part 2 of this series I showed how to install and configure the Config-Rule-Status tool. I have also included a quick summary of those steps here. If you were following along in the previous post and already have it installed then you can skip it.
Install (quickly revisited)
[code language=”bash” gutter=”false”]# Install Serverlessnpm install –global serverless@0.5.5
# Install Gulp
npm install –global gulp-cli
# clone the repo
git clone https://github.com/stelligent/config-rule-status.git
# enter the project directory
cd config-rule-status
# install NPM packages
npm install
# initialize the project
gulp init \
–region us-east-1 \
–stage prod \
–name config-rule-status \
–awsProfile yourProfileName \
–email user@company.com
[/code]
Local Lambda tests
With a simple gulp task the Lambda functions will be tested locally and test coverage will be analyzed. This task will also be run by the build task, but here I’m showing what happens when you run it by itself.
bash-3.2$ gulp test
[10:11:41] Using gulpfile ~/GoogleDrive/Sync/projects/config-rule-status/gulpfile.js
[10:11:41] Starting 'lint'...
[10:11:41] Finished 'lint' after 230 ms
[10:11:41] Starting 'pre-test'...
[10:11:41] Finished 'pre-test' after 136 ms
[10:11:41] Starting 'test:local'...
[10:11:41] Finished 'test:local' after 632 ?s
[10:11:41] Starting 'test'...
[10:11:41] Finished 'test' after 4.23 ?s
ec2CidrIngress
? should be rejected with undefined invokingEvent.configurationItem
? should be InvalidGroup
? should be COMPLIANT (1017ms)
? should be NON_COMPLIANT (1006ms)
ec2CidrEgress
? should be InvalidGroup
? should be COMPLIANT (1004ms)
? should be NON_COMPLIANT (1006ms)
IAM/userInlinePolicy
? should be NoSuchEntity
? should be COMPLIANT (1004ms)
? should be NON_COMPLIANT
IAM/userManagedPolicy
? should be NoSuchEntity
? should be COMPLIANT
? should be NON_COMPLIANT
IAM/userMFA
? should be NoSuchEntity on call to getUser
? should be COMPLIANT
? should be NON_COMPLIANT
tester
? should error on describeConfigRules
? should PASS
? should FAIL
19 passing (5s)
-----------------------------------|----------|----------|----------|----------|----------------|
File | % Stmts | % Branch | % Funcs | % Lines |Uncovered Lines |
-----------------------------------|----------|----------|----------|----------|----------------|
complianceTest/tester/ | 88.46 | 90 | 100 | 88.46 | |
handler.js | 88.46 | 90 | 100 | 88.46 | 25,28,29 |
configRules/ec2CidrEgress/ | 100 | 100 | 100 | 100 | |
handler.js | 100 | 100 | 100 | 100 | |
configRules/ec2CidrIngress/ | 100 | 100 | 100 | 100 | |
handler.js | 100 | 100 | 100 | 100 | |
configRules/iamUserInlinePolicy/ | 100 | 100 | 100 | 100 | |
handler.js | 100 | 100 | 100 | 100 | |
configRules/iamUserMFA/ | 100 | 100 | 100 | 100 | |
handler.js | 100 | 100 | 100 | 100 | |
configRules/iamUserManagedPolicy/ | 100 | 100 | 100 | 100 | |
handler.js | 100 | 100 | 100 | 100 | |
lib/ | 92.41 | 70.73 | 100 | 92.41 | |
aws.js | 90.91 | 71.43 | 100 | 90.91 | 29,30 |
config.js | 87.5 | 50 | 100 | 87.5 | 24,25 |
ec2.js | 100 | 100 | 100 | 100 | |
global.js | 100 | 100 | 100 | 100 | |
iam.js | 100 | 100 | 100 | 100 | |
rules.js | 89.23 | 67.86 | 100 | 89.23 |... 45,46,69,72 |
template.js | 100 | 100 | 100 | 100 | |
-----------------------------------|----------|----------|----------|----------|----------------|
All files | 92.47 | 74.51 | 100 | 92.47 | |
-----------------------------------|----------|----------|----------|----------|----------------|
=============================== Coverage summary ===============================
Statements : 92.47% ( 172/186 )
Branches : 74.51% ( 38/51 )
Functions : 100% ( 38/38 )
Lines : 92.47% ( 172/186 )
================================================================================
Managing versions and deployments
When preparing to deploy the Lambda functions the gulp build task needs to be run to copy and stage the files into a dist folder. It copies the function folders to dist and then injects each one with a copy of the lib folder and a copy of the node_modules. Then running the gulp deploy:lambda task will package and deploy all the functions that reside in that dist folder. Here is what the generated dist folder looks like. Notice that each function now contains its own copy of the shared dependencies that originally reside in the components folder.
With the build step done, the Lambda functions are ready for deployment to AWS. When the deployment step is run it will deploy the function package and publish a new version of the Lambda function. The –stage parameter that is included in the gulp deploy call is used to attach an alias to the version. This mechanism makes it possible to deploy a new version of the function code to a “dev” or “beta” stage which allows you to then smoke test it on AWS before deploying it to production. If the smoke test passes then prod deployment is done by running the gulp deploy task again with –stage set to “prod”. Once that is done then the Config Rules will be using the newly deployed Lambda functions as their evaluation logic. This works because the Config Rules were defined to reference only a specific aliased versions of the Lambda functions. So if you ran “gulp deploy:config” with –stage set to “prod”, then the Config Rules only use Lambda functions with a “prod” alias.
In this example we can see that version 4 of the function has the “prod” alias (stage), and AWS allows us to reference it with an ARN that includes the alias.
Here we can see that the Config Rule, generated by executing the gulp deploy task for the prod stage uses the alias qualified ARN to define its association to a Lambda function.
So that’s a lot of explanation, but as you saw in Part 2, the execution required to configure all this is very simple. A couple of CLI tasks sets it all up:
[code language=”bash” gutter=”false”]# deploy the Lambda functions
gulp deploy:lambda –stage prod –region us-east-1
# deploy the CFN stacks that will setup the Config service
# and create Config Rules for each of the Lambda functions
gulp deploy:config –stage prod –region us-east-1[/code]
Post deployment smoke test
This process was covered in Part 2, but it bears repeating here because it is the interface to the security testing functionality and integral to CD pipeline integration as shown in the following section.
[code language=”bash” gutter=”false”]# Run the tester Lambda to get the overall Config Rule compliance status
# and verify that the deployment was successful.
gulp test:deployed –stage prod –region us-east-1[/code]
In this example the overall test result is FAIL because at least one of the Config Rules has an evaluation status of NON_COMPLIANT.
[10:05:17] Starting 'test:deployed'...
Serverless: Running tester...
Serverless: -----------------
Serverless: Success! - This Response Was Returned:
Serverless: {
"result": "FAIL",
"results": [
{
"rule": "ConfigRuleStatus-EC2-SecGrp-Cidr-Ingress-Rule",
"status": "NON_COMPLIANT",
"result": "FAIL"
},
{
"rule": "ConfigRuleStatus-EC2-VPC-Rule",
"status": "COMPLIANT",
"result": "PASS"
},
{
"rule": "ConfigRuleStatus-IAM-MFA-Rule",
"status": "NON_COMPLIANT",
"result": "FAIL"
},
{
"rule": "ConfigRuleStatus-IAM-User-InlinePolicy-Rule",
"status": "NON_COMPLIANT",
"result": "FAIL"
},
{
"rule": "ConfigRuleStatus-IAM-User-ManagedPolicy-Rule",
"status": "NON_COMPLIANT",
"result": "FAIL"
}
],
"timestamp": "2016-04-06T14:05:19.047Z"
}
[10:05:19] Finished 'test:deployed' after 1.72 s
CD Pipeline integration
The CD pipeline integration is where the value of this framework is realized. This is done by adding a simple call to the Tester Lambda function to your pipeline’s Acceptance stage. The particular implementation will vary depending on the tools that implement your pipeline, but the fundamental logic will be the same. The logic is as follows:
IF the object.result returned from the Tester Lambda function equals “PASS”
THEN the pipeline action succeeds
ELSE the pipeline action fails
Here is an example of how this could be implemented in javascript as a mocha test.
[code language=”javascript” gutter=”false”]’use strict’;
var chai = require(‘chai’);
var chaiAsPromised = require(‘chai-as-promised’);
var expect = chai.expect;
var lambdaRunner = require(‘./lib/remoteRunner.js’).lambdaRunner;
chai.use(chaiAsPromised);
describe(‘ConfigRuleStatus-tester’, function() {
it(‘should PASS’,
function() {
var event = {};
var lambdaResult = lambdaRunner(‘ConfigRuleStatus-tester’, ‘us-east-1’, ‘prod’, event);
return expect(lambdaResult).to.eventually.have.property(‘result’, ‘PASS’);
}
);
});[/code]
If any Config Rules are non-compliant then the Tester Lambda will return an object containing “result”: “FAIL” and exit with an error. This will stop the pipeline and prevent vulnerabilities from being added to the production infrastructure.
1 failing
1) ConfigRuleStatus-tester should PASS:
AssertionError: expected { Object (result, results, ...) } to have a property 'result' of 'PASS', but got 'FAIL'
events.js:154
throw er; // Unhandled 'error' event
^
Error: 1 test failed.
Wrapping up
This blog series on Security Integration Testing has touched on many topics and tools, but the concept that ties it all together is what Stelligent calls “Continuous Security”. We at Stelligent believe that enforcing infrastructure security policies within the software delivery process is a fundamental requirement and should be a integral part of all Continuous Delivery pipelines. Static analysis of infrastructure code, security integration testing, and penetration testing are the three main building blocks of the Continuous Security capability. This series showed how to leverage AWS Config to enable continuous infrastructure monitoring, laying the groundwork to build the tooling to implement security integration tests that can be run during the Acceptance stage of a CD pipeline. Running these tests ensures that all existing infrastructure resources, and any newly provisioned resources comply with the security rules. Thanks for reading and stay tuned for more posts on Continuous Security.
Stelligent is hiring! Do you enjoy working on complex problems like security in the CD pipeline? Do you believe in the “everything-as-code” mantra? If your skills and interests lie at the intersection of DevOps automation and the AWS cloud, check out the careers page on our website.
|
Stelligent Amazon Pollycast
 |