Security Integration Testing (Part 2): Building and deploying a testing framework on AWS
Continuous Security: Security in the Continuous Delivery Pipeline is a series of articles addressing security concerns and testing in the Continuous Delivery pipeline. This is the fifth article in the series.
Introduction
The purpose of this blog series is to show how AWS Config and Lambda can be used to add Security Integration tests to a Continuous Delivery pipeline. Part 1 covered the basics of the Config service and how to define AWS-managed Config Rules in a CloudFormation template. It also highlighted the Config screens in the AWS Console and showed how to verify that the CFN stack created and configured everything we requested. Part 1 was high level outline to achieve a basic understanding of the Config service and Config Rules. This article will dig a little deeper and show how to use Stelligent’s Config-Rule-Status tool to implement custom Lambda-backed Config Rules to expand the scope of security testing for a CD pipeline. I’ll start by describing the structure of the application and showing how the Lambda functions are defined and how they get associated to Config Rules. Next we’ll step through installation and configuration of the tool, including how it implements the CFN stacks that were covered in the Part 1 blog post. Last, I will show how the application can be extended to define additional Lambda functions and Config Rules so you can adapt it to meet your security testing needs. Let’s get started.
Config-Rule-Status
Overview
The purpose of Config-Rule-Status is to provide a framework for creating and deploying Config Rules and using those rules for security testing in a Continuous Delivery pipeline. It provides a Tester function that acts as an API to get aggregated compliance status from all Config Rules. The output of this function is used by an Acceptance stage pipeline action determine whether the pipeline can continue executing or needs to stop because of security vulnerabilities. Here is a simple visualization of how this tool fits into a pipeline.
Config-Rule-Status was built using the Serverless framework. Serverless provides tools to streamline the management of Lambda functions, most notably packaging, deployment, and versioning. The Serverless commands that Config-Rule-Status depends on are wrapped as Gulp tasks in order to simplify install/config as well as provide synchronous, sequential execution of multiple operations. Using Config-Rule-Status does not require using the Serverless CLI commands directly but all that functionality is still available to use if you choose to use it.
The Rules
This project will create several Config Rules. The CloudFormation template is the best place to look to see what will be created, but here’s a summary.
AWS-Managed Rules
- INSTANCES_IN_VPC – Checks whether your EC2 instances belong to a virtual private cloud (VPC).
- INCOMING_SSH_DISABLED – Checks whether security groups that are in use disallow unrestricted incoming SSH traffic.
- ENCRYPTED_VOLUMES – Checks whether EBS volumes that are in an attached state are encrypted.
Custom Lambda-backed Rules
- EC Rules
- cidrIngress – Checks whether a Security Group has an ingress rule with a CIDR range that disallows unrestricted traffic and applies to a single host.
- cidrEgress – Checks whether a Security Group has an egress rule with a CIDR range that disallows unrestricted traffic and applies to a single host.
- IAM Rules
- userInlinePolicy – Checks whether Users have an inline policy.
- userManagedPolicy – Checks whether Users have a managed policy directly attached.
- userMFA- Checks whether Users have an MFA Device configured.
Project Structure
Within the project Config-Rule-Status makes use of two core Serverless concepts: Functions and Resources. Functions are the core code and config files that define a Lambda function. Resources refer to CFN stacks that define any other non-Lambda resources. Here are the project files related directly to the Lambda functions. I’ve structured it so that all the Lambda functions share the same lib folder during development so as not to duplicate common shared logic in each function folder. At build time the lib gets copied into each function folder so that it can be bundled into the deployment package.
Here we see the other project components.
Installation
Prerequisites: AWS CLI, Node.js/NPM
Install Required Modules
# Install Serverless npm install --global serverless@0.5.5 # Install Gulp npm install --global gulp-cli
Clone and install the project
# clone the repo git clone https://github.com/stelligent/config-rule-status.git # enter the project directory cd config-rule-status # install NPM packages npm install
Initialization
Initialization requires that you have an AWS profile defined. The credentials of the profile will be used by Serverless to deploy resources into your AWS account. This Gulp task is a wrapper for the Serverless action “project init”. It will create a region and stage configuration and then run a CFN stack to build the core resources (as defined in stack template s-resources-cf.json) to support Lambda execution.
# initialize the project gulp init --region us-east-1 --stage prod --name config-rule-status --awsProfile yourProfileName --email user@company.com
Build, Deploy and Test
The initialization step created configurations for the given stage and region and then deployed the required AWS resources.
Next I need to build the deploy packages. This step will run tests and coverage and then create a dist directory and copy the Lambda directories into it. It will also copy the lib and node_modules directories into each of them.
# build the deployment packages gulp build
I can now deploy the Lambda functions to that stage and region by executing the following task:
# deploy the Lambda functions gulp deploy:lambda --stage prod --region us-east-1
Next I need to setup the Config service and create the Config Rules that interface with the Lambda functions. The CloudFormation templates that define these resources are the ones we covered in Part 1 and are located in the otherResources directory. These are kept separate from the Lambda dependencies stack (s-resources-cf.json) because they must be created after the Lambda functions rather than before. To make this possible I have created a new Serverless plugin that defines a new Serverless Action called “synchronousResources deploy”. This gets called for each CFN template in the otherResources directory. These calls are wrapped by the gulp task below:
# deploy the CFN stacks that will setup the Config service # and create Config Rules for each of the Lambda functions gulp deploy:config --stage prod --region us-east-1
After running these two deployment tasks the security testing functionality will be available for use. One of the Lambda functions that gets deployed is the “tester”. It is the component that would be added to a CD pipeline and called during the Acceptance stage to test for security vulnerabilities. It is also useful to call it after these deployment steps to ensure that all the required resources are available and correctly configured. The Gulpfile contains a task that wraps a call to the tester function. So by running this:
# Run the tester Lambda to get the overall Config Rule compliance status # and verify that the deployment was successful. gulp test:deployed --stage prod --region us-east-1
I should get a response with a structure similar to the example below. In this example the overall test result is FAIL because at least one of the Config Rules has an evaluation status of NON_COMPLIANT.
[10:05:17] Starting 'test:deployed'...
Serverless: Running tester...
Serverless: -----------------
Serverless: Success! - This Response Was Returned:
Serverless: {
"result": "FAIL",
"results": [
{
"rule": "ConfigRuleStatus-EC2-Encryption-Rule",
"status": "NON_COMPLIANT",
"result": "FAIL"
},
{
"rule": "ConfigRuleStatus-EC2-SSH-Rule",
"status": "NON_COMPLIANT",
"result": "FAIL"
},
{
"rule": "ConfigRuleStatus-EC2-SecGrp-Cidr-Egress-Rule",
"status": "NON_COMPLIANT",
"result": "FAIL"
},
{
"rule": "ConfigRuleStatus-EC2-SecGrp-Cidr-Ingress-Rule",
"status": "NON_COMPLIANT",
"result": "FAIL"
},
{
"rule": "ConfigRuleStatus-EC2-VPC-Rule",
"status": "COMPLIANT",
"result": "PASS"
},
{
"rule": "ConfigRuleStatus-IAM-MFA-Rule",
"status": "NON_COMPLIANT",
"result": "FAIL"
},
{
"rule": "ConfigRuleStatus-IAM-User-InlinePolicy-Rule",
"status": "NON_COMPLIANT",
"result": "FAIL"
},
{
"rule": "ConfigRuleStatus-IAM-User-ManagedPolicy-Rule",
"status": "NON_COMPLIANT",
"result": "FAIL"
}
],
"timestamp": "2016-04-06T14:05:19.047Z"
}
[10:05:19] Finished 'test:deployed' after 1.72 s
Adding new Config Rules
The config-rule-status tool can be extended by adding new Config Rules and associated Lambda functions. In the following sections I’ll show how it is done and explain how in a bit more detail how the system works. The repo already contains a rule called ec2CidrIngress, but we’ll pretend that i’m creating it from scratch so we can examine the steps.
Step 1: Scaffolding
The first step is to run a Serverless CLI command to generate the scaffolding for the new function. This command accepts two paramaters: functionPath and runtime. For functionPath I will enter components/configRules/ec2CidrIngress which is a path relative to the project root. For runtime I will enter nodejs, though python2.7 is also supported.
# This will generate the Lambda scaffolding serverless function create components/configRules/ec2CidrIngress -r nodejs
After running this command I should now see a new function directory in the project tree. It contains handler.js which the Lambda service will interface with, s-function.json which contains configuration for the function, and event.json which contains event data that Serverless will use when testing the function locally.
Step 2: Update the generated Lambda configuration and handler function
The s-function.json configuration file can be mostly left alone, but there are two critical edits that must be made so the new function will behave properly during the build process. 1) The name value must be updated with an underscore prefix, and 2) the handler value must be handler.handler. Here’s what that looks like:
{
"name": "_ec2CidrIngress",
"runtime": "nodejs",
"description": "Serverless Lambda function for project: config-rule-status",
"customName": "$${functionName}",
"customRole": false,
"handler": "handler.handler",
"timeout": 6,
"memorySize": 128,
"authorizer": {},
"custom": {
"excludePatterns": [],
"envVars": []
},
"endpoints": [],
"events": [],
"environment": {
"SERVERLESS_PROJECT": "${project}",
"SERVERLESS_STAGE": "${stage}",
"SERVERLESS_REGION": "${region}"
},
"vpc": {
"securityGroupIds": [],
"subnetIds": []
}
}
The hander.js file must also be updated. Here you can see the handler is kept very simple and all the evaluation logic is pushed down into library modules. The handler function calls template.defineTest and passes it the typical event and context, but also three additional strings that are used to lookup the required evaluation logic.
'use strict';
/**
* Serverless Module: Lambda Handler
*/
// Require Logic
var template = require('../../lib/template');
// Lambda Handler
module.exports.handler = function(event, context) {
template.defineTest(event, context, 'EC2', 'SecurityGroup', 'CidrIngress');
};
Step 3: Add evaluation logic
There are several lib modules that come into play when defining a test. I recommend you look at aws.js, ec2.js, and config.js. I am going to skip down to the bottom of the stack and show the core execution logic. This snippet shows the test function of the EC2.CidrIngress object, which is what eventually gets called to check the compliance of each security group.
'use strict';
module.exports.getRules = function() {
var globLib = require('./global');
var iam = globLib.iam;
return {
'IAM': {
'MFADevice': {
test: function(user, configurator) {
var compliance = 'NON_COMPLIANT';
var params = {
'UserName': user.UserName
};
iam.listMFADevices(params, function(err, data) {
var responseData = {};
if (err) {
responseData = {
Error: 'listMFADevices call failed'
};
console.error(responseData.Error + ':n', err.code + ': ' + err.message);
} else {
if (data.MFADevices.length >= 1) {
compliance = 'COMPLIANT';
}
console.info('compliance: ' + compliance);
configurator.setConfig(compliance);
}
});
}
},
'InlinePolicy': {
test: function(user, configurator) {
var params = {
'UserName': user.UserName
};
var compliance = 'UNKNOWN';
iam.listUserPolicies(params, function(err, data) {
var responseData = {};
if (err) {
responseData = {
Error: 'listUserPolicies call failed'
};
err = responseData.Error + ':n' + err.code + ': ' + err.message;
console.error(err);
} else {
if (data.PolicyNames.length === 0) {
compliance = 'COMPLIANT';
} else {
compliance = 'NON_COMPLIANT';
}
console.info('compliance: ' + compliance);
configurator.setConfig(compliance);
}
});
}
},
'ManagedPolicy': {
test: function(user, configurator) {
var params = {
'UserName': user.UserName
};
var compliance = 'NON_COMPLIANT';
iam.listAttachedUserPolicies(params, function(err, data) {
var responseData = {};
if (err) {
responseData = {
Error: 'listAttachedUserPolicies call failed'
};
console.error(responseData.Error + ':n', err.code + ': ' + err.message);
} else {
if (data.AttachedPolicies.length === 0) {
compliance = 'COMPLIANT';
}
console.info('compliance: ' + compliance);
configurator.setConfig(compliance);
}
});
}
}
},
'EC2': {
'CidrIngress': {
test: function(secGrp, configurator) {
var compliance;
var nonCompCnt = 0;
var cidrRangeRegex = '^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(/([0-9]|[1-2][0-9]|3[0-2]))$';
secGrp.IpPermissions.forEach(function(ipPerm) {
ipPerm.IpRanges.forEach(function(ipRange) {
//check if cidrIp is populated with a cidr or a security group
if (ipRange.CidrIp.search(cidrRangeRegex) !== -1) {
//if it's a cidr then make sure it's not open to the world
if (ipRange.CidrIp === '0.0.0.0/0') {
nonCompCnt++;
}
//make sure it applies to a single host
if (ipRange.CidrIp.split('/')[1] !== '32') {
nonCompCnt++;
}
}
});
});
compliance = nonCompCnt === 0 ? 'COMPLIANT' : 'NON_COMPLIANT';
console.info('compliance: ' + compliance);
configurator.setConfig(compliance);
}
},
'CidrEgress': {
test: function(secGrp, configurator) {
var compliance;
var nonCompCnt = 0;
var cidrRangeRegex = '^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(/([0-9]|[1-2][0-9]|3[0-2]))$';
secGrp.IpPermissionsEgress.forEach(function(ipPerm) {
ipPerm.IpRanges.forEach(function(ipRange) {
//check if cidrIp is populated with a cidr or a security group
if (ipRange.CidrIp.search(cidrRangeRegex) !== -1) {
//if it's a cidr then make sure it's not open to the world
if (ipRange.CidrIp === '0.0.0.0/0') {
nonCompCnt++;
}
//make sure it applies to a single host
if (ipRange.CidrIp.split('/')[1] !== '32') {
nonCompCnt++;
}
}
});
});
compliance = nonCompCnt === 0 ? 'COMPLIANT' : 'NON_COMPLIANT';
console.info('compliance: ' + compliance);
configurator.setConfig(compliance);
}
}
}
};
};
Step 4: Write some tests
The testing framework uses mocha, sinon, and chai (with chai-as-promised). These, along with my lambdaRunner test harness, make it easy to locally test the Lambda function even though it’s evaluation is asynchronous. Here’s an example of a test from ec2-tests.js. Most of the real estate is consumed by the stubbed output and dummy event. The two most interesting lines are highlighted below.
it('should be COMPLIANT',
function() {
secGrpStub.yields(null, {
'SecurityGroups': [{
'IpPermissionsEgress': [{
'IpProtocol': '-1',
'IpRanges': [{
'CidrIp': '0.0.0.0/0'
}],
'UserIdGroupPairs': [],
'PrefixListIds': []
}],
'Description': 'launch-wizard-1 created 2016-03-10T10:51:56.616-05:00',
'IpPermissions': [],
'GroupName': 'launch-wizard-1',
'VpcId': 'vpc-f399e097',
'OwnerId': '592804526322',
'GroupId': 'sg-ed58dd95'
}]
});
var event = {
"invokingEvent": "{"configurationItem":{"configurationItemCaptureTime":"2015-09-25T04:05:35.693Z","configurationItemStatus":"OK","resourceId":"sg-ed58dd95","resourceType":"AWS::EC2::SecurityGroup","tags":{},"relationships":[]}}",
"ruleParameters": "{}",
"resultToken": "null",
"eventLeftScope": false
};
var lambdaResult = lambdaRunner('components/configRules/ec2CidrIngress', event);
return expect(lambdaResult).to.eventually.have.deep.property('compliance', 'COMPLIANT');
}
);
Step 5: Update the Config Rule Stack template
To accommodate a new Config Rule, two new resource definitions must be added to config-rule-resources.json. The first resource is for the new Lambda function. The second is for a permission that allows the Config service to invoke the function. These look complicated but it’s mainly just concatenation gymnastics to build the ARNs for the SourceIdentifier attribute.
"ec2SecGrpCidrIngressRule": {
"Type": "AWS::Config::ConfigRule",
"Properties": {
"ConfigRuleName": "ConfigRuleStatus-EC2-SecGrp-Cidr-Ingress-Rule",
"Description": "Checks whether a Security Group has an ingress rule with a CIDR range that disallows unrestricted traffic and applies to a single host.",
"Scope": {
"ComplianceResourceTypes": [
"AWS::EC2::SecurityGroup"
]
},
"Source": {
"Owner": "CUSTOM_LAMBDA",
"SourceDetails": [{
"EventSource": "aws.config",
"MessageType": "ConfigurationItemChangeNotification"
}],
"SourceIdentifier": {
"Fn::Join": ["", [{
"Fn::FindInMap": ["LambdaArn", "Base", "Segment"]
}, {
"Fn::Join": [":", [{
"Ref": "AWS::Region"
}, {
"Ref": "AWS::AccountId"
}]]
}, {
"Fn::FindInMap": ["LambdaArn", "FunctionPrefix", "Segment"]
}, {
"Fn::Join": [":", ["-ec2CidrIngress", {
"Ref": "LambdaStage"
}]]
}]]
}
}
},
"DependsOn": "ec2SecGrpCidrIngressPerm"
},
"ec2SecGrpCidrIngressPerm": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": {
"Fn::Join": ["", [{
"Fn::FindInMap": ["LambdaArn", "Base", "Segment"]
}, {
"Fn::Join": [":", [{
"Ref": "AWS::Region"
}, {
"Ref": "AWS::AccountId"
}]]
}, {
"Fn::FindInMap": ["LambdaArn", "FunctionPrefix", "Segment"]
}, {
"Fn::Join": [":", ["-ec2CidrIngress", {
"Ref": "LambdaStage"
}]]
}]]
},
"Action": "lambda:InvokeFunction",
"Principal": "config.amazonaws.com"
}
}
Wrapping up
In this article we learned how to install Config-Rule-Status and use it to deploy Config Rules for security monitoring on AWS. We also learned how to extend its coverage by adding new Config Rules. In Part 3 we’ll take a deeper dive into testing and deploying Config-Rule-Status and also show how to use it to add security integration testing to a Continuous Delivery pipeline. Stay tuned.
Stelligent is hiring! Do you enjoy working on complex problems like security in the CD pipeline? Do you believe in the “everything-as-code” mantra? If your skills and interests lie at the intersection of DevOps automation and the AWS cloud, check out the careers page on our website.
|
Stelligent Amazon Pollycast
 |