Stelligent

Validating AWS CloudFormation templates with cfn_nag and mu

  • Paul Duvall

    • This is an older post. For newer information on cfn_nag and mu, please check out these posts:

    Stelligent cfn_nag is an open source command-line tool that performs static analysis of AWS CloudFormation templates. With cfn_nag you can check for:

    Here are some examples of the types of checks cfn_nag can perform:

    You can also extend cfn_nag’s capabilities through custom rules. To learn more on custom rule development, go here.

    One of the key benefits of cfn_nag is that you can learn about security vulnerabilities prior to provisioning AWS resources which can help reduce costs and risk.
    Stelligent mu is an open source DevOps on AWS framework that automatically provisions environments, pipelines, and services using declarative configuration code. The benefits of mu include applying standard best practices to your infrastructure, pipeline, and services and it significantly reduces your programming effort.
    What’s more, you can extend mu’s core capabilities using extensions that you write in AWS CloudFormation.
    In this blog post, you’ll learn how to use the mu-cfn_nag extension to run a set of rules against the CloudFormation templates generated and used by the mu framework. 

    Run cfn_nag

    You can run cfn_nag various ways but for this example, I’m going to use the AWS Cloud9 Integrated Development Environment (IDE). With Cloud9, you can code with only your web browser. No need to download software, configure your environments, or setup your IDE and its configuration on each of your computers. You can collaborate in the same environment with other developers while seeing each of your changes in real time.
    Here are the steps for setting up Cloud9:

    It should take less than one minute to launch the Cloud9 instance. Once it’s provisioned, you should be presented with an IDE that looks similar to Figure 1.

    Figure 1 – AWS Cloud9 IDE

    To run a cfn_nag example, type the following from the Cloud9 terminal: 

    git clone https://github.com/stelligent/cfn_nag_examples
gem install cfn-nag
cd cfn_nag_examples
cfn_nag_scan --input-path cfn/volume.yml

    The above commands clone a GitHub repository that has some cfn_nag examples, installs cfn_nag in the Cloud9 environment, and runs built-in cfn_nag rules against a CloudFormation template called volume.yml. After running the command, you should see an error similar to the output shown below: 

    ------------------------------------------------------------
cfn/volume.yml
------------------------------------------------------------
| FAIL F1
|
| Resources: ["EBSVolume"]
|
| EBS volume should have server-side encryption enabled
Failures count: 1
Warnings count: 0

    As you can see, you receive an error indicating the you must encrypt your EBS Volume.
    To fix the error, run the command below against a modified version of the volume.yml template that sets the EBS volume encryption to true. This CloudFormation template with the encrypted volume should return success. 

    cfn_nag_scan --input-path cfn/volume-encrypted.yml

    Now that you have run failure and success scenarios against CloudFormation templates using cfn_nag, you will provision a full AWS infrastructure solution using mu and then integrate cfn_nag into the deployment pipeline of AWS CodePipeline.

    Run mu to provision AWS resources

    To get started, you’ll need to install mu by running the command below from your Cloud9 IDE terminal:

    curl -s https://getmu.io/install.sh | sudo sh
mu --version

    All mu implementations require a mu.yml file in the root of the repository. The mu.yml defines how mu provisions the AWS resources including VPC, EC2, ECS, RDS, CodePipeline, CodeBuild, CodeDeploy, etc. By default, this solution is deployed into a VPC and a deployment pipeline is generated using the AWS Developer Tools.  
    As a starting point, you can use one of the examples. To get the latest, you can clone the entire mu repository into your Cloud9 environment as shown below:

    git clone https://github.com/stelligent/mu.git

    I chose to use the ecs-fargate example as my starting point. To do this, I created a new GitHub repository and copied the files from the ecs-fargate example directory to my new repository – as shown below:

    git clone https://github.com/PaulDuvall/fargate.git
cp ~/environment/mu/ecs-fargate -r ~/environment/fargate/

    You’ll use the mu.yml file (in the root of the repository) provided by the example as a starting point and then add the two lines in bold below to reference the mu-cfn_nag extension and provide make changes to the existing mu behavior using a local path (url: application):

    ---
environments:
 - name: acceptance
 provider: ecs-fargate
 - name: production
 provider: ecs-fargate
service:
 name: ecs-fargate-example
 healthEndpoint: /
 port: 80
 pathPatterns:
 - /*
extensions:
- url: https://github.com/stelligent/mu-cfn_nag/archive/v0.2.zip
- url: application

    The above causes the mu-cfn_nag extension to be used in the AWS resource provisioning performed by mu. To get the URL of the latest extension, go to the releases section of the GitHub repository. 
    When using url: application as the local path, it’s extending the capabilities of the mu configuration. In this example, it means you need a directory directly under the root of your repository called application. This is a name of your own choosing but the url: extension name must match the name of the local path. To be useful, there must be at least two files within the application path: mu-extension.yml and at least one CloudFormation template that merges its contents with one of the files located in the mu assets directory. To learn more about defining mu extensions, see Create extensions for the mu DevOps on AWS framework.
    To add and commit all the files, run the commands shown below:

    cd ~/environment/fargate/
git add .
git commit -am "add fargate example with mu-cfn_nag"
git push

    Finally, run the command below (see GitHub Personal Access Token section to learn how to generate a GitHub token) to provision the entire solution:

    mu pipeline up -t [GitHub Personal Access Token]

    It will take approximately 30 minutes to automatically provision the environments, services, and pipeline that composes the solution. It will automatically invoke an instance of a deployment pipeline via CodePipeline. To view the AWS provisioning status during this time, go to the CloudFormation and CodePipeline consoles – or via the mu command line utilities.

    GitHub Personal Access Token

    To integrate with GitHub, AWS CodePipeline uses OAuth tokens. Go to GitHub’s Token Settings to generate your token and ensure you enable the following two scopes:

    Submit Approval in Production Stage

    Once the environment, pipeline, and service are successfully provisioned, a pipeline instance in CodePipeline runs a series of actions within stages. Once it gets to the Approve action of the Production stage, the pipeline stops and you must review the application in the acceptance environment. To review the acceptance environment, run the command below and use the output to launch the application or services that have been provisioned by mu.

    mu environment show acceptance

    If everything is working, you will click the Review button in the Approve action and then click the Approve button as shown in Figure 2. Once you do this, the deployment pipeline will complete the action and deploy the software to production.

    Figure 2 – Manual Approval in fully automated CodePipeline workflow

    Verify web application is working

    After the pipeline has successfully deployed the software to production, you can launch the web application by typing a mu command to show the production attributes as shown below. 

    mu environment show production

    After running this command, copy or click on the URL next to Base URL: similar to what is shown below to launch the application in your browser. It should display the words “Automation for the People”.

    Loaded extension cfn_nag (version=0.2)
Loaded extension application (version=0.1)
Environment: production
Cluster Stack: mu-environment-production (UPDATE_COMPLETE)
VPC Stack: mu-vpc-production (UPDATE_COMPLETE)
Bastion Host:
Base URL: http://mu-load-Elb-17UUHVGDSZ4GU-4522229041.us-east-1.elb.amazonaws.com
Services:
+---------------------+----------+------------------+---------------------+
| SERVICE | REVISION | STATUS | LAST UPDATE |
+---------------------+----------+------------------+---------------------+
| ecs-fargate-example | c69d1c4 | UPDATE_COMPLETE | 2018-03-23 01:30:31 |
+---------------------+----------+------------------+---------------------+

    Use cfn_nag with mu

    In this section, you will run through a few scenarios using cfn_nag with AWS CodePipeline. First, you’ll review the default behavior by viewing the warnings generated by cfn_nag, next you will create a failure scenario with cfn_nag, and then fix the error while using CodePipeline to get feedback as you’re conducting these scenarios.

    View CodePipeline

    To view the warnings generated by cfn_nag, follow these steps:

    1. Go to the CodePipeline console and choose the pipeline generated by mu. In my example, it’s named mu-ecs-fargate-example
    2. Go to the CfnNag action within the Acceptance Stage and click on Details
    3. Under the Build Logs section, click on the View Entire Log link (as shown in Figure 3)

    Figure 3 – CodeBuild console success with link to CloudWatch Logs

    After clicking the link, you should see a collection of warnings like the one shown below:

    "violations": [
{
"id": "W29",
"type": "WARN",
"message": "Security Groups found egress with port range instead of just a single port",
"logical_resource_ids": [
"ElbInstanceSecurityGroup",
"ElbSG"
]
},

    These warnings are part of the default scenarios and will not fail the CodePipeline action.

    Failure Scenario

    Next, you will add a failure scenario by copying the cfn folder from the cfn_nag_examples repository to the root of your GitHub repository. To do this, type these commands in your Cloud9 IDE terminal:

    git clone https://github.com/stelligent/cfn_nag_examples.git
cp ~/environment/cfn_nag_examples/cfn -r ~/environment/fargate/
cd ~/environment/fargate/

    Within the application folder, create a new file called vpc.yml and copy the contents from the Resource section of the volume.yml file to the vpc.yml and save, add, and commit the vpc.yml to the Git repository. The contents of the vpc.yml should look like what you see in the code snippet below.

    Resources:
 EBSVolume:
 Type: "AWS::EC2::Volume"
 Properties:
 Encrypted: false
 AvailabilityZone: us-east-1a
 Size: 100

    After you commit the new file to the Git repository, a new pipeline instance is launched. Once it gets to the CfnNag action of the Acceptance stage, it will fail (see Figure 4).

    Figure 4 – CfnNag action error in CodePipeline

    As you did during the success scenario, you can click on the Details of the action to launch CodeBuild and click on the View Entire Log link to see the error details. You should see an error similar to what is shown below.

    [Container] 2018/03/22 19:18:21 Running command find /tmp/mu-cloudformation -name "template-*.yml" -print0 | xargs -0 -n1 cfn_nag
{
19:18:21
"failure_count": 1,
19:18:21
"violations": [
19:18:21
{
19:18:21
"id": "F1",
19:18:21
"type": "FAIL",
19:18:21
"message": "EBS volume should have server-side encryption enabled",
19:18:21
"logical_resource_ids": [
19:18:21
"EBSVolume"
19:18:21
]
19:18:21
}
19:18:21
]
19:18:21
}

    You can undo the changes in your Git repository to see it succeed. 

    Pricing

    If you go through the steps in the demo and terminate the resources after one day, you will pay less than $1 for your usage. The pricing pages for each of the relevant services in the solution are included below:

     

    Additional Resources

    Acknowledgements

    Thanks to Casey Lee, Eric Kascic, and Jonny Sywulak for their contributions to this blog post.

    Stelligent Amazon Pollycast