Introduction

Many enterprises attempt to drive software development and delivery towards a DevOps mindset. Likewise, organizations struggle with increasing security challenges while adopting these innovative software practices.

Embedding security within the deployment lifecycle is non-negotiable. Therefore, integration of security into CI/CD workflows need to be done cautiously to meet an ever-evolving technology landscape.

DevSecOps is an important concept that automates and integrates security into the software delivery lifecycle. Some challenges exist with adding security controls into containerized solutions.

Furthermore, considering open source containers, many contain known and unknown vulnerabilities. Most organizations struggle to ascertain, with a high level of confidence, how secure these containers are. These tools introduce additional functionality and beneficial features that result in faster solution implementation. In contrast, many are not compliant with organizational security goals. The reality is that failure to design secure deployment pipelines can cost a company millions.

This blog post provides a deployment guide on how to create, launch, and implement a standalone container scanning solution – Trend Micro Deep Security Smart Check – within the AWS ecosystem. It describes how to set up a release pipeline for the automated vulnerability scanning of container images.

 

Solution Overview

This approach uses Trend Micro’s container scanning tool, Deep Security Smart Check (DSSC), as a proof-of-concept and provides examples of how DSSC can be integrated with AWS CodeBuild and AWS CodePipeline.

DSSC is a container compliance platform that ensures the security and stability of container deployments.

DSSC automates the inspection, analysis, and evaluation of images against a compliance checklist that includes items from PCI-DSS, HIPAA, and NIST 800-190.  Checks for source code vulnerabilities are performed using the vulnerability database from Snyk.  Therefore, it delivers a high confidence in container deployments.  Each scan process provides a policy evaluation result for each image which can be used in your CI/CD process to pass or fail the pipeline.

For more detailed information and an overview, please visit Trend Micro Deep Security Smart Check.

Pipeline Architecture

High level overview of the pipeline architecture.

AWS Services Used

  • AWS CloudFormation
  • AWS ECR
  • AWS EKS
  • AWS ApiGateway
  • AWS Lambda
  • AWS Parameter Store
  • AWS CodePipeline
  • AWS CodeBuild
  • AWS CloudWatch

 

Getting Started

This application requires a few steps that must be executed as specified.  Please refer to the prerequisites [link added in WP] section prior to running any commands contained in this article to ensure you have the required packages and software installed.

WARNING:  You may incur charges with the use of some of these AWS Services, specifically EKS if you leave your cluster up and running over time.

Prerequisites

Ensure that the following are installed or configured on your workstation before deploying DSSC.

Installation

Deployments

Create EKS Cluster

DSSC requires a Kubernetes cluster to run on so we’ll be using AWS EKS.

          make deploy-cluster

This can take upwards of 15 minutes to create.

This command uses eksctl to create the cluster and two unmanaged nodes in preparation for the DSSC deployment.

Deploy DSSC

Deploy DSSC to the EKS cluster created previously.  Once deployed, the default `password` will be automatically changed to allow API access.

          make deploy-dssc

This command uses helm to deploy DSSC to the cluster.  Detailed instructions are located at https://github.com/deep-security/smartcheck-helm.

There will be instructions output to the console on where and how to login to the DSSC administrative UI, but it is not required for this example.

This command will create four AWS Parameter Store variables to be used by the pipeline, all prefixed with `/pipeline/example/trendmicro/dssc`.

  • username – The DSSC username.
  • password – The new DSSC password.
  • url – The URL for UI/API access.
  • secret – The secret phrase used to sign the X-Scan-Events-Signature header.

Create ECR Repository

We need an image repository to store our sample application image.  We will be using AWS ECR.

          make deploy-ecr

This command will create an AWS ECR image repository.  This repository is used to house the sample app image which DSSC will use when a scan is initiated.

Push Sample Image

We need an image to scan, so a sample has been provided.  To build and push the sample image to the ECR repository we just created, run the following.

          make build-and-push-docker-image

This will build a Docker image using the application located in ./sample_app and push it to the ECR repository created previously.

Create Webhook

DSSC supports webhooks, which is a way for DSSC to communicate with other processes when certain events occur, like when a scan completes.  Leveraging a webhook is how we will integrate DSSC into our CI/CD pipeline. This means we need something for DSSC to call when scans are complete, so we will create a webhook for DSSC to call.

          make deploy-webhook

This command creates an AWS ApiGateway, AWS Lambda, and an AWS SSM parameter storing the URL to access it.

The Lambda code is in `cloudformation/webhook.yaml’.  This code takes the incoming JSON posted to it from DSSC and simply looks at the ‘critical’ and ‘high’ errors counts in the JSON.  The code then approves or rejects the ‘Approve Deployment’ pipeline action.  The code also checks the X-Scan-Event-Signature header (using the DSSC Secret that was created with the deploy-dssc-ssm command) to determine if the call is a valid one from DSSC.  A 401 is returned if the signature is invalid.  Refer to Securing Web Hooks for more information.

Create Pipeline

Now we are ready to create the Pipeline.

          make deploy-pipeline

This command creates an AWS CodePipeline pipeline and an AWS CloudWatch event rule.  The event rule will trigger the pipeline when new images are pushed to ECR.

This pipeline has four stages

  • Source: Triggered by a new image uploaded to ECR and tagged ‘latest’.
  • Build: Calls DSSC API to initiate a new scan using the new image in ECR.
  • Approve Deployment: A manual approval step that is automatically set to Approved/Rejected by the DSSC scan.
  • Deployment: A mock deployment of the image.

The pipeline will trigger automatically upon creation.  Now move on to the Verification [link] section.

Verification

Pipeline

We have deployed everything and created our pipeline.  Now, let’s check the status.

The following command displays the status of the different stages in the pipeline:

          make get-pipeline-status

Console output:

          === Getting trend-mirco-dssc-pipeline Status ===
          -------------------------------------
                    GetPipelineState          |
          +--------------------+--------------+
          |  Source            |  Succeeded   |
          |  Build             |  InProgress  |
          |  ApproveDeployment |  None        |
          |  Deploy            |  None        |
          +--------------------+--------------+

Once the pipeline stage ‘ApproveDeployment’ is Succeeded or Failed, run the following command to get a description of the results:

          make get-pipeline-stage-result

Console output:

          === Getting trend-mirco-dssc-pipeline stage 'ApproveDeployment' Summary ===
          There are 0 malware issue(s) and 3 critical issue(s) detected. 
          Full results available at https://<dssc url>/scans/<scan id>

DSSC

You can also check the scan status in DSSC at any time.

The following command accesses DSSC via the API and retrieves the scan status:

          make get-scan-status

Console output:

         === Getting DSSC Scan Status ===
         <scan id>: in-progress

The output is scan_id: status.

The scan is complete when it shows a status of ‘complete-with-findings’ in this example.

You can view the full results in JSON format by running the following:

          make get-scan-results

Adjust Failure Tolerance

Built into the Lambda code is the ability to adjust the error tolerances for both critical and high errors.  We will now relax those settings in order to allow the pipeline to succeed and execute the deployment action with the following:

          make adjust-error-tolerance

Retrigger Pipeline

Now that the error tolerances have been adjusted, retrigger the pipeline with the following:

          make retrigger-pipeline

Once the DSSC status (make get-scan-status) is showing ‘completed-with-findings’, rerun the pipeline status command:

          make get-pipeline-status

Console output:

        === Getting trend-mirco-dssc-pipeline Status ===
         -------------------------------------
         |         GetPipelineState          |
         +--------------------+--------------+
         |  Source            |  Succeeded   |
         |  Build             |  Succeeded   |
         |  ApproveDeployment |  Succeeded   |
         |  Deploy            |  InProgress  |
         +--------------------+--------------+

To view the pipeline stage comments, once again run the stage result command:

          make get-pipeline-stage-result

Console output:

          === Getting trend-mirco-dssc-pipeline stage 'ApproveDeployment' Summary ===

         Full results available at https://<dssc url>/scans/<scan id>

Add Test Malware

Now, let’s intentionally introduce a test virus file and see what happens.  We have already relaxed the errors above to allow the pipeline to succeed.

WARNING: This could trigger the Antivirus Scanner locally if one is running!  Please refer to https://www.eicar.org/?page_id=3950.

          make add-eicar-test-malware

This command copies `./eicar/eicar_test_file.com` to `./sample_apps/src/eicar.com`.

Now run the following command to build and deploy the image to trigger the pipeline.

          make build-and-push-docker-image

Then run

          make get-pipeline-stage-result

Console Output (eventually):

          === Getting trend-mirco-dssc-pipeline stage 'ApproveDeployment' Summary ===
          There are 1 malware issue(s) and 3 critical issue(s) detected. 
          Full results available at https://<dssc url>/scans/<scan id>

Clean-Up

Remember to tear everything down to avoid excessive charges in your AWS account.

          make teardown

Summary

Enterprises can scan containers for security flaws using Trend Micro Deep Security Smart Check.  This blog showed how to implement automated container image security scans as part of a CI/CD pipeline to protect against common vulnerabilities using AWS CodeBuild and AWS CodePipeline.

The code for the examples demonstrated in this port are located on GitHub here [https://github.com/stelligent/aws-trend-micro-dssc].

References

Special Thanks

Special thanks to Jason Dablow, John Haden, Andre Alves, DavidRoth and all the folks at Trend Micro who helped with the creation of this blog and to Jelili Adebello for his excellent blog on Anngine which served as a model and motivation for this one.

Featured Image Jacek Dylag on Unsplash

Stelligent Amazon Pollycast
Voiced by Amazon Polly