Is My Container Image Secure? CI/CD Container Scanning using Trend Micro Deep Security Smart Check and AWS CodePipeline
Many enterprises attempt to drive software development and delivery towards a DevOps mindset. Likewise, organizations struggle with increasing security challenges while adopting these innovative software practices.
Embedding security within the deployment lifecycle is non-negotiable. Therefore, integration of security into CI/CD workflows need to be done cautiously to meet an ever-evolving technology landscape.
DevSecOps is an important concept that automates and integrates security into the software delivery lifecycle. Some challenges exist with adding security controls into containerized solutions.
Furthermore, considering open source containers, many contain known and unknown vulnerabilities. Most organizations struggle to ascertain, with a high level of confidence, how secure these containers are. These tools introduce additional functionality and beneficial features that result in faster solution implementation. In contrast, many are not compliant with organizational security goals. The reality is that failure to design secure deployment pipelines can cost a company millions.
This blog post provides a deployment guide on how to create, launch, and implement a standalone container scanning solution – Trend Micro Deep Security Smart Check – within the AWS ecosystem. It describes how to set up a release pipeline for the automated vulnerability scanning of container images.
This approach uses Trend Micro’s container scanning tool, Deep Security Smart Check (DSSC), as a proof-of-concept and provides examples of how DSSC can be integrated with AWS CodeBuild and AWS CodePipeline.
DSSC is a container compliance platform that ensures the security and stability of container deployments.
DSSC automates the inspection, analysis, and evaluation of images against a compliance checklist that includes items from PCI-DSS, HIPAA, and NIST 800-190. Checks for source code vulnerabilities are performed using the vulnerability database from Snyk. Therefore, it delivers a high confidence in container deployments. Each scan process provides a policy evaluation result for each image which can be used in your CI/CD process to pass or fail the pipeline.
For more detailed information and an overview, please visit Trend Micro Deep Security Smart Check.
High level overview of the pipeline architecture.
AWS Services Used
- AWS CloudFormation
- AWS ECR
- AWS EKS
- AWS ApiGateway
- AWS Lambda
- AWS Parameter Store
- AWS CodePipeline
- AWS CodeBuild
- AWS CloudWatch
This application requires a few steps that must be executed as specified. Please refer to the prerequisites [link added in WP] section prior to running any commands contained in this article to ensure you have the required packages and software installed.
WARNING: You may incur charges with the use of some of these AWS Services, specifically EKS if you leave your cluster up and running over time.
Ensure that the following are installed or configured on your workstation before deploying DSSC.
- Clone this repository [ https://github.com/stelligent/aws-trend-micro-dssc ]
- Configure AWS CLI [https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html]
- Configure eksctl [https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html]
- Configure helm [https://helm.sh/docs/intro/]
Create EKS Cluster
DSSC requires a Kubernetes cluster to run on so we’ll be using AWS EKS.
This can take upwards of 15 minutes to create.
This command uses eksctl to create the cluster and two unmanaged nodes in preparation for the DSSC deployment.
Deploy DSSC to the EKS cluster created previously. Once deployed, the default `password` will be automatically changed to allow API access.
This command uses helm to deploy DSSC to the cluster. Detailed instructions are located at https://github.com/deep-security/smartcheck-helm.
There will be instructions output to the console on where and how to login to the DSSC administrative UI, but it is not required for this example.
This command will create four AWS Parameter Store variables to be used by the pipeline, all prefixed with `/pipeline/example/trendmicro/dssc`.
- username – The DSSC username.
- password – The new DSSC password.
- url – The URL for UI/API access.
- secret – The secret phrase used to sign the X-Scan-Events-Signature header.
Create ECR Repository
We need an image repository to store our sample application image. We will be using AWS ECR.
This command will create an AWS ECR image repository. This repository is used to house the sample app image which DSSC will use when a scan is initiated.
Push Sample Image
We need an image to scan, so a sample has been provided. To build and push the sample image to the ECR repository we just created, run the following.
This will build a Docker image using the application located in ./sample_app and push it to the ECR repository created previously.
DSSC supports webhooks, which is a way for DSSC to communicate with other processes when certain events occur, like when a scan completes. Leveraging a webhook is how we will integrate DSSC into our CI/CD pipeline. This means we need something for DSSC to call when scans are complete, so we will create a webhook for DSSC to call.
This command creates an AWS ApiGateway, AWS Lambda, and an AWS SSM parameter storing the URL to access it.
The Lambda code is in `cloudformation/webhook.yaml’. This code takes the incoming JSON posted to it from DSSC and simply looks at the ‘critical’ and ‘high’ errors counts in the JSON. The code then approves or rejects the ‘Approve Deployment’ pipeline action. The code also checks the X-Scan-Event-Signature header (using the DSSC Secret that was created with the deploy-dssc-ssm command) to determine if the call is a valid one from DSSC. A 401 is returned if the signature is invalid. Refer to Securing Web Hooks for more information.
Now we are ready to create the Pipeline.
This command creates an AWS CodePipeline pipeline and an AWS CloudWatch event rule. The event rule will trigger the pipeline when new images are pushed to ECR.
This pipeline has four stages
- Source: Triggered by a new image uploaded to ECR and tagged ‘latest’.
- Build: Calls DSSC API to initiate a new scan using the new image in ECR.
- Approve Deployment: A manual approval step that is automatically set to Approved/Rejected by the DSSC scan.
- Deployment: A mock deployment of the image.
The pipeline will trigger automatically upon creation. Now move on to the Verification [link] section.
We have deployed everything and created our pipeline. Now, let’s check the status.
The following command displays the status of the different stages in the pipeline:
=== Getting trend-mirco-dssc-pipeline Status === ------------------------------------- GetPipelineState | +--------------------+--------------+ | Source | Succeeded | | Build | InProgress | | ApproveDeployment | None | | Deploy | None | +--------------------+--------------+
Once the pipeline stage ‘ApproveDeployment’ is Succeeded or Failed, run the following command to get a description of the results:
=== Getting trend-mirco-dssc-pipeline stage 'ApproveDeployment' Summary === There are 0 malware issue(s) and 3 critical issue(s) detected. Full results available at https://<dssc url>/scans/<scan id>
You can also check the scan status in DSSC at any time.
The following command accesses DSSC via the API and retrieves the scan status:
=== Getting DSSC Scan Status === <scan id>: in-progress
The output is scan_id: status.
The scan is complete when it shows a status of ‘complete-with-findings’ in this example.
You can view the full results in JSON format by running the following:
Adjust Failure Tolerance
Built into the Lambda code is the ability to adjust the error tolerances for both critical and high errors. We will now relax those settings in order to allow the pipeline to succeed and execute the deployment action with the following:
Now that the error tolerances have been adjusted, retrigger the pipeline with the following:
Once the DSSC status (make get-scan-status) is showing ‘completed-with-findings’, rerun the pipeline status command:
=== Getting trend-mirco-dssc-pipeline Status === ------------------------------------- | GetPipelineState | +--------------------+--------------+ | Source | Succeeded | | Build | Succeeded | | ApproveDeployment | Succeeded | | Deploy | InProgress | +--------------------+--------------+
To view the pipeline stage comments, once again run the stage result command:
=== Getting trend-mirco-dssc-pipeline stage 'ApproveDeployment' Summary === Full results available at https://<dssc url>/scans/<scan id>
Add Test Malware
Now, let’s intentionally introduce a test virus file and see what happens. We have already relaxed the errors above to allow the pipeline to succeed.
WARNING: This could trigger the Antivirus Scanner locally if one is running! Please refer to https://www.eicar.org/?page_id=3950.
This command copies `./eicar/eicar_test_file.com` to `./sample_apps/src/eicar.com`.
Now run the following command to build and deploy the image to trigger the pipeline.
Console Output (eventually):
=== Getting trend-mirco-dssc-pipeline stage 'ApproveDeployment' Summary === There are 1 malware issue(s) and 3 critical issue(s) detected. Full results available at https://<dssc url>/scans/<scan id>
Remember to tear everything down to avoid excessive charges in your AWS account.
Enterprises can scan containers for security flaws using Trend Micro Deep Security Smart Check. This blog showed how to implement automated container image security scans as part of a CI/CD pipeline to protect against common vulnerabilities using AWS CodeBuild and AWS CodePipeline.
The code for the examples demonstrated in this port are located on GitHub here [https://github.com/stelligent/aws-trend-micro-dssc].
- Trend Micro Deep Security Smart Check – https://www.trendmicro.com/en_us/business/products/hybrid-cloud/smart-check-image-scanning.html
- Trend Micro Deep Security Smart Check API – https://automation.deepsecurity.trendmicro.com/article/11_3/smart-check-api-reference?platform=on-premise
- Trend Micro Deep Security Smart Check Deployment Guide – https://deep-security.github.io/smartcheck-docs/admin_docs/admin.html
- AWS CLI – https://docs.aws.amazon.com/cli/latest/reference/index.html#cli-aws
- AWS Cloudformation – https://docs.aws.amazon.com/cloudformation/index.html
- Eicar – https://www.eicar.org/?page_id=3950
Special thanks to Jason Dablow, John Haden, Andre Alves, DavidRoth and all the folks at Trend Micro who helped with the creation of this blog and to Jelili Adebello for his excellent blog on Anngine which served as a model and motivation for this one.
Featured Image Jacek Dylag on Unsplash
Stelligent Amazon Pollycast