This is an older post. For newer information please check out these posts:

How does compliance work in many organizations? From what I’ve seen, it might look something like this:

‘Internal compliance is conducting an audit in three weeks, we need to make sure we’re compliant with all of our internal policies’. What ensues is a heroic effort of people discovering or creating documents and other artifacts indicating what their policies are along with some checklists stipulating that they have indeed been executed against ‘the software system’ and this often includes signed approvals from bodies like Change Advisory Boards and such. If the scope is at the enterprise level, people might spend late nights and weekends just to prepare themselves for the audit or they might run through mock Q&As. After the audit, the team receives a report based on a snapshot in time and – viewing with a cynical lens – people try to determine who was most at fault for any arising issues. 

Most of the typical activities involved in “compliance” are nothing more than Risk Management Theatre: a mere spectacle that has very little to do with assessing or reducing actual risk. It’s also very often focused on human failure versus systems failure (aka: “someone to blame” without fixing the root cause of the problem). Moreover, the activities are focused on a single point in time; a single event and not considering the reality that software is a dynamic system.

What if there was a way to codify these policies in code and have these policies automatically triggered and continuously based on events? For example, what if there was a way to ensure that all of your EBS volumes and CloudTrail logs are encrypted, and your Lambda function settings are in compliance with your team or organization’s standards?

With AWS Config Rules, these types of active configuration monitoring checks and many more are possible. What’s more, since you can write these rules in code, you can establish a deployment pipeline that applies any changes to these rules against your infrastructure using a fully automated workflow.

With Continuous Compliance, you no longer conduct a separate event for which everyone works overtime and prepares for weeks and, ultimately, neglects to meet your true policy objectives. Instead, your team and organization is continually compliant based on the policies that you have codified in versioned, tested code.

In this post, I will describe the basic components of Continuous Compliance for AWS using Config Rules, AWS CodePipeline, and AWS CloudFormation. There are many levels of policy compliance including your AWS accounts, your compute infrastructure (e.g. configuration of instances and of your deployment pipelines), and the code that creates applications/services. For this post, I will focus only on a slice of compliance for your AWS accounts to give you an idea of what’s possible with Continuous Compliance on AWS.

While compliance regimes like HIPAA, PCI DSS, and SOC are very relevant to Continuous Compliance, it won’t be the focus of this post. That said, all of the same techniques will apply.

Challenges

Without Continuous Compliance, teams cannot scale, cannot properly assess their compliance, spend too much time and resources preparing for audits, conduct massively manual efforts, and release features to end users less often. These challenges are described in some more detail here:

  • Inability to Scale – Inability to adapt to speed of change at scale. In the cloud, large scale can be achieved instantly and manual compliance cannot adapt to this type of scale.
  • Inability to Properly Assess  – Inability to quickly and accurately assess infrastructure state changes. Even before the cloud, infrastructure changes were regularly occurring that might violate an organization’s policies without the knowledge of those who managed this infrastructure. With the ability to make changes through APIs via the cloud, these changes are exponential making it extremely difficult to assess without a systematic and automated approach
  • Compliance “Events” – Monumental efforts to comply with internal and external audits. Martin Fowler is fond of saying that Continuous Integration makes integration a “nonevent”. Without Continuous Compliance, audits and other reviews become an event that takes up a lot of time and resources.
  • Manual Efforts – Repeatedly performing the same manual efforts to achieve “compliance” with organizational policies. Without continuous compliance, your organization is always “noncompliant” because of the state of change and the inability to assess compliance without a fully automated approach.
  • Less Effective Feedback; Release Less Often – Reduce product teams’ ability to increase the speed of change to their end users. Because compliance is either performed fully or partially manually, there’s often a separate event in the release lifecycle in which a person or team audits the software system. Since there’s minimal integrity in a manual process, the results are suspect and often do not reduce risk. What’s worse, because it takes so much time, teams will forego these checks exposing themselves to greater risk. This approach reduces effective feedback from end users and increases risk when software systems are noncompliant to the organization’s policies.

Solution

While there are several ways to achieve Continuous Compliance on AWS, the solution on which I will focus is one that uses AWS native services. For this particular solution, I’m using AWS Config and Config Rules, AWS CodePipeline, AWS Lambda, and AWS CloudFormation.

  • AWS Config (and Config Rules) – a fully-managed service for tracking AWS resource inventory and changes. With Config Rules, you can ensure existing and new AWS resources conform to your company’s security policies and best practices.
  • AWS CodePipeline – a fully-managed service for releasing software using Continuous Delivery. You can use CodePipeline to orchestrate AWS Config and Config Rules changes to your AWS account.
  • AWS CodeBuild – a fully-managed service for building and testing code. You can use CodeBuild to build and deploy Config Rules artifacts to your AWS account.
  • AWS Lambda – a fully-managed service for running your code in response to events. You can use Lambda to write custom Config Rules in your preferred programming language.
  • AWS CloudFormation – a tool for creating and managing AWS resources with templates. You can automate the entire AWS Config solution in code to achieve continuous compliance.

I describe how these services are used in the following sections.

AWS Config

AWS Config is a fully managed service that provides a detailed view of the resources associated with your AWS account, including how they are configured, how they are related to one another, and how the configurations and their relationships have changed over time [Source].

AWS Config uses Amazon Simple Notification Service (SNS) to stream configuration changes and notifications. Config uses Amazon Simple Storage Service (S3) to store configuration items; they are delivered in a configuration stream. Permissions to the AWS Config service is managed through AWS IAM. All AWS Config API calls are logged via CloudTrail.

When setting up AWS Config, you can configure settings such as turning on/off Config recording, which AWS resources are recorded, data retention, S3 bucket, SNS Topic, and IAM Role. These settings from the AWS console are shown below.

 

Once AWS Config is enabled, you can view its dashboard as shown here.

There are lots of ways to configure, interface with, and get notified of anomalies including its API, SDK, Console, and CloudFormation.

Config Rules

An AWS Config rule represents your desired configuration settings for specific AWS resources or for an entire AWS account. If a resource violates a rule, AWS Config flags the resource and the rule as noncompliant, and AWS Config notifies you through Amazon SNS [Source].

Managed Config Rules

AWS Config provides customizable, predefined rules to help you get started. At the time of this post, there are 82 managed Config Rules for Compute, Database, Management Tools, Network & Content Delivery, Security, Identity & Compliance, and Storage. More rules are regularly added. Here are a few examples:

  • vpc-flow-logs-enabled – Checks whether Amazon Virtual Private Cloud flow logs are found and enabled for Amazon VPC.
  • iam-root-access-key-check – Checks whether the root user access key is available. The rule is compliant if the user access key does not exist.
  • iam-user-mfa-enabled – Checks whether the AWS Identity and Access Management users have multi-factor authentication (MFA) enabled.
  • s3-bucket-public-read-prohibited – Checks that your Amazon S3 buckets do not allow public read access.
  • rds-multi-az-support – Checks whether high availability is enabled for your RDS DB instances. In a Multi-AZ deployment, Amazon RDS automatically provisions and maintains a synchronous standby replica in a different Availability Zone.

Just like AWS Config itself, you can administer these managed rules through the API, SDK, CloudFormation, and the AWS Console – as shown below.

 

Once the Config Rule has been added, you can view compliance for each rule on the Config Rules dashboard  – as shown here.

Later, you will see how to automate the provisioning of Config Rules using AWS CloudFormation.

Custom Config Rules

In addition to the managed config rules, you can write your own custom config rules that are specific to your team and/or organization using an AWS Lambda function that you write in a high-level programming language such as Node.js, Python, or Go. Consider using the Rule Development Kit (RDK) when developing your own Config Rules. The RDK creates and atomically deploys the Config, AWS Lambda, and IAM resources necessary for your custom Config use case. There’s also a curated AWS Community repository of custom config rules at https://github.com/awslabs/aws-config-rules.

To add a custom config rule, from the AWS Config Console:

  1. Go to Rules
  2. Click Add rule
  3. Click Add custom rule
  4. Enter a Name
  5. Enter a Description
  6. Enter an AWS Lambda function ARN (if you haven’t done so already, create a Lambda function)
  7. Configure a Trigger (Configuration changes and/or Periodic)
  8. Enter Rule parameters

AWS CodePipeline

AWS CodePipeline is a fully managed service that orchestrates a release workflow for continuous integration, continuous delivery, and continuous deployment. With CodePipeline, you define a series of stages composed of actions that perform tasks in a release process from a code commit all the way to production. CodePipeline provides a part of the continuous capability that makes up Continuous Compliance.

In the screenshot here, you see a three stage pipeline that deploys Config Rules to a region within an AWS account.

 

 

In the Source action, the pipeline obtains the source files from GitHub and stores them in a secure location in S3. Then, it uses the CodeBuild service to build and store the Config Rules artifacts. Finally, it deploys the Config Rules. In a more comprehensive deployment pipeline, you might include static analysis and tests to verify that the CloudFormation template configuration and other code meets your team’s standards. In the next section, you will see how you can automate the provisioning of your CodePipeline and related resources as well using AWS CloudFormation.

AWS CloudFormation

AWS CloudFormation provides a common language for you to describe and provision all the infrastructure resources in your cloud environment. CloudFormation allows you to use a simple text file to model and provision, in an automated and secure manner, all the resources needed for your applications across all regions and accounts. This file serves as the single source of truth for your cloud environment [Source].

AWS CloudFormation provides support for the following resources relevant to this solution:

Examples

In the following sections, you will see examples of using AWS CloudFormation to automatically provision Config Configuration Recorder, CodePipeline Pipeline, CodeBuild Project, and Config Rule resources. You will see how all of it is defined in configuration code that can be versioned, tested, and made a part of their own continuous delivery processes as well.

Config Configuration Recorder

To begin recording configuration items using AWS Config, you must begin recording these events. To do this in CloudFormation, you use the AWS::Config::ConfigurationRecorder to begin this recording. An architecture diagram and CloudFormation template snippet is demonstrated below.

  ConfigRecorder:
    Type: AWS::Config::ConfigurationRecorder
    Properties:
      Name: aws-config-recorder
      RecordingGroup:
        AllSupported: true
        IncludeGlobalResourceTypes: true
      RoleARN: !GetAtt 'ConfigRole.Arn'

CodePipeline Pipeline

An architecture diagram is shown below that illustrates the solution that defines a deployment pipeline for deploying AWS Config Rules. It includes a CodePipeline pipeline, KMS key, IAM permissions, S3 bucket & policy, and multiple CodeBuild projects.

 

The snippet below shows the AWS::CodePipeline::Pipeline definition in the CloudFormation template. This defines all the CodePipeline stages and actions for this deployment pipeline.

  Pipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      RoleArn: !GetAtt CodePipelineRole.Arn
      Name: !Sub ${AWS::StackName}-AWS-Config-Pipeline
      Stages:
      - Name: Source
        Actions:
        - Name: Source
          InputArtifacts: []
          OutputArtifacts:
          - Name: SourceOutput
          ActionTypeId:
            Category: Source
            Owner: ThirdParty
            Version: '1'
            Provider: GitHub
          Configuration:
            Owner: {"Fn::Select": ["0", {"Fn::Split":["/", {"Ref": "SourceRepo"}]}]}
            Repo: {"Fn::Select": ["1", {"Fn::Split":["/", {"Ref": "SourceRepo"}]}]}
            Branch: !Ref SourceBranch
            OAuthToken: !Ref GitHubToken
          RunOrder: 10
      - Name: Build
        Actions:
        - Name: Artifact
          ActionTypeId:
            Category: Build
            Owner: AWS
            Version: '1'
            Provider: CodeBuild
          InputArtifacts:
          - Name: SourceOutput
          OutputArtifacts:
          - Name: ArtifactOutput
          Configuration:
            ProjectName: !Ref CodeBuildArtifact
          RunOrder: 10
      - Name: Deploy
        Actions:
        - Name: DeployConfigRules
          ActionTypeId:
            Category: Build
            Owner: AWS
            Version: '1'
            Provider: CodeBuild
          InputArtifacts:
          - Name: ArtifactOutput
          OutputArtifacts:
          - Name: DeployConfigRulesOutput
          Configuration:
            ProjectName: !Ref CodeBuildConfigRules
          RunOrder: 10
CodeBuild Project

As part of the same pipeline.yml file, several CodeBuild projects are defined in the CloudFormation template that build and deploy Config Rules and other resources. This snippet shows the CodeBuild configuration including encryption settings, IAM role, cache, container image and type, and S3 bucket location. Most notably, however, is the buildspec.yml it calls. This buildspec file performs the calls to deploy multiple CloudFormation templates. For example, you could configure it to call all of the Managed Config Rules you wish to use within a region of your AWS account.

  CodeBuildConfigRules:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: !Sub ${AWS::StackName}-pipeline-configrules
      EncryptionKey: !GetAtt CodePipelineKey.Arn
      Description: Package and deploy Config rules
      ServiceRole: !GetAtt CodeBuildRole.Arn
      Cache:
        Type: S3
        Location: !Sub ${Bucket}/${AWS::StackName}/artifact
      Artifacts:
        Type: CODEPIPELINE
      Environment:
        Type: !Ref BuildType
        ComputeType: !Ref BuildComputeType
        Image: !Sub ${BuildImage}
        EnvironmentVariables:
          - Name: S3_BUCKET
            Value: !Sub ${Bucket}/${AWS::StackName}/config_rules
      Source:
        Type: CODEPIPELINE
        BuildSpec: buildspec.yml
      TimeoutInMinutes: 30
buildspec.yml

While the snippet of the buildspec.yml below is not part of the CloudFormation template, it’s a key part of the solution and is referenced in the CodeBuild Project resource of the CloudFormation template.

version: 0.2

env:
  variables:
    CFN_TEMPLATES: |
      config-rules-s3
      config-rules-security
phases:
  build:
    commands:
      - echo Build started on `date`
      - |
        for cfn_template in $CFN_TEMPLATES; do
          echo "Deploying CloudFormation template $cfn_template"
          aws cloudformation deploy --template-file $cfn_template.yml --stack-name $cfn_template --capabilities CAPABILITY_NAMED_IAM
        done
  post_build:
    commands:
      - echo Build completed on `date`
artifacts:
  files:
    - '**/*'

Config Rule

A snippet from a CloudFormation template that generates AWS Config Rules is shown below. The key attributes are the ConfigRuleName and the Source. The SourceIdentifier is a keyword to identify a specific managed rule considering the Owner is AWS.

  AWSConfigRuleForS3PublicRead:
    Type: AWS::Config::ConfigRule
    Properties:
      ConfigRuleName: !Ref 'ConfigRuleNameA'
      Description: Checks that your Amazon S3 buckets do not allow public read access.
      Scope:
        ComplianceResourceTypes:
          - AWS::S3::Bucket
      Source:
        Owner: AWS
        SourceIdentifier: S3_BUCKET_PUBLIC_READ_PROHIBITED

ProTips

Here are several tips to consider when applying Continuous Compliance practices:

  1. Configure AWS Config and related resources in a CloudFormation template that is committed to a source control repository such as GitHub or CodeCommit
  2. Create a deployment pipeline using AWS CodePipeline to deploy the managed and custom config rules to your AWS account(s). Define the CodePipeline and associated resources within a CloudFormation template
  3. While custom Config Rules can be written directly as Lambda within a CloudFormation template, you should extract them into separate files  so that proper linting and unit testing can be performed on these functions
  4. Configure SNS for notifications

Other AWS Services for Continuous Compliance

Here are some of the other AWS services you might consider using as part of Continuous Compliance on AWS:

  • AWS Control Tower – Set up and govern a secure, compliant, multi-account environment
  • AWS Security Hub – Unified security and compliance center. In addition to several of the security services, it’s a portal for compliance services such as a AWS Config and provides insights, findings, and a dashboard to view security and compliance activities for your AWS account.
  • Amazon Macie – Uses machine learning to automatically discover, classify, and protect sensitive data in AWS. You can extend Macie to ensure data changes are always in compliance with your organization’s policies
  • Amazon Inspector – Automated security assessment service to help improve the security and compliance of applications deployed on AWS.
  • Amazon GuardDuty – Intelligent threat detection and continuous monitoring to protect your AWS accounts and workloads.
  • AWS Trusted Advisor – Optimize Performance and Security
  • Amazon CloudWatch Events – Amazon CloudWatch Events delivers a near real-time stream of system events that describe changes in AWS resources. You can use Amazon CloudWatch Events to detect and react to changes in the status of AWS Config events.

Costs

Here’s a summary of some of the AWS costs you will incur with a similar setup:

AWS Config

You pay $0.003 per configuration item recorded in your AWS account per AWS Region. You are charged only once for recording the configuration item. Here’s the pricing for AWS Config Rules:

First 10 rules $2.00 per active AWS Config rule per Region per month
Next 40 rules (11-50 rules)
 $1.50 per active AWS Config rule per Region per month
51 rules and more $1.00 per active AWS Config rule per Region per month

 

AWS Config Rules also provides a button to click that provides a cost estimate based on your current rules in a given region. For more information, see AWS Config Pricing.

AWS CodePipeline

With AWS CodePipeline, there are no upfront fees or commitments. You pay only for what you use. AWS CodePipeline costs $1.00 per active pipeline (An active pipeline is a pipeline that has existed for more than 30 days and has at least one code change that runs through it during the month. There is no charge for pipelines that have no new code changes running through them during the month. An active pipeline is not prorated for partial months) per month. To encourage experimentation, pipelines are free for the first 30 days after creation. For more information, see AWS CodePipeline Pricing.

AWS Lambda

If you’re simply experimenting with Config Rules using Lambda, you’ll unlikely get charged for any of your Lambda usage as it’ll fall within the free tier pricing model. With AWS Lambda, you pay only for what you use. You are charged based on the number of requests for your functions and the duration, the time it takes for your code to execute. For more information, see AWS Lambda Pricing.

Additional Resources

Summary

In this post, you learned that you do not need to rely on humans to follow outdated compliance documents. Instead, people can codify your teams’ policies into executable rules that can be versioned, tested, and run as part of a continuous delivery pipeline. You can use tools such as AWS Config Rules, AWS CodePipeline, and AWS Lambda – among others – to ensure that your AWS accounts are always in compliance with your organization’s policies so that you can deliver features to end users with speed and confidence.

Stelligent Amazon Pollycast
Voiced by Amazon Polly