Stelligent

Continuous Delivery for AWS Secrets Rotation

  • Paul Duvall

    • One of the biggest challenges with managing usernames, passwords, API keys, and other secrets is balancing the need to make it simple for authorized accounts, databases, and APIs to securely access these secrets while adhering to the principle of least privilege. Simply put, most everyone knows not to store sensitive configuration information in plain text files or environment variables because the values can be exposed to other people or systems. However, it has not always straightforward in how to manage these secrets appropriately with available services and tools.

    This is why AWS introduced Secrets Manager in 2018.  AWS Secrets Manager can easily rotate, manage, and retrieve secrets throughout their lifecycle. It secures, stores, and tightly controls access to tokens, passwords, certificates, API keys, and other secrets while performing automatic secrets rotation, generating random secrets, and providing cross-account access.

    With Secrets Manager, you can safely rotate secrets without anyone ever knowing the values, limit human access to these secrets through fine-grained policies, and centrally and securely audit access to these secrets.

    In this post, you will walk through the steps of launching an AWS CloudFormation stack that creates an AWS CodePipeline pipeline which orchestrates a continuous delivery workflow that provisions a secret in AWS Secrets Manager, creates an Amazon RDS database, and provisions an AWS Lambda function for auto rotation of secrets. You will also run a Python program to securely use these secrets to connect to a MySQL database running on Amazon RDS.

    The links and examples assume you are using the us-east-1 region but – with some minor changes – the code will work in any region that supports the AWS services in the examples.

    I’ve also included a screencast below that provides a walkthrough of the steps covered in this post.

    Setup AWS Cloud9

    In this section, you will configure the Cloud9 IDE to run the commands that deploy this secrets management solution.

    1. Go to the AWS Cloud9 console and select Create environment.
    2. Enter a Name and Description.
    3. Click the Next step button.
    4. Select Create a new instance for environment (EC2).
    5. Select t2.micro.
    6. Leave the Cost-saving setting at the After 30-minute (default) option enabled.
    7. Select Next step.
    8. Review best practices and select Create environment.
    9. Once your Cloud environment has been launched, open a new terminal in Cloud9.

    Solution Architecture

    The figure below shows the key resources generated by the CloudFormation templates along with how you can connect to the MySQL database using secrets managed by AWS Secrets Manager.

    Here are some of the key components that make up this solution:

    Automate RDS Database, Secrets, and Auto Rotation in Lambda

    NOTE: Some of the code in this section is based on Automating Secret Creation in AWS CloudFormation, Cloudonaut Templates, and aws-secrets-manager-rotation-lambdas.

    In this section, your will run a CloudFormation template that creates RDS, Secrets Manager, and Lambda resources that creates a database, automatically generates and securely stores secrets, and rotate these secrets every 30 days.

    Launch the CloudFormation Stack

    To begin, you will zip and sync all of the local source files to S3 so that they can be used by the CloudFormation template. Then, you will launch the CloudFormation stack that provisions a pipeline. This pipeline will run through five stages to create the remaining resources.

    From your AWS Cloud9 IDE terminal, type the following:

    See the gist on github.

    It will take about one minute to launch the CloudFormation stack. Once the stack is CREATE_COMPLETE, click on the stack. Then, go to the Outputs pane and click on the Value for the PipelineUrl Key. This launches an CodePipeline pipeline. It will take another 25-30 minutes for the pipeline to provision a VPC, create and populate a CodeCommit repository, build and deploy a Lambda function that rotates the MySQL admin password, create a MySQL Database in Amazon RDS, create a Secret and Rotation Schedule in AWS Secrets Manager, and link all the relevant services.

    While it is going through the pipeline, let’s go over the key components of the solution.

    Deployment Pipeline

    The CloudFormation stack you launched is based on the ceoa-3-rotation-1-pipeline.yml template. This template provisions several AWS resources including CodeCommit, CodeBuild, CodePipeline, and IAM permissions so that the pipeline properly operates. In this section, you will walkthrough the relevant parts of this template. The end result is a deployment pipeline as shown in the image below.

    CodeCommit

    The ceoa-3-rotation-1-pipeline.yml template provisions a private CodeCommit Git repository using the AWS::CodeCommit::Repository resource. It initially populates this repository using the zip file that you created and passed as a parameter to the CloudFormation stack. The relevant files include the CodeBuild buildspec file and the CloudFormation templates that are used to provision the rest of the solution as part of the deployment pipeline.

    See the gist on github.

    To see the CodeCommit resource provisioning in context with the rest of the solution go to ceoa-3-rotation-1-pipeline.yml.

    CodeBuild

    A CodeBuild project is provisioned so that the Lambda function can be built using a SAM template. This CodeBuild project calls the ceoa-3-buildspec-lambda.yml buildspec file to transform the ceoa-3-rotation-3-lambda-sam.yml SAM template to CloudFormation so that it can be deployed in the next CodePipeline stage.

    See the gist on github.

    The CodeBuild project calls a CodeBuild buildspec file named ceoa-3-buildspec-lambda.yml. A snippet of this buildspec file is shown below. You will note that it transforms a SAM template into a CloudFormation template so that it can be executed in the next action of the pipeline.

    See the gist on github.

    To see the CodeBuild provisioning in context with the rest of the solution go to ceoa-3-rotation-1-pipeline.yml.

    CodePipeline

    A CodePipeline resource is provisioned to model all the stages and actions of the software delivery workflow. These stages include getting all the source files from CodeCommit, creating a VPC and its networking components, building a Lambda function via CodeBuild, deploying the Lambda function through CloudFormation, provisioning an RDS database instance, and creating a secret in AWS Secrets Manager. The snippet below shows the initial provisioning of the AWS::CodePipeline::Pipeline resource.

    See the gist on github.

    To see the CodePipeline provisioning in context with the rest of the solution go to ceoa-3-rotation-1-pipeline.yml.

    Amazon VPC

    With Amazon VPC, you can create an isolated part of the AWS Cloud in which you can define the resources that can run in this network. The ceoa-3-rotation-2-vpc.yml template is launched as a stack from CodePipeline in the second stage. This template defines the provisioning for a VPC, Subnets, Routing, and a NAT Gateway. By creating the VPC, you can put the RDS database and the Lambda function in a private subnet. You create a NAT Gateway so that the Lambda function can access the Secrets Manager endpoint when performing secrets rotation.

    Since there are many resources created in this template, I have provided a snippet showing the provisioning of the Outputs of this stack as these will be referred later when configuring another Cloud9 IDE instance.

    See the gist on github.

    AWS Lambda – Rotate Secrets

    The third stage of the deployment pipeline builds a Python Lambda function provided by AWS through the AWS SAR. It uses CodeBuild to run this action. When the CodeBuild action is called, it calls the ceoa-3-buildspec-lambda.yml which uses the SAM template shown below to build the Lambda function that is available in the AWS SAR for rotating secrets with a MySQL database. The SAM is an open-source framework for building serverless applications. It provides shorthand syntax to express functions, APIs, databases, and event source mappings. It is based on the CloudFormation language using the Fn::Transform intrinsic function.

    You can find this SAM template at ceoa-3-rotation-3-lambda-sam.yml.

    See the gist on github.

    After it builds the Lambda function, in the fourth stage, it deploys it onto the AWS Lambda service using the CloudFormation deploy provider. There are two actions in this DeployLambda stage. The first action transforms the SAM template into CloudFormation. The second action launches the CloudFormation stack from the generated template. The CodePipeline actions to transform/build and deploy the function are shown in the CloudFormation template below.

    See the gist on github.

    To see the SAM template that provisions the Lambda function which rotates the Secret in Secrets Manager, go to ceoa-3-rotation-3-lambda-sam.yml.

    Amazon RDS and AWS Secrets Manager

    The last stage of the deployment pipeline provisions an RDS database and a secret in Secrets Manager.  This stage also attaches the secret to the RDS database instance. The CodePipeline pipeline calls out to the ceoa-3-rotation-4-rds.yml template to create these resources.

    The snippet below shows the provisioning of the AWS::SecretsManager::Secret resource. This resource automatically generates a random secure string that will be used as the master MySQL password in the RDS database instance.

    See the gist on github.

    The snippet below shows the provisioning of the AWS::RDS::DBInstance resource. This resource creates a MySQL RDS database instance. It also attaches the secret from Secrets Manager to this database instance using the AWS::SecretsManager::SecretTargetAttachment resource.

    See the gist on github.

    To see the full working template that provisions an Amazon RDS Database, creates a random secret in Secrets Manager, establishes a rotation schedule, and attaches the secret to RDS, see ceoa-3-rotation-4-rds.yml.

    Configure Cloud9 Environment to Connect to a MySQL database

    Once the rds-secrets-rotation CloudFormation stack is CREATE_COMPLETE and CodePipeline has run through all of its stages, you will create a separate Cloud9 IDE environment that launches into the VPC that was provisioned from the CloudFormation stack from the pipeline. This way, you can securely access your MySQL database from Cloud9 within your VPC.

    1. Create a new Cloud9 environment by going to the AWS Cloud9 console and clicking Create environment.
    2. Enter a Name and Description in the appropriate fields.
    3. Click Next step.
    4. Click Create a new instance for environment (EC2).
    5. Select t2.micro.
    6. Leave the Cost-saving setting at the After 30-minute (default) selected.
    7. Select Network settings (advanced).
    8. From the Network (VPC) dropdown in the Network settings (advanced) section, select the value that is equivalent to the MyVPC Output value of the rds-secrets-rotation-us-east-1-vpc CloudFormation stack that was launched from the pipeline stack.
    9. Click on the public subnet attached to the VPC from the Subnet dropdown. Obtain the public subnet id from the PublicSubnet1 Output value of the rds-secrets-rotation-us-east-1-vpc CloudFormation stack that was launched from the pipeline stack.
    10. Click Next step.
    11. Review best practices and click Create environment.
    12. Once your Cloud9 environment has been launched, open a new terminal in Cloud9.
    13. Run through the Install and Configure Python on Cloud9 instructions to install and configure Python on this newly launched IDE.
    14. Run the command below from your Cloud9 IDE to install the MySQL connector.
    git clone https://github.com/PaulDuvall/aws-encryption-workshop.git
cd ~/environment/aws-encryption-workshop/lesson3-develop
python3 -m pip install mysql-connector --user

    Connect to MySQL database

    In this section, you will connect to the RDS database that was created as part of the deployment pipeline.

    1. Go to Secrets Manager Console.
    2. Select the link for the Secret you just created.
    3. Scroll down and click on the Retrieve secret value button.
    4. Copy the username, host, and password values into a text file.
    5. Test your connection by running this command from your Cloud9 terminal (replacing the host value with the value from Secrets Manager secret) : mysql -u admin -h host -p. For example (your host will be different):
    mysql -u admin -h rotation-instance.b4fdxe6njttc.us-east-1.rds.amazonaws.com -p

    After entering your password, you should see something similar to the following:

    Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 10
Server version: 5.7.22 Source distribution
    Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
    Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
    mysql>
    1. In your AWS Cloud9 IDE, open the ceoa-3-connect-mysql.py file located in your ~/environment/aws-encryption-workshop/lesson3-develop directory.
    2. Modify MySecretName in the ceoa-3-connect-mysql.py file to the generated Secret in Secrets Manager. The name will begin with MyRDSInstanceRotationSecret. Save the file.
    3. Run the Python program from the Cloud9 terminal to view the results.
    cd ~/environment/aws-encryption-workshop/lesson3-develop
python ceoa-3-connect-mysql.py

    When successful, you will receive a response similar to this:

    <mysql.connector.connection.MySQLConnection object at 0x8f53a7bg9800>

    Rotate Password Secret

    In the solution you launched, Secrets Manager was configured to automatically rotate secrets every 30 days. To see how this works, you will force Secrets Manager to automatically rotate the secret and then verify that you remain capable of connecting to the database without manually updating the password or any code.

    1. Go to Secrets Manager Console
    2. Click the link for the Secret created by the CloudFormation stack you launched.
    3. Scroll down and click on the Rotate secret immediately button.
    4. Click the Rotate button.
    5. Click on the Retrieve secret value button again to see the new password value.
    6. Do not make any changes to the Python program and run it again from the Cloud9 terminal to view the results.
    cd ~/environment/aws-encryption-workshop/lesson3-develop
python ceoa-3-connect-mysql.py

    When successful, you will receive a response similar to this:

    <mysql.connector.connection.MySQLConnection object at 0x9f23b7bg4400>

    This means that without needing to make any configuration or code changes, you can connect to the MySQL database with the newly generated password from Secrets Manager. As previously indicated, it will automatically rotate this password every 30 days as well.

    Summary

    In this post you learned how you can use AWS Secrets Manager to automatically generate, rotate, and manage secrets. You also learned how to attach these secrets to an RDS database instance. What’s more, you walked through the steps of automatically the provisioning of all the necessary AWS resources to create this automatic secrets rotation solution. This included services and tools such as AWS CloudFormation, AWS IAM, AWS CodePipeline, AWS CodeCommit, AWS CodeBuild, Amazon VPC (and associated resources such as NAT Gateway), AWS Lambda, AWS SAR,  Amazon RDS, and AWS Secrets Manager.

    By treating this secrets management infrastructure as code and deploying it as part of a Continuous Delivery process, you can make incremental changes and run them through a series of checks before deploying them onto your AWS account(s).

    Appendix

    Here are a few references that helped inform this post and/or may be useful for you when automatically rotating secrets.

    Reset Secrets Manager Rotation

    In some cases, your secrets can get into an inoperable state as a result of a number of factors including forcing secrets rotation multiple times in succession or modifying the database password directly in RDS. In any case, in order to reset the secrets rotation, one remedy is to run the command below which updates the secrets version stage to get it back into a stable state.

    aws secretsmanager list-secret-version-ids --secret-id <name-of-your-secret>
    aws secretsmanager update-secret-version-stage --secret-id <name-of-your-secret> \
    --remove-from-version-id <version-id-from-previous-command> \
    --version-stage AWSPENDING

    Cleanup

    First, delete the Cloud9 IDEs you created. Then, delete the multiple CloudFormation stacks so that you are no longer charged for AWS resources. It can take over 40 minutes for CloudFormation to delete all of the resources.

    Stelligent Amazon Pollycast