Stelligent

Using Dependabot with AWS CodeCommit

  • Stephen Goncher

    • Introduction

    In the DevSecOps world, maintaining code dependencies is key to acquiring the latest security vulnerability updates, bug fixes, and new features.  While locking down to “known-good” revisions of dependencies may avoid potential bugs and incompatibilities during continuous integration, critical updates could be missed.  Dependabot provides an automated solution to dependency management that can be applied to AWS CodeCommit source control repositories.

    Dependabot

    Overview

    Dependabot is an automated dependency management tool that offers a variety of options around source control clients, package managers, and configuration settings.  It became popular enough to be picked up by and directly integrated into GitHub.  While this makes it easiest to use with GitHub repositories, it can also be leveraged for use with other clients.  In fact, Dependabot is the only tool of its kind on the market right now that can actually be configured to work directly with CodeCommit, the Amazon cloud’s source control repository service.  Because CodeCommit is Git-based, this is a logical extension of the GitHub-hosted Dependabot.

    Features

    Dependabot is currently compatible with the following source control clients: GitHub (direct integration), GitLab, Azure DevOps, Atlassian BitBucket, and CodeCommit.  It is expected that support for GitHub Enterprise Edition will be in beta around Quarter 1 of 2021.  

    For package managers, Dependabot currently supports: bundler for Ruby, npm and yarn for Javascript, pip and pipenv for Python, Apache maven, gradle, cargo for Rust, hex for Elixir, composer for PHP, NuGet for .NET, dep and modules for Go, Elm, actions and submodules for Git, Docker, and Terraform.

    When Dependabot detects new updates for dependencies, it creates separate branches and pull requests (PRs) in the source control repository per dependency.  These PRs have several configurable options based on the source control client.  For CodeCommit, these include: author details (name, email, and date), commit message, PR description and name, and the branch name to be created;  the branch name created for the PR defaults to the format of

    dependabot/[PACKAGE_MANAGER]/[BRANCH]/[DEPENDENCY_NAME]-[UPDATE_TO_VERSION_NUMBER]

    So in the case of updating a dependency in the master branch with the pip package manager for pytest, it may look like this:

    dependabot/pip/master/pytest-6.0.1

    Dependabot Flow with CodeCommit

    The above diagram demonstrates the flow of updating dependencies with Dependabot for CodeCommit.  Once authenticated to the dependabot-core GitHub and CodeCommit repositories, Dependabot’s FileFetcher class searches the given CodeCommit branch and directory for dependency files based on the package manager parameter; if no compatible dependency files exist for a chosen package manager, an exception will be thrown.  The FileParser class extracts the dependency list from those files.  Then the UpdateChecker class determines if those dependencies can be safely resolved if not up-to-date; this check is made against the package manager’s source repository, as determined in the dependency file, to find the latest resolvable version.  For those that require it, the FileUpdater class will output an array of updated dependency files.  Lastly, the PullRequestCreator class creates a new branch and associated PR per updated dependency in the CodeCommit repository.

    Dependabot and CodeCommit

    Setup and Prerequisites

    Dependabot-Script

    While Dependabot may seem at first glance to be a GitHub-only tool, the source code for dependabot-core opens up the possibilities to other clients, including CodeCommit.   Configuration begins with the dependabot-script GitHub repository, a demonstration of how to utilize dependabot-core. While dependabot-script does not directly show how to set up a connection to CodeCommit, we can use this as a starting point in conjunction with the core code.

    The basis of the code we’ll be using is housed in the GitHub repository dependabot-codecommit, which was created from a fork of dependabot-script.  The README file instructs us to install the ruby version from .ruby-version using:

     rbenv install

    or your desired method, as long as it matches that version.  Then, to pull down dependencies, run:

     bundle install

    Native Helpers

    While some package manager support is built into dependabot-core, there are others (Terraform, Python, Go Dep and Modules, Elixir, PHP, and Javascript) that require native helpers to be installed.  The README file describes how to do a one time setup, but we can also create a script that allows us to build up the helpers in a repeatable and reliable fashion, without having to directly alter our native PATH environment variable.  

    The below Bash script, dependabot_helpers.sh, performs a fresh install of the native helpers and is expected to live in the root directory of our code.  It is dependent upon the existence of environment variables that will be in our main script; the reason for this is primarily to keep the PATH variable preserved locally, though you can of course export it in your desired fashion.  While the script can be called separately, this blog will incorporate it as part of the update script described in the next section.  Note that the installation of the helpers requires that each of the package manager binaries already exists locally.  A good reference point would be to look at the dependabot-core Dockerfile, as for instance, it is currently necessary to have both Python 2.7 and 3.8 installed via pyenv for full compatibility.

    See the gist on github.

    Permissions and Authentication

    Before we can configure the Dependabot update script, the following will be required:

    See the gist on github.

    The Dependabot Update Script

    Overview

    Within the dependabot-script repo is generic-update-script.rb, which will serve as the template for our aws_codecommit_update_script.rb.  This script executes the Dependabot flow described in the diagram above.

    The generic script contains examples of authenticating to multiple clients but not for CodeCommit;  as such, all source declarations of the providers github, gitlab, and azure can be removed as well any other references to Azure or GitLab.

    There are a few required local environment variables in the script that we will be using or adding.  The rest will be removed as they will either become a CLI option or not necessary at all:

    In addition, it is of course necessary to authenticate to the AWS account and its CodeCommit repository.  While you can optionally configure the credentials to set up the username and password with an AWS access and secret key respectively, our script will assume the usage of  MFA with a tool like aws-vault, providing a more secure method following DevSecOps best practices.

    Configuration

    The following will walk through and break down the aws_codecommit_update_script.rb script that will be based on the above.

    From the top of the script, in addition to the dependabot APIs, ensure that the aws-sdk-codecommit and optimist packages are included in the script:

    See the gist on github.

    Add the Dependabot native helper environment variables:

    See the gist on github.

    Set up the list of possible package managers:

    See the gist on github.

    Next, we’ll add CLI option support via Optimist:

    See the gist on github.

    Then ruby will execute the dependabot_helpers.sh script as a subshell process:

    See the gist on github.

    We’ll next set up the credentials to reach out to the core Dependabot API as well as the parameters necessary to communicate with the CodeCommit repository:

    See the gist on github.

    Lastly, we will loop through the list of package managers that has been passed in, and run through the Dependabot flow of FileFetcher, FileParser, UpdateChecker, FileUpdater, and PullRequestCreator:

    See the gist on github.

    Execution and Expected Results

    To run aws_codecommit_update_script.rb, you must first decide which package managers you want to run against.  Our script is designed to handle one or more package managers using the command line option –package-manager-list or -p, which takes in a space-delimited list of package managers currently supported by Dependabot; or all package managers using the option –all-package-managers or -a, which defaults to false.  It’s important to note that the parameter options are designed such that you must use either the –package-manager-list or –all-package-managers option, but not both.  While the –all-package-managers option can be useful, in general we’ll likely want to run against one or just a few package managers, based on the dependency files that exist in the CodeCommit repository.  The script is designed to fail when attempting to run against a package manager whose dependency files do not exist in the source repository.

    With that in mind, the remaining command line options to consider are: –project-path or -r, which is required and takes in the name of the CodeCommit repository; –directory-path or -d, which accepts the location of the base dependency files and otherwise defaults to the root directory of the CodeCommit repository; and –codecommit-branch or -c, which can be provided with the branch of the CodeCommit repository to run against but otherwise defaults to master.

    Execution Examples

    1. One Package Manager (bundler) with directory path and branch
      1. If compatible bundler file(s) exist(s) in the CodeCommit repository for the project path, directory path, and branch
      2. Then Dependabot creates branches based on the my-feature-branch branch and PRs for each bundler-specific dependency in the /dependencies directory that requires updating

    See the gist on github.

    1. Multiple Package Managers with default directory path and branch
      1. If compatible composer, pip, AND Dockerfiles exist in the CodeCommit repository for the project path, directory path, and branch
      2. Then Dependabot creates branches based on the master branch and PRs for each composer, pip, and docker-specific dependency in the / (root) directory that requires updating.

    See the gist on github.

    1. All Package Managers with defaults
      1. If compatible files exist for ALL package managers in the CodeCommit repository for the project path, directory path, and branch
      2. Then Dependabot creates branches based on the master branch and PRs for each package manager-specific dependency in the / (root) directory that requires updating

    See the gist on github.

    1. Missing Dependency Files
      1. If dependency files are missing or malformed in any way for the specified package manager, branch, or directory
      2. Then the script will fail with appropriate error information.

    Future Improvements and Additions

    One of the ways that is recommended to execute dependabot-core is through the Dockerfile.  Setting up the Dependabot script for CodeCommit in a Docker container would be a strong step towards automating this process.  This would allow quick changes to be made and tested in a reproducible fashion, especially when coupled with AWS Elastic Container Service.

    Both dependabot-core and dependabot-script have preconfigured GitHub Action workflows that demonstrate how to setup Dependabot for Continuous Integration.  These GitHub Actions showcase how CI can build up and run the docker image as well as pick up new dependencies and security updates.  With this in mind, an ideal continuation of this blog post would include setting up a CI/CD pipeline that would be triggered by any updates to dependency files in a CodeCommit repository, as well as running on a daily basis.

    Conclusion

    Dependabot, while now deeply embedded in GitHub, is capable of supporting automated dependency management and security vulnerability updates for CodeCommit through the usage of the open source dependabot-core repository.  Dependabot-script can be extended to fulfill this role and support a pathway to customizable Continuous Integration and Delivery for CodeCommit that might not otherwise be currently possible.   Continuous Feedback through automated PRs and alerts about the latest dependency updates make Dependabot an indispensable tool to help CodeCommit achieve DevSecOps best practices.

    References

    Stelligent Amazon Pollycast