Introduction

In the DevSecOps world, maintaining code dependencies is key to acquiring the latest security vulnerability updates, bug fixes, and new features.  While locking down to “known-good” revisions of dependencies may avoid potential bugs and incompatibilities during continuous integration, critical updates could be missed.  Dependabot provides an automated solution to dependency management that can be applied to AWS CodeCommit source control repositories.

Dependabot

Overview

Dependabot is an automated dependency management tool that offers a variety of options around source control clients, package managers, and configuration settings.  It became popular enough to be picked up by and directly integrated into GitHub.  While this makes it easiest to use with GitHub repositories, it can also be leveraged for use with other clients.  In fact, Dependabot is the only tool of its kind on the market right now that can actually be configured to work directly with CodeCommit, the Amazon cloud’s source control repository service.  Because CodeCommit is Git-based, this is a logical extension of the GitHub-hosted Dependabot.

Features

Dependabot is currently compatible with the following source control clients: GitHub (direct integration), GitLab, Azure DevOps, Atlassian BitBucket, and CodeCommit.  It is expected that support for GitHub Enterprise Edition will be in beta around Quarter 1 of 2021.  

For package managers, Dependabot currently supports: bundler for Ruby, npm and yarn for Javascript, pip and pipenv for Python, Apache maven, gradle, cargo for Rust, hex for Elixir, composer for PHP, NuGet for .NET, dep and modules for Go, Elm, actions and submodules for Git, Docker, and Terraform.

When Dependabot detects new updates for dependencies, it creates separate branches and pull requests (PRs) in the source control repository per dependency.  These PRs have several configurable options based on the source control client.  For CodeCommit, these include: author details (name, email, and date), commit message, PR description and name, and the branch name to be created;  the branch name created for the PR defaults to the format of

dependabot/[PACKAGE_MANAGER]/[BRANCH]/[DEPENDENCY_NAME]-[UPDATE_TO_VERSION_NUMBER]

So in the case of updating a dependency in the master branch with the pip package manager for pytest, it may look like this:

dependabot/pip/master/pytest-6.0.1

Dependabot Flow with CodeCommit

The above diagram demonstrates the flow of updating dependencies with Dependabot for CodeCommit.  Once authenticated to the dependabot-core GitHub and CodeCommit repositories, Dependabot’s FileFetcher class searches the given CodeCommit branch and directory for dependency files based on the package manager parameter; if no compatible dependency files exist for a chosen package manager, an exception will be thrown.  The FileParser class extracts the dependency list from those files.  Then the UpdateChecker class determines if those dependencies can be safely resolved if not up-to-date; this check is made against the package manager’s source repository, as determined in the dependency file, to find the latest resolvable version.  For those that require it, the FileUpdater class will output an array of updated dependency files.  Lastly, the PullRequestCreator class creates a new branch and associated PR per updated dependency in the CodeCommit repository.

Dependabot and CodeCommit

Setup and Prerequisites

Dependabot-Script

While Dependabot may seem at first glance to be a GitHub-only tool, the source code for dependabot-core opens up the possibilities to other clients, including CodeCommit.   Configuration begins with the dependabot-script GitHub repository, a demonstration of how to utilize dependabot-core. While dependabot-script does not directly show how to set up a connection to CodeCommit, we can use this as a starting point in conjunction with the core code.

The basis of the code we’ll be using is housed in the GitHub repository dependabot-codecommit, which was created from a fork of dependabot-script.  The README file instructs us to install the ruby version from .ruby-version using:

 rbenv install

or your desired method, as long as it matches that version.  Then, to pull down dependencies, run:

 bundle install

Native Helpers

While some package manager support is built into dependabot-core, there are others (Terraform, Python, Go Dep and Modules, Elixir, PHP, and Javascript) that require native helpers to be installed.  The README file describes how to do a one time setup, but we can also create a script that allows us to build up the helpers in a repeatable and reliable fashion, without having to directly alter our native PATH environment variable.  

The below Bash script, dependabot_helpers.sh, performs a fresh install of the native helpers and is expected to live in the root directory of our code.  It is dependent upon the existence of environment variables that will be in our main script; the reason for this is primarily to keep the PATH variable preserved locally, though you can of course export it in your desired fashion.  While the script can be called separately, this blog will incorporate it as part of the update script described in the next section.  Note that the installation of the helpers requires that each of the package manager binaries already exists locally.  A good reference point would be to look at the dependabot-core Dockerfile, as for instance, it is currently necessary to have both Python 2.7 and 3.8 installed via pyenv for full compatibility.

Permissions and Authentication

Before we can configure the Dependabot update script, the following will be required:

  • Within a given AWS Account:
    • A way to authenticate to the account, through MFA or another secure means (this is advised) or with an authentication key and secret.  
    • The authenticated user or role will need, at minimum, the following CodeCommit permissions for an IAM policy that should be restricted to the relevant repositories:
      • ListPullRequests
      • BatchGetCommits
      • GetBranch
      • GetCommit
      • GetFile
      • GetFolder
      • GetPullRequest
      • GetRepository
      • CreateBranch
      • CreateCommit
      • CreatePullRequest
    • Example Policy:
    • A CodeCommit repository that includes dependency files supported by Dependabot.  Supported file types per package manager include:
      • Bundler: Gemfile, .gemspec file, gems.rb
      • Cargo: Cargo.toml
      • Composer: composer.json
      • Go Dep: Gopkg.toml. Gopkg.lock
      • Go Modules: go.mod
      • Docker: Dockerfile
      • Elm: elm-package.json, elm.json
      • Git Submodules: .gitmodules
      • GitHub Actions: .github/workflows directory with YAML files
      • Gradle: build.gradle
      • Hex: mix.exs
      • Maven: pom.xml
      • NPM and Yarn: package.json
      • Nuget: .(cs|vb|fb)proj file, packages.config
      • Python: requirements.txt, setup.py, pyproject.toml
      • Terraform: .tf(vars) file
  • A GitHub personal access token with full repo access permissions to communicate with  public and private repositories

The Dependabot Update Script

Overview

Within the dependabot-script repo is generic-update-script.rb, which will serve as the template for our aws_codecommit_update_script.rb.  This script executes the Dependabot flow described in the diagram above.

The generic script contains examples of authenticating to multiple clients but not for CodeCommit;  as such, all source declarations of the providers github, gitlab, and azure can be removed as well any other references to Azure or GitLab.

There are a few required local environment variables in the script that we will be using or adding.  The rest will be removed as they will either become a CLI option or not necessary at all:

  • GITHUB_ACCESS_TOKEN
    • This must be configured in order to communicate with the dependabot-core API
  • AWS_REGION
    • For CodeCommit, this will act as the hostname parameter for the Dependabot source.

In addition, it is of course necessary to authenticate to the AWS account and its CodeCommit repository.  While you can optionally configure the credentials to set up the username and password with an AWS access and secret key respectively, our script will assume the usage of  MFA with a tool like aws-vault, providing a more secure method following DevSecOps best practices.

Configuration

The following will walk through and break down the aws_codecommit_update_script.rb script that will be based on the above.

From the top of the script, in addition to the dependabot APIs, ensure that the aws-sdk-codecommit and optimist packages are included in the script:

Add the Dependabot native helper environment variables:

Set up the list of possible package managers:

Next, we’ll add CLI option support via Optimist:

Then ruby will execute the dependabot_helpers.sh script as a subshell process:

We’ll next set up the credentials to reach out to the core Dependabot API as well as the parameters necessary to communicate with the CodeCommit repository:

Lastly, we will loop through the list of package managers that has been passed in, and run through the Dependabot flow of FileFetcher, FileParser, UpdateChecker, FileUpdater, and PullRequestCreator:

Execution and Expected Results

To run aws_codecommit_update_script.rb, you must first decide which package managers you want to run against.  Our script is designed to handle one or more package managers using the command line option –package-manager-list or -p, which takes in a space-delimited list of package managers currently supported by Dependabot; or all package managers using the option –all-package-managers or -a, which defaults to false.  It’s important to note that the parameter options are designed such that you must use either the –package-manager-list or –all-package-managers option, but not both.  While the –all-package-managers option can be useful, in general we’ll likely want to run against one or just a few package managers, based on the dependency files that exist in the CodeCommit repository.  The script is designed to fail when attempting to run against a package manager whose dependency files do not exist in the source repository.

With that in mind, the remaining command line options to consider are: –project-path or -r, which is required and takes in the name of the CodeCommit repository; –directory-path or -d, which accepts the location of the base dependency files and otherwise defaults to the root directory of the CodeCommit repository; and –codecommit-branch or -c, which can be provided with the branch of the CodeCommit repository to run against but otherwise defaults to master.

Execution Examples

  1. One Package Manager (bundler) with directory path and branch
    1. If compatible bundler file(s) exist(s) in the CodeCommit repository for the project path, directory path, and branch
    2. Then Dependabot creates branches based on the my-feature-branch branch and PRs for each bundler-specific dependency in the /dependencies directory that requires updating
  1. Multiple Package Managers with default directory path and branch
    1. If compatible composer, pip, AND Dockerfiles exist in the CodeCommit repository for the project path, directory path, and branch
    2. Then Dependabot creates branches based on the master branch and PRs for each composer, pip, and docker-specific dependency in the / (root) directory that requires updating.

  1. All Package Managers with defaults
    1. If compatible files exist for ALL package managers in the CodeCommit repository for the project path, directory path, and branch
    2. Then Dependabot creates branches based on the master branch and PRs for each package manager-specific dependency in the / (root) directory that requires updating
  1. Missing Dependency Files
    1. If dependency files are missing or malformed in any way for the specified package manager, branch, or directory
    2. Then the script will fail with appropriate error information.

Future Improvements and Additions

One of the ways that is recommended to execute dependabot-core is through the Dockerfile.  Setting up the Dependabot script for CodeCommit in a Docker container would be a strong step towards automating this process.  This would allow quick changes to be made and tested in a reproducible fashion, especially when coupled with AWS Elastic Container Service.

Both dependabot-core and dependabot-script have preconfigured GitHub Action workflows that demonstrate how to setup Dependabot for Continuous Integration.  These GitHub Actions showcase how CI can build up and run the docker image as well as pick up new dependencies and security updates.  With this in mind, an ideal continuation of this blog post would include setting up a CI/CD pipeline that would be triggered by any updates to dependency files in a CodeCommit repository, as well as running on a daily basis.

Conclusion

Dependabot, while now deeply embedded in GitHub, is capable of supporting automated dependency management and security vulnerability updates for CodeCommit through the usage of the open source dependabot-core repository.  Dependabot-script can be extended to fulfill this role and support a pathway to customizable Continuous Integration and Delivery for CodeCommit that might not otherwise be currently possible.   Continuous Feedback through automated PRs and alerts about the latest dependency updates make Dependabot an indispensable tool to help CodeCommit achieve DevSecOps best practices.

References

Stelligent Amazon Pollycast
Voiced by Amazon Polly