Create extensions for the mu DevOps on AWS framework
Stelligent mu is an open source DevOps on AWS framework that automatically provisions environments, pipelines, and services in a few lines of configuration code. What’s more, you can extend mu’s core capabilities using extensions that you write in AWS CloudFormation. In this post, you’ll learn the process for creating and testing a mu extension. mu provisions environments. Environments consist of the compute, database, and storage resources required to deploy an application/service. This includes the Amazon VPC network and its associated components. If necessary, it provisions the Amazon RDS database. Environments also compose the Amazon EC2 compute infrastructure – which might include Auto Scaling, load balancing, Amazon ECS, Amazon ECR, AWS Fargate, ECS clusters and tasks, EC2 instance(s), and other resources on which the application/service will run.
mu also provisions deployment pipelines. This consists of source, build, test, approval, and deploy actions and its associated stages in AWS CodePipeline. It provisions all the AWS CodePipeline, AWS CodeCommit, AWS CodeBuild, and AWS CodeDeploy resources. CodePipeline defines the workflow that automates the process from when an engineer commits code to a version-control repository until the application/service is deployed to production on the environment provisioned by mu. In addition to CodeCommit, mu also supports GitHub and S3 as source providers..
With the environments, pipelines, and services provisioned, you’ll likely want to extend mu’s capabilities. This could include myriad options such as Compute, Storage, Database, Machine Learning, Analytics, Security, Identity, and Compliance, Mobile Services, Application Integration, and Internet of Things.
In this post, you’ll learn how to extend mu to provision Amazon GuardDuty resources along with the environment, pipeline, and service provisioned by mu. Amazon
GuardDuty is a managed threat detection service that continuously monitors for malicious or unauthorized behavior to help you protect your AWS accounts and workloads. It monitors for activity such as unusual API calls or potentially unauthorized deployments that indicate a possible account compromise. GuardDuty also detects potentially compromised instances or reconnaissance by attackers.
While this post focuses on writing a mu extension for a single AWS product, the steps I walkthrough will be similar when you’re developing other mu extensions.
Create a New GitHub Repository
NOTE: While you can use any version-control system with mu, in this post, I’m using GitHub for all of the examples.
To begin, in the upper right hand corner of any GitHub page, you can select the + sign to add a new GitHub repository. You can also go directly to https://github.com/new/. In creating the new repo, use the convention mu-extension. extension is the name of the product for which you’re writing the extension. It could be the name of an AWS or non-AWS product or it could be a name you’re selecting that represents a collection of AWS products you’re automating. In GitHub, select whether it’s a public or private repo. Select the Initialize this repository with a README checkbox and select a license from the Add a license drop down. Optionally, you can choose to create a .gitignore file. Using this file, you can determine which file types are ignored when committing files to the repository. Click Create repository. A new repository will be generated by GitHub.
At this point, the file structure should look similar to what you see below.
.gitignore LICENSE README.md
For more information, see https://help.github.com/articles/create-a-repo/.
Define and Test CloudFormation Template
In my case, I was writing an extension for a single AWS product: GuardDuty. I looked up examples on the AWS CloudFormation documentation portal and found documentation for these resources: AWS::GuardDuty::Detector, AWS::GuardDuty::ThreatIntelSet and AWS::GuardDuty::IPSet. Based on these examples, I created a CloudFormation template that looks like this:
Parameters: ThreatIntelSetUrl: Type: String Description: Url for ThreatIntelSet IpSetUrl: Type: String Description: Url for IpSet Resources: GuardDutyDetector: Type: AWS::GuardDuty::Detector Properties: Enable: true GuardDutyThreatIntelSet: Type: AWS::GuardDuty::ThreatIntelSet Properties: Activate: true DetectorId: Ref: GuardDutyDetector Format: TXT Location: Ref: ThreatIntelSetUrl GuardDutyIpSet: Type: AWS::GuardDuty::IPSet Properties: Activate: true DetectorId: Ref: GuardDutyDetector Format: TXT Location: Ref: IpSetUrl Outputs: GuardDutyDetector: Description: GuardDuty Detector ID Value: Ref: GuardDutyDetector
You will save the file with the name of an existing mu asset (i.e. a CloudFormation file) that this configuration will extend. You can find a list of these CloudFormation files at https://github.com/stelligent/mu/tree/master/templates/assets. In my case, I chose to extend vpc.yml so I created a new vpc.yml file and committed it to my mu-guardduty repository. Once I include the extension in the mu.yml file of my application, it will merge the contents of the mu-guardduty vpc.yml with the vpc.yml defined in the mu assets folder and run as part of the mu core invocation.
You will want to update the README to give users an idea of the costs of the resources you’re automating. In the case of Amazon GuardDuty, I took the first paragraph from the Amazon GuardDuty pricing page and added it to the README: “Amazon GuardDuty is priced along two dimensions. The dimensions are based on the quantity of AWS CloudTrail Events analyzed (per 1,000,000 events) and the volume of Amazon VPC Flow Log and DNS Log data analyzed (per GB).”
You’ll also want to describe the basic usage including its parameters along with an example of your mu extension in the README.
Create mu-extension.yml
Create a new file called mu-extension.yml. In it, you define the name and the version. You can optionally specify a value for templateUpdateMode. Default is retain. The other option is replace. You use replace when you want to avoid trying to “merge” your CloudFormation with the template mu has and you’d rather just completely replace the template with CloudFormation in your extension.
name: mu-guardduty version: '0.7'
Create GitHub Release Artifacts
From GitHub, navigate to the main page of your mu-extension repository. Under your repository name, click Releases. Click Draft a new release. Type a version number for your release. Select a branch that contains the project you want to release. Type a title and description that describes your release. If the release is unstable, select This is a pre-release to notify users that it’s not ready for production. If you’re ready to publicize your release, click Publish release. Otherwise, click Save draft to work on it later.
Once you publish the release, right click on the zip file that was created to copy the link. Use this URL in your mu.yml file. See Figure 1.
Figure 1 – Getting the URL for the GitHub Release artifacts
For more information, see the document for which these instructions are based: https://help.github.com/articles/creating-releases/.
Get a Generic mu Example Working
To test the functionality of your mu extension, you should get a generic mu example first and then add the mu extension to your mu.yml file. When I wrote the mu-guardduty extension, I chose to use the ecs-fargate example as my starting point. I downloaded all the files from the example, created a new GitHub repository and then committed the files from the ecs-fargate example directory to my new repository. Then, I performed a git clone of the repository locally in my AWS Cloud9 environment and ran this command (see GitHub Personal Access Token section to learn how to generated a GitHub token):
mu pipeline up -t [GitHub Personal Access Token]
Once the environment, pipeline, and service are successfully up and running, I’m ready to add a call to the mu extension I wrote to the example mu.yml. See the Add mu Extension Configuration to Example Solution below.
GitHub Personal Access Token
To integrate with GitHub, AWS CodePipeline uses OAuth tokens. Go to GitHub’s Token Settings to generate your token and ensure you enable the following two scopes:
admin:repo_hook, which is used to detect when you have committed and pushed changes to the repository
repo, which is used to read and pull artifacts from public and private repositories into a pipeline
Add mu Extension Configuration to Example Solution
In this section, I’ll cover the different ways to call a mu extension: as a local file, a local path, http/s, and s3.
url: Local File
To speed up development, you’ll likely want to test your extension locally so that you don’t need to create a release and create a zip file of your local GitHub repo. Update the extensions|url property in your mu.yml file to point to the local file.
parameters: mu-vpc-acceptance: ThreatIntelSetUrl: https://s3.amazonaws.com/delete-pmd-guardduty/GuardDutyIpSet.txt IpSetUrl: https://s3.amazonaws.com/delete-pmd-guardduty/GuardDutyThreatIntelSet.txt extensions: - url: file:///Users/paulduvall/Downloads/mu-guardduty-0.7.zip
url: Local Paths
You can also store extensions in paths within your repository. To do this create a local directory and include the extension files (CloudFormation, mu-extension.yml, etc.) in this directory. Then, referring to this directory using the url property within extensions as shown below. Commit the files to your GitHub repository.
extensions: - url: guardduty
Using this approach is a way of creating a private extension as well.
url: http(s)
Once you’ve tested your changes locally, you can go through the release process as described in one of the previous sections. Get the URL from the zip that’s generated during the release process and modify the url property of the extensions block in your mu.yml. Save your changes and then run mu.
parameters: mu-vpc-acceptance: ThreatIntelSetUrl: https://s3.amazonaws.com/delete-pmd-guardduty/GuardDutyIpSet.txt IpSetUrl: https://s3.amazonaws.com/delete-pmd-guardduty/GuardDutyThreatIntelSet.txt extensions: - url: https://github.com/stelligent/mu-guardduty/archive/v0.6.zip
Here’s an example of running mu from the command line:
mu pipeline up -t [GitHub Personal Access Token]
S3 Option
You can also use the s3 property within extensions to include a mu extension. This is particularly useful for creating a private extension as you can configure your S3 bucket to restrict access to the extension. That said, you can also make the extension public within S3. See the example below. You will use your own bucketname and extension name and version.
extensions: - s3: s3://bucketname/mu-guardduty/v0.6.zip
Dry Runs
You can use the dry-run option to create CloudFormation templates locally on your computer. This is useful when you want to see the templates that are generated by mu without actually creating the stacks on your AWS account.
To do this, run the env list command as you see below.
mu env list
Once you get the name of the environment, you can run the -dry-run (or -d) option to create the CloudFormation templates locally. Getting the environment names are also useful when you want to know how to call the extension from within a mu.yml configuration file.
mu -d env up [env-name]
After you run the above command, mu will output the name and location of the files it generated locally in your environment.
IAM Permissions
Most likely you will need to add additional Identity and Access Management (IAM) permissions for the resources you’re creating as a part of your mu extension. When I was creating the GuardDuty extension, I needed to assign a GuardDuty IAM role to existing resources.
To do this, you need to go back to the https://github.com/stelligent/mu/tree/master/templates/assets and determine which CloudFormation template you’d like to extend. When assigning the GuardDuty IAM role, I chose to extend the https://github.com/stelligent/mu/blob/master/templates/assets/common-iam.yml template. I did this by creating a new common-iam.yml file in the mu-guardduty repository that hosts the extension. I added the following configuration code:
Resources: CloudFormationRole: Properties: Policies: - PolicyName: deploy-guardduty PolicyDocument: Version: '2012-10-17' Statement: - Action: - guardduty:* - iam:PutRolePolicy Resource: '*' Effect: Allow - Action: - iam:CreateServiceLinkedRole - iam:PutRolePolicy Resource: 'arn:aws:iam::*:role/aws-service-role/guardduty.amazonaws.com/AWSServiceRoleForAmazonGuardDuty*' Condition: StringLike: iam:AWSServiceName: guardduty.amazonaws.com Effect: Allow
Getting a mu extension reviewed by Stelligent
If you’d like Stelligent to review your mu extension, you can reach out to us on Gitter at https://gitter.im/stelligent/mu. As part of the review, we typically review the following and more:
- Review README documentation – is it clear and useful. It should mention the AWS product costs.
- Review naming conventions for extension, parameters, and other resources
- Review GitHub Release artifacts
- Test your extension
- Review the extension for overlapping functionality with other mu extensions
Additional Resources
|
Stelligent Amazon Pollycast
 |