Stelligent was 50 people strong at the AWS re:Invent 2018 conference in Las Vegas, NV. We were particularly proud as we announced that we are the only group at our size that is 100% AWS certified (yes, this means executives, sales, marketing, operations, and engineers).

There were over 80 product announcements at the conference. Of this, there were probably 15 new DevOps-related features announced at or prior to re:Invent. Instead of providing a broad brush across all of these announcements, Casey Lee and I decided to – once again – choose the top 5 and provide a bit more detail on why we think they matter to the DevOps community. Here are the top 5:

AWS CodeDeploy now supports blue/green deployments for ECS
AWS CodePipeline announced support for ECR as a source provider, faster pipelines and more actions, and VPC Endpoints
AWS Lambda now supports for Ruby, Layers, and custom runtimes
AWS announced two new services for security and governance called AWS Control Tower and AWS Security Hub
Amazon VPC now supports shared VPCs and announced the new AWS Transit Gateway service

When we think of DevOps we think about increasing the speed of feedback between customers and developers. This notion is illustrated below.

If you think of a traditional development process, you build, test, and release software to customers. On these traditional teams, it’s usually a slow and arduous process of getting feedback from customers. This is often the result of organizational, process, cultural, and tooling barriers that throttle this feedback. This isn’t necessarily intentional but more the result of systemic dysfunction that has built up over time in which there isn’t a focus on speeding up effective feedback between customers and developers.

When effectively applying DevOps practices, you compress the time by which developers get this feedback from customers while dramatically increasing the quality by breaking down organizational silos, treating everything as code, creating fully-automated workflows that build, test, deploy, and release software to production whenever there’s a business need to do so. By getting regular, effective feedback from customers, developers are more likely to build features the customers want.

There are two key points to consider:

How fast you’re able to get through this feedback loop determines how responsive you can be to customers and how innovative you are.
From your customer’s perspective, you are only delivering value when you’re spending time on developing high-quality features.

Ultimately, DevOps is any organization, process, culture, or tooling changes that help speed up these effective feedback loops between customers and developers.

Our focus at Stelligent is in helping customers increase the speed and safety by which they deliver their software to production. Consequently, the AWS re:Invent announcements we will be focusing on in this post are the DevOps-related features that best help safely accelerate these feedback loops.

CodeDeploy

An exciting announcement at re:Invent was support for blue/green deployments of a containerized application in an Amazon ECS service via AWS CodeDeploy. Previously, AWS CodeDeploy supported deployments to EC2 instances as well as AWS Lambda, but now with support for Amazon ECS you can leverage an AWS CodeDeploy application that uses the new Amazon ECS compute platform to deploy a containerized application to a new or replacement task set in the same Amazon ECS service.

To best understand how this works, let’s walk through some of the slides that Principal Engineer Clare Liguori shared in her demo. First, we see in the figure below that we have an AWS Fargate service defined with v1 of the code running. We call this collection of tasks the “blue tasks”. There is an ALB (Application Load Balancer) configured with a listener on port 80 that routes all traffic to a target group that contains these blue tasks. Additionally, we can see that we have a second listener on the ALB on port 9000 and a second target group that has no tasks running in it yet. These will be useful later on when we want to test out a new version of the code.

When we are ready to deploy a new version of the code, we can leverage AWS CodeDeploy to define a new revision of the application that refers to the newest docker image. AWS CodeDeploy will the leverage Amazon ECS to launch a new set of tasks with v2 of the code, which we will refer to as the “green tasks”. The new tasks will be added to the test target group for testing before any production traffic is routed to them as shown in the figure below.

The AWS CLI now has support for accomplishing all of this with a single command:

aws ecs deploy --service my-service-name --task-definition task-def.json --codedeploy-appspec appspec.yml

You can also register Lambda functions as validation hooks in your appspec file to be invoked at various times in the deployment process to ensure the deployment is working as expected.

Hooks:
 - BeforeInstall: "LambdaFunctionToValidateBeforeInstall"
 - AfterInstall: "LambdaFunctionToValidateAfterTraffic"
 - AfterAllowTestTraffic: "LambdaFunctionToValidateAfterTestTrafficStarts"
 - BeforeAllowTraffic: "LambdaFunctionToValidateBeforeAllowingProductionTraffic"
 - AfterAllowTraffic: "LambdaFunctionToValidateAfterAllowingProductionTraffic"

You can monitor the progress of the traffic shift from blue to green in the AWS CodeDeploy console as shown below.

Once all the hooks complete successfully, the traffic at the ALB is switched over from the “blue tasks” that represent v1 of the code to the “green tasks” that represent v2 of the code. As you can see in the figure below, the blue tasks are left running with 0% of traffic routed to them. This allows for an immediate rollback in the event of an issue with the green tasks.

By default, AWS CodeDeploy will wait one hour before terminating the blue tasks. The deployment group can be configured with CloudWatch alarms to monitor during this time so that it will automatically initiate a rollback in the event that one of the alarms triggers. If no alarms trigger during the waiting period, then AWS CodeDeploy terminates the blue tasks as seen in the figure below.

One final note worth mentioning with this new feature of AWS CodeDeploy is that it is also available as an action in AWS CodePipeline. You can configure a deploy action to initiate a blue/green deployment to ECS via AWS CodeDeploy as shown below.

CodePipeline

AWS CodePipeline is a fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates. The CodePipeline team announced support for ECR as a source provider. CodePipeline is faster and supports up to 50 actions per stage, and now supports VPC Endpoints.

ECR as a source provider

In the DEV309 breakout session, AWS announced that ECR is now available as a source provider in CodePipeline. This feature better supports a pattern that many teams have been using in defining a pipeline for baking their container images and a separate application pipeline that uses these images. On many teams, these container images might be used by multiple application teams. Previously, teams developed workarounds to accommodate.

This brings the number of supported source providers in CodePipeline to four: S3, GitHub, CodeCommit, and ECR.

When defining ECR as a source, you can identify the image name and Docker tag in its configuration.

The API response from the CodePipeline definition looks like the code snippet below. To define the provider as ECR, simply use ECR in the provider attribute of the Source actionTypeId – when CodePipeline via the CLI, SDK, or CloudFormation.

"name": "Source",
"actionTypeId": {
   "category": "Source",
   "owner": "AWS",
   "version": "1",
   "provider": "ECR"
},

Below you see a deployment pipeline for creating a base image that is deployed to ECR via AWS CodeBuild. It uses source configuration stored in AWS CodeCommit to create this Docker image.

base-image pipeline that uses CodeBuild to deploy to ECR [Source]

As part of the base-image pipeline, CodeBuild deploys the Docker image to ECR. This container image can be referred to in the source as shown in the hello-world pipeline below.

App pipeline that refers to image in ECR that was deployed by base-image pipeline [Source]

Note that the pipeline uses two source actions: ECR and CodeCommit. This way the pipeline is triggered on a change to ECR or the rest of the code (application, configuration, data, etc.) that makes up the software system.

Triggers

While there were no specific announcements around this at re:Invent, CodePipeline also supports triggers through CloudWatch Events and Webhooks. With CloudWatch Events, you can trigger a CodePipeline to run on a schedule, based on an external event (e.g. Lambda), or via AWS Health Events (e.g. Fargate platform retirement). With Webhooks, you can trigger CodePipeline based on changes to systems such as DockerHub, Quay, and Artifactory.

Faster pipelines

By moving from a polling to an event-based model, CodePipeline is now remarkably faster. In the past, when we spoke to customers about using CodePipeline, they liked the idea that they didn’t need to manage their own servers in order to run CI/CD, but the most common complaint we heard was the speed of execution of the pipelines. With these faster pipelines, engineers can get quicker feedback from customers and from the pipeline itself.

Support for VPC endpoints

After re:Invent, the CodePipeline team announced support for VPC Endpoints. While the artifacts used by CodePipeline were and are encrypted in transit and at rest in S3, previously, they were transferred over Amazon’s regional network through public IP addresses. Now, with VPC Endpoints, you can choose to have this traffic isolated to a VPC that you control so that there are no public endpoints.

To configure this, you use the Amazon VPC Console. From the console, you choose Endpoints and then choose the CodePipeline service, subnets, and security groups to create the VPC Endpoint for CodePipeline. In doing this, all CodePipeline traffic for this combination of region, VPC, subnet and security groups is isolated from the rest of the Internet. The figure below shows the VPC Console for creating a VPC Endpoint for CodePipeline.

Create a VPC Endpoint for CodePipeline using VPC Console

More actions per stage

You can now run up to 50 actions per stage in CodePipeline. This provides the ability to create more complex deployment pipelines. Prior to this change, you could only run a total of 20 actions per stage and 10 actions in parallel. For example, breaking down tasks into a set of parallel actions (such as tests) in order to get quicker feedback.

Lambda

There were a few notable announcements at re:Invent for AWS Lambda. The first announcement was the long awaited news that Ruby was added as a supported runtime. Ruby is still a popular language in use at many of our customers for tools such as Chef and Puppet. Similar to other runtimes that existed with AWS Lambda, you can now register a Ruby function as a handler for your AWS Lambda function.

The next announcement for AWS Lambda is Lambda Layers. This addresses a challenge with each Lambda function having to manage its dependencies. Previously, all dependencies had to be packaged up with the code for the Lambda function. Now, you can independently register a Lambda Layer from a zip file. Layers are immutable and versioned just like Lambda functions.

To consume the layer, a Lambda function can reference up to five layers as shown in the image below. The layers are extracted into the /opt directory and the content is referenceable by the Lambda function on invocation. The layer can be shared with Lambda functions within the same account, across accounts, or even publicly.

This approach not only allows developers to DRY up their code, but it also offers an opportunity to enforce governance by providing a layer for code dependencies that has been approved by security personnel. Support for defining and referencing layers is available now in AWS CloudFormation.

The final announcement relating to AWS Lambda was support for custom runtimes. Rather than having to choose a runtime that AWS manages, you specify that your Lambda function will provide its own runtime as shown below.

By choosing a custom runtime, you must include an executable file called bootstrap in one of your layers or in the Lambda function itself. The executable bootstrap file is executed and becomes responsible for communication between your Lambda code for the function and the Lambda environment.

Information about how the runtime bootstrap should load the Lambda code is provided through environment variables. The executable will continuously poll for invocations via a simple HTTP endpoint with the response from the endpoint being a JSON payload representing the Lambda event object.

http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next

The runtime bootstrap must then either post a result or an error to the HTTP endpoint.

http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/$REQUEST_ID/response

http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/$REQUEST_ID/error

The combination of Lambda layers and custom runtimes allows new runtimes to be developed and packaged as shared layers that can be consumed by others both within an organization or even publicly.

Account Management and Governance

Keeping your infrastructure secure it of utmost importance. This is particularly essential in enterprises that have many applications and the cost of failure can be significant. Two related announcements at re:Invent were around how to apply guardrails and get better visibility into potential security issues. These new services are AWS Control Tower and AWS Security Hub. Both of these new services use several other AWS services.

These services accelerate the speed and safety of feedback by getting you properly up and running on AWS and ensuring you have the means to mitigate any potential security incidents.

AWS Control Tower

AWS Control Tower automates the set-up of a baseline environment, or landing zone, that is a secure, well-architected multi-account AWS environment. Control Tower automates the landing zone with best practices such as:

Configuring AWS Organizations to create a multi-account environment
Providing for identity management using AWS SSO Users and Groups
Federating access using AWS Single Sign-On
Centralizing logging using AWS CloudTrail and AWS Config
Enabling cross-account security audits using AWS IAM
Implementing network design using Amazon VPC
Defining workflows for provisioning accounts using AWS Service Catalog [Source]

It’s currently available for preview. Keep in mind that Control Tower only works with a new AWS Organization and accounts.

With AWS Control Tower, you pay only for AWS services enabled by Control Tower which include the setup of your AWS Landing Zone, mandatory guardrails, or customized options including:

AWS Service Catalog
AWS CloudTrail
AWS Config
Amazon VPC (NAT gateway)
Amazon S3
Amazon SNS
Amazon CloudWatch
AWS Lambda
AWS Config

AWS Security Hub

AWS Security Hub gives you a comprehensive view of your high-priority security alerts and compliance status across AWS accounts. It’s currently in public preview.

It aggregates, organizes and provides visibility across multiple AWS services including GuardDuty, Inspector, Config, Macie. In addition, it currently provides 29 additional partner integrations such as CrowdStrike, Cloud Custodian, Palo Alto, Twistlock, Splunk, and Trend Micro. It’s normalized the findings data using the AWS Findings Format so you don’t need to spend time or data normalization.

[Source]

AWS Security Hub conducts automated checks to evaluate your resources against compliance standards (such as CIS AWS Foundations), recommendations for improvement (known as Insights), Findings – which document a security or compliance issue, and other automated compliance checks via AWS Config. All of these capabilities can be customized and extended. There is full API, CLI, and SDK support.

Security Hub is currently a multi-account single region service. Eventually, it will become a multi-account, multi-region service.

You can also use Security Hub to respond and remediate via CloudWatch Events.

AWS Security Hub is offered at no cost during the preview period and is available as a region service in 15 of the current AWS regions. Pricing will be finalized when the service becomes generally available.

VPC

There were a few very interesting announcements in relation to Amazon VPC at re:Invent. First AWS announced Amazon Route53 Resolver the week before re:Invent. There are two important components of the service that allow DNS resolution in a hybrid cloud. First, inbound endpoints allow DNS queries to originate from an on-premises network and resolve Amazon Route53 hosted domains. Next, Conditional Forwarding Rules can enable outbound DNS queries from resources in your VPC to domains hosted within an on-premises DNS infrastructure.

Pricing for resolver endpoints is based on the number of IP addresses created to support the endpoint, currently $0.125 per ENI/hour. There is an additional charge for queries that pass through Route53 coming or going to on-premises resources, currently $0.400 per million queries.

As shown in the diagram below, these two components help address a challenge that existed with running a hybrid cloud. Both resolver endpoints and resolver rules are configurable via AWS CloudFormation.

The other announcement that was made right before re:Invent was the Resource Access Manager that facilitates the sharing of resources from one AWS account to other accounts within the same organization. This allows resources to be created in one account and the utilized by many other accounts. Currently, only four types of resources are able to be shared between accounts:

VPC Subnets
VPC Transit Gateways
Route53 Resolver Rules
License Configurations

We’ve already covered Route53 Resolver Rules so let’s now look at two of those other resource types, VPC Subnets and VPC Transit Gateways.

At re:Invent, AWS announced support to share subnets across accounts. When you share a VPC Subnet with another account, you are allowing the other account to provision resources that run in the shared subnet. This addresses a common challenge in which customers want to reduce cost and operational overhead by creating resources in a central account and share them across accounts rather than duplicate the resources in each account.

There are a few gotchas to watch out for with shared VPC subnets. First, you must be aware of a new primitive that AWS exposed, the AZ id. From the VPC User Guide:

To ensure that resources are distributed across the Availability Zones for a Region, we independently map Availability Zones to names for each account. For example, the Availability Zone us-east-1a for your AWS account might not have the same location as us-east-1a for another AWS account.

When sharing subnets, you need to be aware of the AZ Name to AZ ID mapping in the owner and participant accounts. For example, the figure below shows the AZ Name to AZ ID mapping for one of my accounts:

With shared VPC Subnets, each participant still pays for any resources they provision in the subnet and they also pay for data transfer charges. The VPC owner pays for hourly charges associated with data transfer across NAT gateways, virtual private gateways, transit gateways, and VPC endpoints.

There are a few limitations to be aware of with shared VPC subnets. First, it is unsupported for Amazon Aurora Serverless, Amazon EMR, Network Load Balancer, AWS CloudHSM and AWS Glue. Next, you can only share subnets with other accounts within the same AWS organization. Finally, you cannot share subnets that are in the default VPC.

The last big announcement related to Amazon VPC is the Transit Gateway. Previously, connectivity between VPCs would often result in a complex network topology diagram as seen below.

Now with the Transit Gateway, you can attach up to 5000 VPCs and support up to 50 Gbits/second of traffic. This allows you to design a hub and spoke network topology that becomes much simpler to implement and govern as shown below.

To best understand the value of the VPC Transit Gateway, let’s review a couple use cases that Principal Solutions Architect Nick Matthews shared in his talk NET402. The first use case shown in the slide below is a flat topology in which all VPCs are able to communicate with each other. A single transit gateway is created and an attachment is created for each VPC. A route table needs to be added on each VPC to route traffic to the transit gateway for the other VPCs. This presents a much simpler solution for VPC connectivity than previously was available through VPC peering.

The other use case that is very common is to provide a single ingress/egress point for all VPCs while denying traffic between VPCs. In this case, two route domains are created in the one transit gateway: one for the VPCs to use with a single default route to the VPN and another for the VPN to use with routes back to each VPC. This solution provides a much simpler approach to implement a single ingress/egress point to enforce network policy at across all VPCs.

Pricing for the Transit Gateway is $0.05 per VPC per hour and $0.02 per GB of data process. Support is already available to provision your Transit Gateway via CloudFormation.

Honorable Mentions

While they didn’t make Stelligent’s DevOps top 5, we thought the following announcements were worthy of honorable mention and we expect these services to be very useful for our customers.

Well Architected Tool – A free tool available in the AWS Management console that helps review the state of your workloads and compares them to the latest AWS architectural best practices. The tool is based on the AWS Well-Architected Framework, developed to help cloud architects build secure, high-performing, resilient, and efficient application infrastructure.
CloudWatch Logs Insights – enables you to explore, analyze, and visualize your logs instantly, allowing you to troubleshoot operational problems with ease.
Announcing AWS Key Management Service (KMS) Custom Key Store – enables you to generate, store, and use your KMS keys in hardware security modules (HSMs) that you control

Conclusion

From a DevOps perspective, we continue to see AWS flesh out its offerings in providing services and tools to help accelerate feedback loops. All of these capabilities continue to move up the stack from primitive building blocks to managed services and as Werner Vogels, the CTO of Amazon, said in his keynote, “The only thing we want to do is to build business logic” and AWS is helping teams achieve this.

The future is that builders will be able to focus on business logic [Source]

The big DevOps announcements helped make it easier to:

apply the blue/green pattern to ECS deployments in AWS CodeDeploy
use ECR as a source provider in CodePipeline
run Ruby code in AWS Lambda and include dependencies with Lambda layers
govern and manage AWS usage according to codified security and compliance policies
Simplify the management of resources across VPCs and AWS accounts with AWS Transit Gateway and Shared VPCs

We expect that all of these features will help speed up feedback loops between customers and developers. We’re excited to get to work with our customers in reducing all of the waiting!

Additional Resources

Stelligent Amazon Pollycast