Stelligent

AWS re:Invent 2019 DevOps and Security re:Cap

  • Paul Duvall

    • We had over 40 people from Mphasis and Mphasis Stelligent at the AWS re:Invent 2019 conference in Las Vegas, NV. There were 77 product launches, feature releases, & services announced at the conference (and many more at “pre:Invent” in November). Of this, there were several DevOps-related features announced at re:Invent or during pre:Invent.

    The theme of this year’s re:Invent was around transformation and how AWS has the services and tools to help enterprises and others completely transform their business models. During his keynote, Andy Jassy, CEO of AWS, described four points to this transformation:

    1. Senior leadership team conviction and alignment.
    2. Aggressive top-down goals (faster than they would otherwise do organically).
    3. Train your Builders.
    4. Don’t let paralysis stop you before you start.

    As John Furrier at SiliconANGLE states, “As the cloud market changes industry structures, it is our view that many of the next-generation leaders will be new companies either born in the cloud or existing enterprises reborn in the cloud.” He went on to say that the rest will likely go out of business. Furrier adds that “Although the cloud is changing everything, it still only represents just 5.6% of total IT spending, which Gartner projects will hit $3.8 trillion this year.”

    Instead of providing a broad brush across all of the announcements from re:Invent 2019, we decided to choose the top 5 that we believe will have a transformative effect on the DevOps community. Here are the top 5 (in alphabetical order):

    1. Amazon CloudWatch Synthetics – collects “canary” traffic, which can continually run tests in production even when there is no customer traffic helping you potentially discover issues before customers do.
    2. Amazon CodeGuru – automate code reviews and identify your most expensive lines of code.
    3. Amazon Detective – analyze and visualize security data to rapidly get to the root cause of potential security issues.
    4. Amazon EKS on AWS Fargate – serverless compute for Kubernetes.
    5. AWS IAM Access Analyzer – generates comprehensive findings that identify resources that can be accessed from outside an AWS account.

    Before diving more deeply into these details of these services and features, I’d like to take a few steps back and describe what DevOps means to us. From our perspective, DevOps is about accelerating the speed of feedback between customers and builders (i.e. engineers, developers, and so forth). This notion is illustrated below.

    If you think of a traditional development process, you build, test, and release software to customers. On these traditional teams, it’s usually a slow and arduous process of getting feedback from customers. This is often the result of organizational, process, cultural, and tooling barriers that throttle this feedback. This isn’t necessarily intentional but more the result of organizational inertia that has built up over time in which there isn’t a focus on speeding up effective feedback between customers and developers.  

    When effectively applying DevOps practices, you compress the time by which developers get this feedback from customers while increasing the quality. You do this by breaking down organizational silos, treating everything as code, creating fully-automated workflows that build, test, deploy, and release software to production whenever there’s a business need to do so. By getting regular, effective feedback from customers, developers are more likely to build features that customers want. 

    There are two key points to consider:

    Ultimately, DevOps is any organization, process, culture, or tooling changes that help speed up these effective feedback loops between customers and developers.

    In the Developer Tools on AWS breakout session the general manager of AWS Developer Tools, Ken Exner, illustrated a pipeline of stages and actions on a typical Amazon or AWS team. These pipelines perform actions like getting the latest code, running automated tests and static analysis, and running synthetic monitoring/tests in production – to name a few. He also described time windows, enterprise and team pipeline policies, code reviews, security scans, and other techniques they use.

    A typical pipeline at Amazon and AWS

    AWS provides a suite of services and tools that support authoring, sourcing, building, testing, deploying, monitoring, and releasing software features. These services and tools include AWS CodePipeline, AWS Cloud9 and IDE toolkits, AWS CodeCommit, AWS CodeBuild, Amazon CodeGuru, AWS CodeDeploy, AWS X-Ray, Amazon CloudWatch, AWS CloudFormation, AWS CDK, and the AWS Serverless Application Model. Of course, they also provide support and integrations for all the third-party tools you might be using.

    AWS Developer Tools for modern software delivery

    Our focus at Mphasis Stelligent is in helping customers increase the speed and safety by which they deliver their software to production. Consequently, the AWS re:Invent announcements we will be focusing on in this post are the DevOps-related features that best help safely accelerate these feedback loops.  

    Amazon CloudWatch Synthetics

    Amazon has been using automated synthetic testing for many years. Netflix engineering open sourced an automated canary testing tool called Kayenta as part of the Spinnaker project. The purpose of this technique is to run tests and analysis in production to learn of potential problems before end users do.

    For example, you can run these automated tests to ensure a key part of your system (e.g. ordering) continues to work. If there are any errors or degradation, you learn of it as soon as possible and, in many cases, even before your end users.

    CloudWatch Synthetics makes it possible to run these tests and monitor them through the CloudWatch dashboard. What’s more, you can integrate Synthetics with AWS X-Ray to accelerate your debugging process.

    CloudWatch Synthetics makes it easy to:

    Amazon CloudWatch Synthetics is available in preview in the following public AWS Regions: US East (N. Virginia), US East (Ohio), and EU (Ireland).

    Running a Canary Test

    To get started, go to the Amazon CloudWatch console, then click on Synthetics on the left panel. The page displayed should look similar to the one below. From here, you will click on the Create canary button.

    Create a canary test from the CloudWatch Synthetics Console

    Next, you will go through the steps of creating a canary by either selecting a blueprint from AWS, uploading a script, or importing from S3.

    Choose the type of canary test you wish to run

    Then, you will give your canary a name, enter the application or endpoint URL, and modify the provided script as necessary. You can also define whether you only want the canary to run once or on a continuous schedule.

    Choose the name, endpoint, and test script to run

    Finally, you can define the data retention of the canary data, any alarm thresholds, and access permissions. Then, you click the Create canary button to begin running your test.

    Configure data retention, alarm thresholds, and permissions

    Because it’s AWS, you can set up the same with the CLI or SDK.

    CloudFormation Support

    None yet, but you can write your own CloudFormation resource provider.

    Pricing

    The pricing for Amazon CloudWatch Synthetics is straightforward. You are charged $0.0012 per canary run. Using the CloudWatch pricing calculator – as shown below – if you run 100,000 canary runs in a given month, you will pay $120.

    CloudWatch Synthetics Pricing Scenario

    Amazon CodeGuru

    Amazon CodeGuru is a machine learning service for automated code reviews and application performance recommendations. It helps you find the most expensive lines of code that hurt application performance and keep you up all night troubleshooting, then gives you specific recommendations to fix or improve your code. As of December 2019, it is in preview.

    Trained own decades of experience and knowledge at Amazon. It evolves with user feedback. The Profiler searches for optimizations continuously, even in production. Provides actionable recommendations to fix identified issues. Automatically inspects code for hard to find defects. Helps you find the most promising methods for optimization in your running application.
    CodeGuru currently supports Java code.
    As AWS states, “It is like having a distinguished engineer on call 24×7” [Source].
    CodeGuru does not persist customer code – it analyzes the code and then deletes its copy. It does not perform security analysis or testing at this time.

    CodeGuru Reviewer

    CodeGuru provides these features:

    An illustration of how CodeGuru Reviewer works is shown below.

    Amazon CodeGuru Reviewer – How it Works [Source]

    To begin with CodeGuru Reviewer, you go to the CodeGuru console and associate a repository as show in the image below.

    Amazon CodeGuru Reviewer – Associate Repository [Source]

    After associating a GitHub or CodeCommit repository, CodeGuru will analyze code changes during pull request workflows and provide detailed and targeted recommendations on how to improve the code – as shown below.

    Amazon CodeGuru Reviewer – Recommendations [Source]

    CodeGuru Profiler

    “CodeGuru Profiler provides specific recommendations so you can take action immediately on issues such as excessive recreation of expensive objects, expensive deserialization, usage of inefficient libraries, and excessive logging.” [Source]

    For example, on Amazon Prime Day, there was a 325% increase in CPU utilization and 39% lower cost between 2017 and 2018 as a result of the technology behind CodeGuru Profiler.

    As of December 2019, CodeGuru is supported in the following AWS regions: US East (N. Virginia) US East (Ohio) US West (Oregon) Europe (Ireland), and Asia Pacific (Sydney).

    Let’s get started by having a look at how CodeGuru Profiler works. First, you need to install a very small agent in your application. The agent collects runtime data from your application as your application runs. It runs in a separate thread within your application.

    Then, it begins to produce visualizations of your application performance using heuristics and machine learning. It also provides actionable recommendations that can save time and money.

    CodeGuru Profiler How it Works [Source]

    An example CodeGuru Profiler visualization for a web application is shown in the figure below.

    CodeGuru Profiler Visualizations [Source]

    You can click on sections in the visualization to display detailed recommendations on how to improve application performance.

    CodeGuru Profiler Recommendations [Source]

    Language Support

    Both CodeGuru Reviewer and Profiler support Java as this point. Expect support for commonly used programming languages in the future.

    CloudFormation Support

    None yet, but you can write your own CloudFormation resource provider.

    Pricing

    Amazon Detective

    Amazon Detective helps you to quickly detect, analyze, and investigate the root cause of a security-related issue. With Amazon Detective, you can:

    As of December 2019, this service is in preview.

    Amazon Detective – How it Works [Source]

    In a typical security incident response scenario, AWS has many services to enhance investigations. These include AWS Systems Manager, Config, and CloudWatch to identify security incidents. Services such as AWS Shield, AWS Secrets Manager, AWS WAF, and AWS IAM to protect access to resources. Amazon Inspector, Amazon Macie, Amazon GuardDuty, and AWS Security Hub to protect and detect security incidents. Amazon CloudWatch and AWS Lambda to investigate and respond. To automate, the detection and response, you can use AWS Lambda. Finally, to recover, you can use things like S3 snapshots and Glacier archives to support disaster recover scenarios.

    AWS Services the Enhance Security Investigations [Source]

    While AWS provides a lot of services, there’s also a lot of complexity and noise in getting access to the right data to appropriately respond to the security incident. What’s more, there’s a skill shortage of security analysts and as a result of the people and complexity, a significant costs associated with responding to these incidents.

    Investigation Challenges [Source]

    This is why AWS announced the preview release of Amazon Detective which provides this built-in data collection, automated analysis augmented by machine learning, and visual insights.

    Amazon Detective – Key Features [Source]

    There are three example use cases that AWS shared at re:Invent: Alert triage, incident investigation, and threat hunting.

    Alert Triage Use Case [Source]

    For Alert triage, you need to get quick answers to questions like whether it’s abnormal traffic, what happened right before, whether it’s a common failure, and how much data was sent.

    Incident Investigation Use Case [Source]

    For Incident investigation, you want to know things like what other EC2 instances communicated with the IP address, do the calls indicate that someone is poking around looking for holes, and if there were other principal IDs being used.

    Threat Hunting Use Case [Source]

    Finally, for threat hunting, you might want to know whether the suspicious user agent made any API calls and whether the same IP Address communicated with any of your instances over the past year.

    Let’s have a look at an example. In this example, we’ll have a look at Amazon GuardDuty which already have deep integration with Amazon Detective. The GuardDuty findings page is shown below. From here, you can click on Actions and then Investigate which opens the Amazon Detective service console for the selected finding.

    Access Amazon Detective from Amazon GuardDuty [Source]

    From here you delve into detail in the Amazon Detective console – as shown below.

    Amazon Detective CloudTrail Details [Source]

    And, get delve into even more detail to help triage, investigate, and solve.

    CloudFormation Support

    None yet, but you can write your own CloudFormation resource provider.

    Pricing

    For more information, see Amazon Detective pricing.

    AWS Fargate Support for Amazon EKS

    With AWS Fargate Support for Amazon EKS, you no longer need to provision and manage your own EC2 cluster for your EKS nodes. This significantly reduces the operational burden as AWS manages the underlying compute resources for you.

    AWS also released AWS Fargate Spot at re:Invent which can help you save up to 70% on your use of AWS Fargate.

    With EKS on Fargate you can bring any existing pods that you have created elsewhere. Each pod runs in an isolated environment across multiple availability zones. You only pay for the resources you need to run your pods. With EKS on Fargate, AWS includes native integrations for networking and security.

     

    EKS on Fargate – Key Features [Source]

    Getting Started

    To get started, I will create a cluster. For more information on using eksctl, see Getting Started with eksctl.

    pip install awscli --upgrade --user
curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
eksctl version

    Now, you can use eksctl to create an EKS Cluster.

    eksctl create cluster --name demo-mphasis --region us-east-1 --fargate

    After the cluster is created (which might take up to 10 minutes or so), you should see an active cluster – as shown below:

    Next, you need to create a Fargate profile. To do, go to the Amazon EKS console and select the demo-mphasis cluster. On the details pane under Fargate profiles, choose Add Fargate profile.

    In the profile configuration page, I enter demo-mphasis then enter the FargatePodExecutionRole generated by the eksctl tool.

    Now, you will configure the pod as shown below:

    Review and create the Fargate profile.

    With EKS for Fargate you no longer need to:

    What you get “out of the box”:

    You cannot:

    [Source]

    Key Aspects of EKS on Fargate [Source]

    Fargate for EKS is generally available. As of December 2019, Fargate support for EKS is available in the following public AWS Regions: US East (N. Virginia), US East (Ohio), AP Northeast (Tokyo), and EU (Ireland).

    CloudFormation Support

    None yet, but you can write your own CloudFormation resource provider.

    Pricing

    AWS IAM Access Analyzer

    IAM Access Analyzer informs you which resources in your account that you are sharing with external principals using logic-based reasoning to analyze resource-based policies in your AWS environment. This feature is generally available.

    Access Analyzer uses automated reasoning to a combinatorial number of scenarios. With Access Analyzer, you can:

    While these are some fantastic benefits to the current set of features in IAM Access Analyzer, I’m probably more excited about the possibilities of applying automated reasoning to resource-based policies. In particular, I’m looking forward to the time when AWS can help us define least-privilege permissions on IAM policies based on usage.

    Until then, let’s learn the basics of IAM Access Analyzer.

    To get started, you go to your IAM console and click on Access analyzer on the left pane. The Analyzer analyzes the resource-based policies for IAM Roles, S3 Buckets, Lambda functions, KMS Keys, and SQS Queues. It then generates findings on who has access to what.

    Next, click the Create analyzer button as shown in the figure below.

    Create an Analyzer from the IAM Access Analyzer Console

    The name of the analyzer is pre-populated for you but you can modify the name as well. You can add any tags and then click the Create analyzer button.

    Provide a name and tag for your Analyzer

    Once you create the analyzer, the findings generally don’t immediately display so you’ll need to give it some time. Once the findings are generated, they’ll look similar to what you see in the figure below.

    Findings generated by IAM Access Analyzer [Source]

    IAM Access Analyzer currently works on an account level but it was announced the support for AWS Organizations – in which you can run Access Analyzer on multiple AWS accounts – will be coming soon.

    Below is an illustration of a solution from Millennium Management that was described in one of the breakout session at re:Invent. In the solution, they’re using an AWS Step Functions workflow to analyze IAM Access Analyzer findings, determine a risk level, automatically remediate through AWS Lambda, and then notify team members.

    Automated Remediation Solution from Millennium Management shown at re:Invent [Source]

    CloudFormation Support

    Yes, using AWS::AccessAnalyzer::Analyzer.

    Here’s an example CloudFormation template:

    AWSTemplateFormatVersion: 2010-09-09
Resources:
  Analyzer:
    Type: 'AWS::AccessAnalyzer::Analyzer'
    Properties:
      AnalyzerName: MyAccountAnalyzer
      Type: ACCOUNT
      Tags:
        -
          Key: Kind
          Value: Dev
      ArchiveRules:
        -
          # Archive findings for a trusted AWS account
          RuleName: ArchiveTrustedAccountAccess
          Filter:
            -
              Property: 'principal.AWS'
              Eq:
                - '123456789012'
        -
          # Archive findings for known public S3 buckets
          RuleName: ArchivePublicS3BucketsAccess
          Filter:
            -
              Property: 'resource'
              Contains:
                - 'arn:aws:s3:::docs-bucket'
                - 'arn:aws:s3:::clients-bucket'

    Pricing

    Free!

    Honorable Mentions

    While they didn’t make Stelligent’s DevOps top 5, we thought the following announcements were worthy of honorable mention and we expect these services to be very useful for our customers.  

    CloudFormation

    Here’s a summary of some of the recent CloudFormation changes related to DevOps and security.

    What’s Next?

    These services and features accelerate the speed and safety of feedback by ensuring your code is performant, your resources are more secure, and that you have the means to mitigate any potential security incidents.

    What’s most exciting to see is how AWS is leveraging accelerant technologies such as machine learning and automated reasoning in the software delivery lifecycle (and, yes, this includes production).

    All of these capabilities continue to move up the stack from primitive building blocks to managed services and as Werner Vogels, the CTO of Amazon, said in his 2018 re:Invent keynote, “The only thing we want to do is to build business logic” and AWS is helping teams achieve this.

    (2018 re:Invent keynote) The future is that builders will be able to focus on business logic [Source]

    The big DevOps announcements helped make it easier to:

    We expect that all of these features will help speed up feedback loops between customers and developers. We’re excited to get to work with our customers in leveraging these accelerants.

    Additional Resources

    Stelligent Amazon Pollycast