Stelligent

CodePipeline Dashboard

  • The Stelligent Team

    • CodePipeline Dashboard

    Continuous delivery seeks to release software to its users faster and more often.  The AWS CodePipeline service is an extremely valuable tool for implementing continuous delivery methodologies.  However, just because you have a continuous delivery pipeline, how do you know if you are realizing the benefits that continuous delivery promises?  In this post, I will demonstrate how to collect metrics and generate a dashboard to assess the health of your continuous delivery pipelines.  Additionally, you will be able to run the same dashboard in your own account by clicking a “Launch Stack” button and going through the AWS CloudFormation steps to launch the solution stack
    As demonstrated in a prior post, we can now write code that reacts to the events that CodePipeline publishes to Amazon CloudWatch Events anytime an action, stage or pipeline changes state.  I will use a similar approach by having a CloudWatch Event Rule Target invoke a Lambda function to create CloudWatch Metrics that captures the activity from CodePipeline as shown in Figure 1.

    pipeline-dashboard

    Figure 1 – Capture CodePipeline events as CloudWatch Metrics

    These AWS resources are provisioned with a CloudFormation template that leverages the AWS Serverless Application Model.  As seen in the resource declaration below, it is very simple to create a Lambda function as well as to configure a CloudWatch Event Rule that targets the Lambda function: 
      EventHandlerFunction: 
    Type: 'AWS::Serverless::Function' 
    Properties: 
      FunctionName: pipeline-dashboard-event-handler 
      Description: Create CloudWatch metrics from CodePipeline events 
      Handler: index.handlePipelineEvent 
      Runtime: nodejs6.10 
      CodeUri: . 
      Role: !GetAtt EventHandlerRole.Arn 
      Events: 
        PipelineEventRule: 
          Type: CloudWatchEvent 
          Properties: 
            Pattern: 
              source: 
                - "aws.codepipeline" 
              detail-type: 
                - "CodePipeline Pipeline Execution State Change" 
                - "CodePipeline Stage Execution State Change" 
                - "CodePipeline Action Execution State Change"

     
    All that remains is to create the CloudWatch Dashboard to visualize the metrics we are now capturing.  Although CloudWatch Dashboards can be created with CloudFormation, we are unable to define dynamic dashboards that list metrics about all the active pipelines in our account.  Therefore, we have another Lambda function that is triggered by CloudWatch Scheduled Events to generate the dashboard as shown in Figure 2.

    Figure 2 – Generate CloudWatch Dashboard based on CloudWatch Metrics

    Again, we use CloudFormation to define the Lambda function and CloudWatch Scheduled Event to trigger the dashboard regeneration at regular intervals: 
      DashboardGeneratorFunction: 
    Type: 'AWS::Serverless::Function' 
    Properties: 
      FunctionName: pipeline-dashboard-generator 
      Description: Build CloudWatch dashboard from CloudWatch metrics 
      Handler: index.generateDashboard 
      Runtime: nodejs6.10 
      CodeUri: . 
      Timeout: 60 
      Role: !GetAtt DashboardGeneratorRole.Arn 
      Events: 
        DashboardEventRule: 
          Type: Schedule 
          Properties: 
            Schedule: "cron(*/5 * * * ? *)"

    Dashboard in Action

    The event runs every 5 minutes and rebuilds the list of pipelines in the dashboard based on the distinct list pipeline names on the metrics that has been recorded.  The resulting dashboard shown in Figure 3 displays 5 metrics for each pipeline:

    Figure 3 – Pipeline dashboard

    The Cycle Time and Lead Time metrics are the primary indications of the health of your pipeline and the other three (MTBF, MTTR, and Feedback Time) are useful in diagnosing issues with Cycle Time and Lead Time.

    Figure 4 – Pipeline Lead Time and Cycle Time

     

    Cycle Time

    Lead Time

    Cycle Time vs Lead Time

    One thing worth highlighting is the difference between Cycle Time?and?Lead Time as they are often conflated.  I appreciate the simplicity with which Paulo Caroli defined them in his post, Continuous Delivery: lead time and cycle time.

    • Lead time: the amount of time a work item takes from the beginning to the end of the workflow
    • Cycle time: the interval of time between two consecutive work items leaving the workflow. 

    To help understand, let’s compare the two metrics in the following scenarios. Notice that?Lead Time?is the same for the pipelines in both scenarios, however the cycle time is much smaller in Figure 5 due to the fact that the second commit happened sooner and the pipelines are running in parallel:

    Figure 5 – Pipelines in parallel with low cycle time

     
    To understand the significance of the additional metrics (MTBF, MTTR, and Feedback Time), it helps to visualize the metrics in the context of the continuous delivery pipeline itself:

    Figure 6 – Pipeline metrics

     

    MTBF

    MTTR

    Feedback Time

    When you are ready to launch the solution in your own account, click the “Launch Stack” button for the region you want to launch the stack into and follow the prompts.

    Once the CloudFormation stack is in the CREATE_COMPLETE state, go ahead and trigger an execution of one of your existing CodePipelines.  Upon completion, navigate to CloudWatch Dashboards and click on the Pipelines dashboard to view the results.  The longer you allow the dashboard to collect data, the more interesting the dashboard will become.
    Remember, it is never enough to just build solutions that ought to benefit your organization. You must also measure results and make corrections to ensure the solution does realize the benefits you expected!

    Additional Resources

    Here are some of the resources referenced in this post:

     

    Stelligent Amazon Pollycast