Delivering software as rapidly as your technology and business will allow should be the main goal of any organized engineering team. Each organization, project and team must define what the cadence of delivery will be. Regardless of the frequency of your deployments, it’s the responsibility of your DevOps team to provide the guidance and technology to support it. As we continue the DevOps Culture series, we look at how your entire team’s experience must come together to make these goals achievable.

So far, we’ve looked at DevOps Culture as it pertains to your engineering team. As with anything else that is challenging, it takes a village to make it successful. Getting out of a long release cycle and closer to continuous delivery requires the buy-in and confidence of your stakeholders, management, engineers and ultimately your customers. In this blog post, we will look at how you can work with each of these groups to highlight the benefits of rapid software delivery and how we mitigate the concerns — both technologically and culturally.

Customer Confidence

An end-user, microservice, a robot or anyone who consumes your services

A successful software delivery model starts with thinking about the customer. We must put ourselves in the mindset of the people (or robots) who will be consuming the software we’re building. Ultimately, it’s these customers who drive the revenue and the whole purpose in our work.

We build trust with our customers by providing a reliable and consistent experience. These days, our services must remain fully functional and revenue generating 24/7/365.

Companies that are successful in this simple concept have uneventful deployments:

There are no maintenance windows
Performance and availability is rarely degraded
Defects are isolated (i.e. low blast radius)

It’s important to understand that your customers ultimately drive the purpose in your work because it’s the metric that we must track in order to be successful. At the end of the day, your leadership team is going to prioritize the revenue over any engineering. By treating your infrastructure to be as important as your application, you can build in reliability and availability right away without having to compromise.

Engineer Confidence

Software Developers, Designers, Security, DevOps and anyone else who commits code

A consistent, reliable experience is delivered to your customers by your engineering team. It’s a combination of software development, infrastructure and delivery best practices that culminate to make software delivery successful. With the goals of the business and customer in mind, we as DevOps engineers must position ourselves as the champions of the delivery process by embedding ourselves within the development team.

Whether you’re starting continuous delivery for the first time or bringing a new employee into the process, we must jump headfirst into building the confidence of your engineering team. Confidence in anything is gained from experience — either shared from a peer or gained through trial and error.

Processes for building confidence naturally

Self Service. Provide the tools for your team to innovate and work in isolation when needed.
Incremental Development. Implementing change imposes risk. Reduce the risk and blast radius by encouraging small and incremental changes.
Peer Review. A powerful tool when executed properly. Peer review should focus on requirements, compatibility and knowledge sharing.
Automated Testing. It may seem obvious, but testing is rarely done properly. A developed feature includes automated testing — they are written by default. Review Test Driven Development or Behavior Driven Development if you need help getting started.

Leadership Confidence

Managers, Stakeholders, Team Leads

Delivering software more frequently equates to more revenue. Customers engage with features more rapidly and your engineering team avoids costly deployment and maintenance windows. This keeps your services serving and your engineers working on more profitable things like feature development and innovation. Probably an easy sell, right? It gets more tricky on the execution. How do we build the trust between leadership and engineering to deliver more frequently?

Fast Feedback, Fast Response

The feedback you receive from the engineering team, pipelines, software, infrastructure and your customers are all key forms of feedback. Consider automating the feedback loop for your engineering components — pipeline, software and infrastructure — and incorporate your customer feedback into the process. When things are working properly your team will know of software defects before they reach production and before your customers notice.

Continuous Integration / Delivery. Engineering teams immediately swarm to remediate issues and restore the pipeline.
Monitoring. Trigger automated responses based on actual key performance indicators where possible.
Communication. Teams should have readily accessible ways to collaborate. A open war room for offices or easy access to screensharing tools for remote workers.

Risk Evaluation & Mitigation

Every software change has an associated risk. It’s our planning, architecture, infrastructure, testing and engineering experience that help evaluate and make the risks of deployment acceptable. Every single deployment should meet the team’s standard for review, testing and quality before it can be committed to master or trunk for immediate release.

In the real world of databases, third party integrations and compliance requirements, you cannot have a one-size-fits-all level of risk and deployment plan. Changes that fit higher risk criteria are great candidates for feature toggles, A/B testing and additional instrumentation. However you manage the various risk levels, always strive to have concise criteria and a clear deployment plan for each.

Note. Even in continuous delivery environments, different levels of effort can be made to ensure a smooth release of a change. In a team that embraces DevOps, there will be no separation of these responsibilities and the necessary application, infrastructure and deployment can be managed by a single person or team.

Handling Defects

Defects making it into production is normal. Treat bugs and defects like any other portion of the software development process. Ensure you have a feedback loop that is fast and effective. Have a plan for remediation handy — like fixing the issue, performing a rollback or disabling the feature — and make sure everyone is trained on how to access & execute it. Any or all of these options are legitimate. Each time you encounter a defect, steps should be taken to prevent its recurrence as part of the final resolution of the issue.

Collective Accountability. Empower your team to own problems and see them to their resolution. Everyone must share the accountability and responsibility of defects regardless of where in the software development lifecycle they are discovered.
Shared Responsibility. Anyone can engage in a break-fix activity, even if they’re not the originator of the issue. If it’s broke, fix it! If you need or can offer support, raise your paw!

Measuring Success: Trust but Verify

Folks often ignore the hundreds of instances where things are working correctly, so I like to say: “Success in the Information Technology space is always measured by your latest problem.” It’s for this reason that you must measure your effectiveness and review it with stakeholders regularly. In the world of DevOps we can measure a few key performance indicators to give insight into our performance.

Lead Time. How long does it take to go from commit to production?
Mean Time To Recovery. How long does it take to recover from an outage?
Builds. How many times are we building? Also measure the count & ratio of failed builds.
Test Coverage. How is our code coverage performance?
Performance Metrics. How does your application’s performance handle deployments?

Pro Tip: Make metrics like these easily accessible for your team. Ideally, have them projected on a wall in a common area so your entire team can understand the performance of their software delivery model. Bonus points for including them in your regular activities such as standups and retrospectives.

Conclusion: Be The Change You Want To See

Your organization can be delivering more frequently today by discussing these concepts and employing them in your regular business. Zero to hero is not something that happens overnight; it begins with a single person or small group. Typically success is infectious and your whole organization will be on board.

As with any cultural change it all starts with people. As leaders of our teams, we cannot simply demand this level of excellence with just policy, we need these positive traits to develop and emerge naturally. As you work with your managers, engineers and stakeholders guide them to the best practices through thoughtful conversation instead of dictation. Bring this blog post with you and use it as a starting point for your next meeting.

Bring in the Experts: Mphasis-Stelligent

Can’t wait? Mphasis Stelligent is a Premier AWS Consulting Partner and we specialize in DevOps and Continuous Delivery. Contact us today, and we can supplement your team with any number of thought leaders in the space of Software Delivery on the AWS Cloud.

Other blogs in this series

Want to know more about how to drive cultural change in your organization, check out the other posts in this series by Robert Murphy

Stelligent Amazon Pollycast