Stelligent

Infrastructure as Code – Soofi’s Gold

  • The Stelligent Team

    • If you read our earlier post , Infrastructure as Code – A Hit at re:Invent 2015, you know how excited we were about presenting at October’s AWS re:Invent conference. For me, the best takeaways of that awesome session came from Soofi Safavi’s inspired delivery regarding his experience as CTO of a large enterprise (Radian Guaranty) applying Infrastructure as Code and Continuous Delivery. Nothing beats hearing from customers with actual experiences.
    What follows are my takeaways (the gold) from Soofi’s presentation.
    Soofi noted that Radian’s success is really defined by how fast they can absorb changes in the market they serve: private mortgage insurance. He noted how, since moving to AWS and applying principles of Infrastructure as Code, they can now more efficiently respond to those changes and ultimately stay competitive.
    Infrastructure as Code is one of the key drivers to achieve the agility, efficiency, and quality Radian requires. When they started on their journey, Soofi noted, they did not have that association. Here are some of his observations:

    Key Metrics

    Some key metrics that illustrated some of Radian’s gains:

    1. Team focus. Before: 60% of the team focused on maintaining hardware through software, and only 40% focused on building new capabilities or responding to regulatory changes. Today: Less than 10% focus on the mundane, and more than 90% focus on new capabilities and responding to regulatory changes.
    2. Loan processing time. Before: It took 4 to 6 hours to process loans in the past, and cost per page was in dollars. Today: Processing time is now less than 7 minutes and may be further reduced. The cost for processing has reduced to 1-2 cents vs. 1-2 dollars.
    3. Release frequency. Before: Once every quarter. Today: They are able to release multiple times in a day (examples of a sample week showed 42 times on a Monday, 38 times on a Tuesday, etc.).

    But the journey hasn’t been easy! Soofi noted three main areas to consider:

    1. Scope and timing of when you start thinking about Infrastructure and Code and CD. This is an explicit strategy you have to come up with and approach. Understand the areas of your business that can take advantage of it.
    2. Cultural shift. The techniques we talk about are different from what traditional developers who grew up before this era have ingrained. Specifically, the old way might look like this:
      1. Write code.
      2. Where is the server for me to deploy it?
      3. What is my plan to test it?
      4. What is the pipeline to move to production?

      CD requires the EXACT OPPOSITE of that process:

      1. It doesn’t matter what functionality you are actually pushing. You need to figure out your pipeline.
      2. It doesn’t matter what the functionality is, you need to build your test framework.
      3. It doesn’t matter what the functionality is, you need to figure out what basic infrastructure you are going to have.
      4. And the LAST thing you do is finally write up the code to perform the functions.

      That reversal, and absorption of those changes for enterprises, is a very powerful paradigm shift that needs to be contemplated.

    3. Limitations of enterprise applications. Not every enterprise application you might have in your organization’s inventory (think business intelligence, business rules, workflow, or policy administration systems, provided by the likes of SAP, TIBCO, etc.) can actually absorb the changes we talked about. Do not ignore this fact. Do not quietly let these applications create problems in your architecture. Spend time to rethink the overall architecture strategy and IT roadmap. That will be key to the success of automation and adoption infrastructure as code.

    Some Executive Advice

    Soofi then offered three bits of executive advice for enterprises thinking of embarking on this journey themselves:

    1. Consider your first step. The first time you hit that blue button that (innocently) says “Launch Instance” is the first technical debt you have accumulated! You must force yourself to avoid the AWS console. Yes, it is nice to experiment and get around, but it is the first step to get yourself into trouble. Instead, think about how you can manage your infrastructure as code—it is key to this process. Soofi joked that, as developers, Radian automates everybody else’s business, but developers remain the most manual of all. Must rethink the paradigm.
    2. Don’t underestimate the impact of people. The concepts, the paradigms of legacy thinking, are as fundamental as gravity. Pure education and training does not completely permit you to manage these changes. Radian identified blue hats and white hats. They moved people thinking the new way into a new environment. They Game-ified the changes. Finally, unfortunately, in some cases, they had to help some resources find a way to flourish elsewhere.
    3. IaC-able? Can it be decomposed? Every year there is a new concept. But there are three fundamental principles in infrastructure that are independent of the changing tools and technology: automation, metrics, and monitoring. These must be followed religiously, and if you do, it will result in a significant advantage for your enterprise:
      1. If it is not automated, it is not done.
      2. If it moves, you measure it.
      3. If it is not monitored, then it does not exist.

    I encourage you to find the time to view the entire presentation, which was so well attended it actually had a second/repeat session added the next day. (I wish that were recorded, because Soofi was even more on fire!) The complete presentation is on YouTube, and runs just under an hour. If you only have a few minutes, I recommend jumping in on Soofi’s part, which starts right around 39:00. For completeness, the slides are on Slideshare, and the complete source for the demo is on GitHub.