In this three-part series, I’ll explain ServerSpec infrastructure testing and how we use it at Stelligent, provide some concrete examples demonstrating how we use it at Stelligent, and discuss how Stelligent has extended Serverspec with custom AWS resources and matchers. We will also walk through writing some new matchers for an AWS resource. If you’re new to Serverspec, don’t worry, I’ll be covering the basics!

What is Serverspec?

Serverspec is an integration testing framework written on top of the Ruby RSpec dsl with custom resources and matchers that form expectations targeted at infrastructure. Serverspec tests verify the actual state of your infrastructure such as (bare-metal servers, virtual machines, cloud resources) and ask the question are they configured correctly? Test can be driven by many of the popular configuration management tools, like Puppet, Ansible, CFEngine and Itamae.
Serverspec allows for infrastructure code to be written using Test Driven Development (TDD), by expressing the state that the infrastructure must provide and then writing infrastructure code that implements those expectation. The biggest benefit is that with a suite of Serverspec expectations in place, developers can refactor infrastructure code with a high degree of confidence that a change does not produce any undesirable side-effects.

Anatomy of a server spec test

Since Serverspec is an extension on top of RSpec it shares the same DSL syntax and language features as RSpec. The commonly used features of the DSL are outlined below. For suggestions on writing better specs see the appropriately named betterspec.org

describe 'testGroupName' do
  it 'testName' do
    expect(resource).to matcher 'matcher_parameter'
  end
end

Concrete example:

describe 'web site' do
  it 'responds on port 80' do
    expect(port 80).to be_listening 'tcp'
  end
end

Describe blocks can be nested to provide a hierarchical description of behavior such as

describe 'DevopsEngineer' do
  describe 'that test infrastructure' do
    it 'can refactor easily' do
    end
  end
end

However, the context block is an alias for describe and provides a richer semantics. The above example can be re-written using context instead of nested describes.

describe 'DevopsEngineer' do
  context 'that test infrastructure' do
    it 'can refactor easily' do
    end
  end
end

There are two syntaxes that can be used for expressing expectations, should and expect. The should form was deprecated with RSpec 3.0, because it can produce unexpected results for some edge cases which are detailed in this Rspec blog post. Betterspec advises that new projects always use the expect syntax in expect vs should. The Serverspec website resources section advises that all of the examples use should syntax due to preference and that  Serverspec works well with expect. Even though the should syntax is more concise and arguably more readable, this post we will adhere to the expect syntax since the should syntax is deprecated and has potential for edge case failures. An example of both are provided because the reader is likely to see it both ways in practice.
Expect Syntax:

describe 'web site' do
  it 'responds on port 80' do
    expect(port 80).to be_listening
  end
end

Should Syntax:

describe port(80) do
  it { should be_listening }
end

A Resource, typically represented as a Ruby class, is the first argument to the expect block and represents the unit under test about which the assertions must be true or false. Matchers form the basis of positive and negative expectations/assertions as predicates on Resource. They are implemented as custom classes or methods on the Resource and can take parameters so that matchers can be generalized and configurable. In the case of the port resource its matcher be_listening can be passed anyone of the following parameters tcp, udp, tcp6, or udp6 like this be_listening ‘tcp’
Positive and negative case are achieved  when matchers are combined with either expect(…).to matcher or expect(…).not_to. Serverspec provides custom resources and matchers for integration testing of infrastructure code and Stelligent has created some custom resources for AWS resource called serverspec-aws-resources.  The following shows the basic expectation form and a concrete example.
Basic form:

expect (resource).to matcher optional_arguments

Example using port resource:

expect(port 80).to be_listening 'tcp'

Test driven development with Serverspec

At Stelligent, we treat infrastructure as code and that code should be written using the same design principles as traditional application software. In that spirit we employ test driven development (TDD) when writing infrastructure code. TDD helps focus our attention on what the infrastructure code needs to do and sets a clear definition of done. It promotes a lean and agile development process where we only build enough infrastructure code to make the tests pass.
A great example of how Serverspec supports test driven development is when writing our chef cookbooks that are used to build AMIs on some projects at Stelligent. Locally a developer can utilize Serverspec, Test Kitchen, Vagrant and a virtualization provider to write the Chef cookbook. A great tutorial about getting setup with these tools and writing you first test is available in Learning Chef: A Guide to Configuration Management and Automation.
Using this toolchain allows the developer to adhere to the four pillars of test-driven infrastructure — writing, running, provisioning, feedback. These pillars are necessary to accomplish the red-green-refactor technique the basis of TDD.
First, we make it RED by write a failing Serverspec test and run kitchen verify. Test Kitchen then takes care of running the targeted platform on the virtualization provider, provisioning the platform and providing feedback. In this case we have not written the Chef cookbook to pass the Serverspec test so Test Kitchen will report a failure for the feedback.
Then, we make it GREEN by writing a cookbook that minimally satisfies the test requirements. After the node is converged with the Chef cookbook we can run kitchen verify and see that the test passes. We are now safe to refactor with the confidence that even a trivial change will not produce an unintended side-effect.
The same Serverspec tests that were used locally to test are then used during the build pipeline to verify the actual state of the infrastructure resources in AWS. On every commit the continuous delivery pipeline executes the Serverspec integration tests against running provisioned resources and fails the build stage if an error occurs. This automated testing provides short iteration cycles and high degree of confidence on each commit. This supports an agile methodology for infrastructure development and an evolutionary architecture process.

How we utilize Serverspec at Stelligent

Provisioning of AMIs

Provisioning AMIs in the continuous delivery pipeline is done with a combination of Packer and Chef cookbooks. Packer is a tool that is used in our continuous delivery to build AWS machine images using a declarative json document that describes the provisioning. The Chef provisioner is declared in the Packer config file along with the Chef runlist. Building the AMI is typically done in three stages build-test-promote. During the build stage Packer is run which launches the AMI that serves as the base AMI and then executes each of the provisioners in the Packer config file. In the case of the Chef provisioner it converges the running EC2 node. When Packer successfully completes the AMI is stored in AWS and the ID is passed to the test stage.
The test stage launches a new EC2 instance with the AMI ID from the build step and  executes the Serverspec test against the running EC2 instance. If the tests pass then the EC2 instance is promoted and stored in AWS, available for other pipelines to utilize as an artifact. In order to promote code reuse and speed up AMI pipelines we have broken down AMI creation into multiple pipelines that depend on the results of another AMI pipeline. It look something like this  base-ami > hardened-ami > application-ami. Each of these AMI pipelines have there own Serverspec test that ensure the AMI’s requirements are met.  We do the same thing for our Jenkins servers, with additional test to verify that the instances jobs are configured correctly.

Documenting existing servers

We document existing infrastructure before moving it to AWS. A pattern that we use is to elicit a client specification of the existing infrastructure and then translate it to an executable Serverspec definition. This allows us to test the spec against the existing infrastructure to see if there are any errors in the client provided specification.
For example, we may be told they have a Centos OS instance running a Tomcat web server to serve their web application. The specification is translated into a suite of Serverspec test and executed against the existing and results in a failure for a test that Tomcat is installed. After debugging the instance we find that it is actually running JBoss application server. We then update the test to match reality and execute it again. After this explorative process is done we have an executable specification that we can use to test the existing infrastructure, but also use as a guide when building our new infrastructure in AWS.

In conclusion

Serverspec is an invaluable tool in your toolkit to support Test Driven Infrastructure . It’s built on top of the widely adopted Ruby RSpec DSL. It can be executed both locally to support development, and as part of your continuous delivery pipeline. It allows you to take a test driven development approach to writing infrastructure code. The resulting suite of tests form a specification, that when automated as part of CD pipeline provide a high degree confidence in the infrastructure code.

Stelligent Amazon Pollycast
Voiced by Amazon Polly