This article is part of the Continuous Delivery Blueprints series. It discusses how to go from no cloud infrastructure and no continuous integration set up to having a functioning Continuous Delivery Pipeline in Amazon Web Services. It discusses high level topics while also providing a reference implementation so it’s easy to follow along with. You can read part one here.
What we’re going to do today:
- • Set up Version Control System
- • Migrate code into Version Control System
- • Launch Jenkins Server
- • Explore Jenkins Server
With your AWS account set up and ready to go, it’s time to start setting up your pipeline. At the end of this article, you’ll have a functioning Continuous Integration server, and can start building out your pipeline.
Set up a Version Control System
To do any kind of continuous integration, delivery, or deployment, you need your source code in source control. It’s the foundation upon which everything else is built. If your source is already under a version control system, you can just skip this step.
Otherwise, it’s time to set up a source control system there are two options when it comes to source control: either you can build and run the server which your source control service lives on, or you can use a hosted source control provider.
Choosing a hosted provider gives you a lot of things — you can instantly start using it without having to configure any sort of server; you don’t have to worry about backups and disaster recovery; you’ll always be able to take advantage of the latest features. On the downside, you are giving your source code to another entity, which may not be an acceptable trade off at your organization. You’ll need to figure out which solution works best for you. If you’re not using source control already, this is probably the easiest option.
For the purposes of these blueprints, we’ll be using Github for our source hosting. If you’d like to set up your own git server, there are directions for that online.
There are other version control options (like Mercurial) and other hosted source options (like bitbucket). All major version control solutions will work with the Continuous Delivery Pipeline. That said, we’d advise against using Subversion. Subversion is easy to pick up, but once you really get going with it you can lose a lot of time dealing with merge headaches. Distributed version control systems like Git and Mercurial and better choices in the long run. (If you already have your source code in Subversion, don’t worry — you’ll still be able to do everything we talk about.)
Throughout the blueprints, we’ll be referring to our reference implementation project, CanaryBoard. It’s an open source project Stelligent built. It is currently hosted in Github, and if you’re just getting started, we recommend you use Github as well. Setting up an account with Github is pretty easy, here’s the account sign up form if you don’t already have an account. Note that if you use the free github account, you do not get any private repositories, and your code is open up to the world. If you need a private repository, Github offers paid plans with private repositories.
With an account set up, it’s time to import your code. If your organization doesn’t use source control, it’s a snap to get it added, just follow these instructions to set up a repo, copy your code in, and then commit the code and push it to github. Boom. Done.
If you’re not ready to import your code, you can fork the CanaryBoard repository and follow along using that as a reference implementation. To do that, just go to the CanaryBoard repo page, and look for the “Fork” button at the top right.
Set up your AWS Access Credentials
To interact with AWS programmatically, you need to provide it with access credentials. These are created via the AWS console and stored in environment variables for the scripts to use. Since we’re going to be running scripts to set up the Jenkins server, we need to create those keys now. To do this, log into the AWS Console:
- 1. Navigate to the IAM panel
- 2. Select “Users” from the left side
- 3. Select the user you want to create credentials for
- 4. In the lower pane, click the “Security Credentials tab”
- 5. Click “Manage Access Keys”
- 6. Click “Create Access Key”
- 7. Click “Download Credentials”
Now we have the credentials created, but we need to store them in environment variables. On Linux or OSX, use these commands (replacing the values with your actual key values)
export AWS_ACCESS_KEY_ID=ASDF1234567890ASDF export AWS_SECRET_ACCESS_KEY=qwerty1234567890qwerty1234567890qwerty
On Windows:
set AWS_ACCESS_KEY_ID=ASDF1234567890ASDF set AWS_SECRET_ACCESS_KEY=qwerty1234567890qwerty1234567890qwerty
Boom. Now we’re ready to programmatically create some computing instances.
Set up a Continuous Integration Server
So, with your source in source control, we’re ready to set up your Continuous Integration server. For the purpose of these blueprints, we’ll be using Jenkins. Jenkins is a free, open source Continuous Integration server, and it’s widely used. In fact, you might already be using Jenkins somewhere in your organization.
Jenkins is pretty easy to set up, but during these lessons we’ll be taking advantage of several Jenkins plugins and using a bunch of job configuration that is pretty static. Because of that, we’re able to provide you with a script to launch a Jenkins server. It creates an Opsworks stack, and then pulls in some Chef cookbooks we collected to set up a Jenkins server that’s ready for you to work on.
The first thing you’ll want to do is clone the Stelligent CDRI repo and run the Jenkins server script:
git clone https://github.com/stelligent/cdri.git cd cdri bundle install ruby bin/create_jenkins_server_stack.rb
(Don’t have Git and/or Ruby installed? Well then, you’ll probably want to follow these directions for installing Git or installing Ruby.) The script will take a couple of minutes to lay down everything it needs to set up a Jenkins server. After the script completes, though, it’ll still take a bit of time for the Jenkins server to build itself and be ready to go. The script actually has a couple options you may want to take advantage of:
$ ruby bin/create_jenkins_server_stack.rb --help Options: --region, -r <s>: The AWS region to use (default: us-west-2) --zone, -z <s>: The AWS availability zone to use (default: us-west-2a) --source, -s <s>: The github repo where the source to build resides (will not work with anything but github!) (default: https://github.com/stelligent/canaryboard.git) --size, -i <s>: The instance size to use (default: m1.large) --help, -h: Show this message
The default region is us-west-2. If you’re in North America, this is probably fine; otherwise you may want to switch it to a region closer to you. (A list of AWS regions is available here.) The default zone us us-west-2a, so if you change the region, you will need to change this as well.
The source repo is configurable, with a couple of caveats. The repository must be a github repository, and it must be a public repo. It is possible to have Jenkins connect to really any other repository type, and it is possible to set up username / password information or SSH keys to authenticate against a private repo. Both increase complexity, so to keep things simple, we’ve restricted the script to only run against public Github repositories.
Finally, you can configure the size of the EC2 instance that the Jenkins server will run on. A list of instance sizes and prices is available here. Note that anything smaller that a c3.large instance will take considerably longer to start up, but shouldn’t impact your ability to run builds at all. (However, Jenkins is a bit too heavy to run on a t1.micro instance, so don’t try going that low; the Chef scripts won’t even successfully run.)
How the Jenkins Server is Set Up
To set up the Jenkins server, we take advantage of two AWS services: CloudFormation and OpsWorks. These services handle a lot of the tedious bits about setting up AWS resources, and handle coordinating all the Chef calls necessary. AWS has a bunch of CloudFormation and OpsWorks documentation you can refer to, and any questions you have about it should be in there. We’ll just give you the highlights in the blueprints to get around the tools we provide; if you need more information, please refer to the documentation.
The first thing the script does is create a CloudFormation stack to create everything that the subsequent OpsWorks stack will need (security groups, roles, and policies). Once those are in place, it then sets up a new OpsWorks stack with a single custom layer and instance for the Jenkins server. If you go to the OpsWorks console, you’ll see a Jenkins stack with an instance starting. Once that instance turns green, your Jenkins server will be up and running. You can access it by clicking on the IP address in the instance details.
Since you’re just getting started, it’s probably a good idea to note that the server you just launched will cost you money for each hour it’s up. If you’re not going to be doing anything with it right away, you may want to stop it when you’re not using it.
Note: when you stop the instance through OpsWorks you’ll lose any configuration changes you’ve made to the server!
Exploring the Pipeline
The Jenkins server comes prepopulated with the beginnings of a continuous delivery pipeline. If you go to the “Continuous Delivery Pipeline” view, you’ll see all the different stages laid out. As you work through this series, you’ll build out each of these stages for your application. For now, let’s take a look at the first two stages, the trigger stage and the commit stage.
The trigger stage is a simple monitoring job. Its purpose is to look for changes in the repositories of your project, and if it detects a change, it kicks off a new run of the pipeline.
The commit stage handles building, packaging, and unit testing the application. It is supposed to run quickly (around five minutes) and give fast feedback to the developers. We will cover the commit stage in depth in the next part of this series.
Wrapping Up
If you didn’t already have source control setup, you do now. That’s huge! Source control is probably the most important part of a continuous delivery pipeline. We also set up a Jenkins server, to automate the different parts of our pipeline. With your code in source control, and your continuous integration server running, you’re ready to start building out the different stages of the pipeline to work with your application. In the next article, we’ll talk about how to set up your commit stage to build and test your application.
Questions? Comments? Reach out on twitter: @jonathansywulak