Blog

DevOps on AWS Radio: mu – DevOps on AWS tool (Episode 10)

In this episode, Paul Duvall and Brian Jakovich cover recent DevOps on AWS news and speak with Casey Lee from Stelligent about the open-source, full-stack DevOps on AWS tool called mu.

Here are the show notes:

DevOps on AWS News

Episode Topics

  1. What is mu and what problem does it solve? What are its benefits?
  2. How does someone use mu (including prereqs)?
  3. What types of programming languages and platforms are supported?
  4. What types of AWS architectures does mu support (i.e. traditional EC2, ECS, Serverless, etc.)?
  5. Which AWS services are provisioned by mu?
  6. Does mu support non-AWS implementations?
  7. What does mu install on my AWS account?
  8. Describe mu’s support for configuration/secrets
  9. Extensibility?
  10. Price?
  11. What’s next on the mu roadmap?
  12. How can listeners learn more about mu?

Additional Resources

About DevOps on AWS Radio

On DevOps on AWS Radio, we cover topics around applying DevOps principles and practices such as Continuous Delivery on the Amazon Web Services cloud. This is what we do at Stelligent for our customers. We’ll bring listeners into our roundtables and speak with engineers who’ve recently published on our blog and we’ll also be reaching out to the wider DevOps on AWS community to get their thoughts and insights.

The overall vision of this podcast is to describe how listeners can create a one-click (or “no click”) implementation of their software systems and infrastructure in the Amazon Web Services cloud so that teams can deliver software to users whenever there’s a business need to do so. The podcast will delve into the cultural, process, tooling, and organizational changes that can make this possible including:

  • Automation of
    • Networks (e.g. VPC)
    • Compute (EC2, Containers, Serverless, etc.)
    • Storage (e.g. S3, EBS, etc.)
    • Database and Data (RDS, DynamoDB, etc.)
  • Organizational and Team Structures and Practices
  • Team and Organization Communication and Collaboration
  • Cultural Indicators
  • Version control systems and processes
  • Deployment Pipelines
    • Orchestration of software delivery workflows
    • Execution of these workflows
  • Application/service Architectures – e.g. Microservices
  • Automation of Build and deployment processes
  • Automation of testing and other verification approaches, tools and systems
  • Automation of security practices and approaches
  • Continuous Feedback systems
  • Many other Topics…

Introduction to NixOS

NixOS, and declarative immutable systems, are a great fit for CI/CD pipelines.  With the entire system in code, ensuring and auditing reproducible environments becomes easy.  Applications can also be “nixified,” so both system and application are fully declarative and in version control. The NixOS system is mounted read-only, which makes for a good fit in immutable autoscaling groups.  Instance userdata may contain a valid NixOS configuration, which is assumed on boot, so as to implement any necessary post-bake changes.

“NixOS. The Purely Functional Linux Distribution. NixOS is a Linux distribution with a unique approach to package and configuration management. Built on top of the Nix package manager, it is completely declarative, makes upgrading systems reliable, and has many other advantages.”
https://nixos.org/

There are four main parts to Nix and NixOS (Expressions, OS, Modules, and Tests). We will examine each one:

The Nix expression language

This is the “package manager” functionality of NixOS, which can be downloaded from here. Each open source project, including the Linux OS, has a “nix expression” that describes how it is to be built. This includes explicitly declaring each dependency. Each dependency is also declared with a nix expression, so the entire system is declarative. All Nix packages exist in the Nixpkgs Github. Here is the one for ElectricsSheep, a distributed screen saver for evolving artificial organisms:

The underlying mechanism of dependency and package management is a system of symbolic links. Packages are built and deployed in an immutable “nix store”. This read only location exists at /nix/store. In this location, the package name has a hash added, which is computed as a result of evaluating all build input dependencies. Therefore, we can have the same package available many times, each with a unique hash, and a unique version of dependencies. Symbolic links from /run/current-system/sw/bin/ to the hashed package name in nix store determine which package is called, as /run/current-system/sw/bin/ would be in the user’s $PATH.

Should any change to a packaging expression happen, all packages depending on it would rebuild. If a mistake is found, it is easy to revert the change to the dependency, and rebuild. Ensuring the reproducibility of system packages is a huge win, and this solution to “dependency hell” works very well in practice. The Nix package manager can be run on any Linux distribution, such as Fedora or Ubuntu, and also works on Darwin/OSX as well. Custom code can also be packaged with nix expressions, and so both the app and the OS are fully declarative and reproducible.

The NixOS Linux distribution

Nix packaging of open source software, including Linux kernel and boot processes, make up the NixOS Linux distribution. Official releases are available online. NixOS Channels are the method for specifying which version of NixOS is to be installed. NixOS is controlled by the /etc/nixos/configuration.nix file, which declaratively defines the NixOS environment, including defining which NixOS channel is to be used on the system. Whenever configuration.nix is updated, a nixos-rebuild switch can be executed, which switches to the new configuration immediately, as well as adding a new “generation” to Grub/EFI, so the new version can be booted. Here is an example configuration.nix:

As of NixOS 16.03, AWS EC2 Instance Metadata support is built in. However, instead of the usual Cloudinit directives, the NixOS instance expects the Userdata to be a valid configuration.nix. Upon boot, the system will “switch” to what is defined in the provided configuration.nix via EC2 Userdata. This allows for immutable declarative instances in AWS AutoScaling Groups.

NixOS Configuration Management Modules

NixOS modules define how services, via SystemD, are to be configured and run. Modules are written so that parameters can be set which correspond how the systemd process is to be run.

This is the Buildbot NixOS Module, which writes out buildbot configuration, based on module parameters, and then ensures the service is running:

NixOS Tests

NixOS tests are a mechanism to ensure NixOS expressions and modules are working as expected.  Virtual machine(s) are spun up so as to perform the declared tests. Currently VM’s spun up for testing use the QEMU hypervisor, although NixOS tests are moving to libvirt, ideally supporting autodetection of available system virtualization technologies. As an example, here is the buildbot continuous integration server test:

After downloading and building all dependencies, the test will perform a build that starts a QEMU/KVM virtual machine containing the nix system.  It is also possible to bring up this test system interactively to facilitate debugging. The virtual machine mounts the Nix store of the host which makes vm creation very fast, as no disk image needs to be created. These tests can then be implemented in a continuous integration environment such as buildbot or hydra.

In addition to declaratively expressing and testing system packages, applications can be nixified in the same way.  Application tests can then be written in the same manner, so both system and application can go thru a continuous integration pipeline. In a Docker microservices environment, where applications are defined as immutable containers, NixOS is the perfect host node OS, running the Docker daemon and Nomad, Kube, etc.

NixOS is in fast active development. Many users are also NixOS contributors, so most nix packaging of open source projects stay up-to-date. Unstable and release channels are available. Installation is very well documented online. The ability to easily “switch” between configuration versions, or “generations,” which include their own grub/efi boot entry, makes for a great workstation distro.  The declarative reproducibility of a long term stable release, with cloudinit userdata support, makes for a great server distribution.

Thanks for reading,
@hackoflamb

DevOps in AWS Radio: Goss (Episode 9)

In this episode, Paul Duvall and Brian Jakovich cover recent DevOps in AWS news and speak with Ahmed Elsabbahy about Goss, a ServerSpec alternative for testing server configuration.

Here are the show notes:

DevOps on AWS News

Episode Topics

  1. What is Goss?
  2. Why was Goss created?
  3. Why would you use Goss over serverspec or other server configuration testing tools
  4. Where does Goss fit into a continuous delivery pipeline
  5. How does Goss work with AWS?
  6. How does Goss work with production testing?
  7. Where can we find out more information about Goss?

Additional Resources

About DevOps in AWS Radio

On DevOps in AWS Radio, we cover topics around applying DevOps principles and practices such as Continuous Delivery in the Amazon Web Services cloud. This is what we do at Stelligent for our customers. We’ll bring listeners into our roundtables and speak with engineers who’ve recently published on our blog and we’ll also be reaching out to the wider DevOps in AWS community to get their thoughts and insights.

The overall vision of this podcast is to describe how listeners can create a one-click (or “no click”) implementation of their software systems and infrastructure in the Amazon Web Services cloud so that teams can deliver software to users whenever there’s a business need to do so. The podcast will delve into the cultural, process, tooling, and organizational changes that can make this possible including:

  • Automation of
    • Networks (e.g. VPC)
    • Compute (EC2, Containers, Serverless, etc.)
    • Storage (e.g. S3, EBS, etc.)
    • Database and Data (RDS, DynamoDB, etc.)
  • Organizational and Team Structures and Practices
  • Team and Organization Communication and Collaboration
  • Cultural Indicators
  • Version control systems and processes
  • Deployment Pipelines
    • Orchestration of software delivery workflows
    • Execution of these workflows
  • Application/service Architectures – e.g. Microservices
  • Automation of Build and deployment processes
  • Automation of testing and other verification approaches, tools and systems
  • Automation of security practices and approaches
  • Continuous Feedback systems
  • Many other Topics…

Devops Benefits of Infrastructure as Code

Infrastructure and operations as code is an essential practice for realizing the advantages of modern clouds.  For enterprises looking to migrate to Amazon Web Services, Azure, or Google Cloud Platform, scripted infrastructure and automation are the key first steps through which other devops practices become accessible.  This post will enumerate some key benefits that become possible once we embrace infrastructure as code practices.

By codifying our infrastructure, we enable better testing and quality control, more efficient and predictable deployments, and decreased recovery times. It provides improved testability and monitoring, lowers the cost of experimentation and innovation, makes deployments more predictable, and decreases the mean time to resolution (MTTR) for issues.

Automate your deployment and recovery processes

With infrastructure automation, reproducible environments become possible.  We can use the same automation scripts to deploy exact copies of production to development, test, and production environments.  With these consistent deployments, we are able to achieve the ever-elusive development-to-prod parity, finally putting an end to the “it worked on my machine!” problems.

The pinnacle of infrastructure automation is the Blue/Green deployment strategy.  This strategy enables zero downtime deployments and allows us to run live tests before releasing our changes to our users.  Blue/Green Deployments take advantage of our ability to run exact copies of our environments in parallel.  By controlling when traffic is routed to our new copy, we can defer a release until we are 100% confident that our new environment is ready.

In a Blue/Green deployment, we deploy a new, isolated copy of our environment.  This new, copied environment is named Green.  It is our release candidate.  It contains our new changes and is isolated from the live environment, which we call Blue.  The Green environment is configured for production and is ready to go live, but it is launched darkly – that is, no traffic is routed to Green.  

Next, we run our acceptance tests against the live Green environment.  If we encounter an error, we can simply log the error, remove the Green environment and go back to the drawing board.  No users ever know a difference, as we never routed any live traffic to Green.  

If our acceptance tests do pass, we promote our Green environment to be the new live environment.  This can be done by changing a DNS entry to point at the Green environment or by removing the Blue environment from our load balancer and adding the Green environment to the load balancer.  

The Blue environment does not need to be automatically deleted.  If necessary, we can keep it around for a short grace period in case we need to rollback.  The rollback process would consist of reversing the traffic swap to point back at Blue.

This is merely an overview of the Blue/Green deployment strategy.  For an in-depth discussion of Blue/Green techniques in an AWS environment, see the AWS whitepaper on the topic.

Rollback with the same tested processes

Our deployment scripts are also our rollback scripts.  Because our deployments are automated, we can reproduce the state of the infrastructure any number of times by simply re-running the deployment scripts with the same inputs.  With our codified infrastructure, we can reach back in version control to grab any commit since the repository began.  By reverting to the desired commit and re-running our deployment scripts, we can restore the state of the infrastructure as it was on any given day.

Don’t Repair, Redeploy

Server time is cheap, but engineer time is expensive.  Further, troubleshooting server performance issues can be very time-consuming.  For these reasons, it no longer makes sense to troubleshoot and repair our servers.  Rather it is now more economical to destroy the old server instance and replace it with a new, working copy.

We can use our automated deployment scripts to deliver working servers to replace broken and impaired servers.  We can now follow an immutable infrastructure pattern, in which nothing ever changes on a server after it is deployed.  This helps avoid the problem of configuration drift and also greatly simplifies our operations.  Now, the only repair operation is to redeploy the service.  A service crashed?  Redeploy.  Having performance issues on a host?  Redeploy.  Lost connectivity to a host?  Redeploy.

Focus on Mean Time To Recovery

They say you can’t fix what you don’t measure, but it’s important to choose the right metrics to measure and improve upon. To traditional IT organizations, the key metric is Mean Time Between Failures (MTBF).   Server uptime is paramount, and this is the metric that gets optimized.  This leads to a reluctance to accept changes, as each change can potentially introduce a failure.  Moreover, configuration changes are generally made manually by administrators.  This leads to long-running  snowflake servers which are virtually impossible to reproduce.  This presents a very nasty challenge in restoring service availability when the inevitable failures do occur.  

Failure of an IT component means the organization is losing money.  But failures do and will happen.  In a cloud-native world, we solve this problem by turning it on its head.  Rather than trying to avoid failures, devops organizations accept that failures are a part of life and design our applications to minimize the impact of those failures by recovering gracefully.  To accomplish this, we focus on Mean Time To Recovery (MTTR) as our key metric.  By minimizing the time it takes to recover from failure, we minimize the impact of each failure.  Optimizing for MTTR necessitates automation of our processes.  Our recovery processes must be consistent and reliable.

Practice makes perfect

If we want to improve at anything, we have to practice.  Recovering from failures is no different.  We do not want the first test of our recovery processes to be during an actual disaster.  Rather, we want to test our recovery process numerous times before we actually need it.  Doing so gives us confidence that our recovery process will work as intended and restore the availability of our service.

Traditionally, creating an isolated environment for disaster recovery was too cost-prohibitive and time-consuming to be a feasible strategy.  The only way to test our process was to actually have a disaster.  However, with modern cloud environments, we no longer have this limitation.  Creating a new environment is an api call away.  Once we’ve codified our infrastructure, we can create a copy of our production environment by running the same code we used to create production.

We create our new copy environment to be totally isolated from our production environment.  We are now free to simulate disasters and test our recovery processes.  This can be done regularly in a low stress environment, allowing our engineering teams to troubleshoot and strategize without the added pressure of an actual outage.

Each time the process fails, we learn a little bit more.  We can then use this information to correct the problem and improve our automated recovery scripts.  At the very least, we document the known issues and add the solutions to common problems in our standard procedures.  

We should practice these failures regularly.  By the time an actual disaster occurs, we should have multiple practice runs of recovering from the disaster, as well as hundreds or even thousands of trial runs from the deployments being run with the same scripts.

Use testing tools to verify your infrastructure

With our infrastructure codified and our restore process automated, the next step is to design a set of automated tests that will verify.  Because we now think of our infrastructure as a software application, we should use software testing tools to test our infrastructure.  By using tools like Python’s Behave or Ruby’s Rspec, we can test that our service is behaving as expected.

These tests don’t have to be complicated, and can start out very simply.  The first test can just be “Is the service up and reachable?”  After all, this is the entire goal of the software project – if it is not up and working, it is of no use.  Then we can start to further refine our tests to include those behaviors we expect a healthy service to exhibit.  A good starting point is to hit each of our service’s endpoints in an automated fashion.  These basic tests give us a high level of certainty that the app is behaving as expected, and we can add more detailed testing to test for specific failure cases.

As we practice our failures and recovery process, we will find new issues that can cause our system to not operate correctly.  As these issues are discovered, we test for them and add those tests to our suite of automated tests.  These tests also double as regression tests.  When a new feature is added and a test breaks, we know exactly which change caused the service tests to fail.  As time goes by, we build a more comprehensive test suite and incrementally increase our confidence in our recovery process.  

Hook your tests into your monitoring system

Our automated test suite gives us confidence that our service is behaving correctly during deployment and recovery.  In these situations, the conditions are known and assumptions can be hidden.  But what happens when our service is used in unexpected ways, as is bound to happen when real users start using the application?  We can hook these tests into our monitoring systems and run these tests on a periodic basis.  In this way, we can be alerted the moment something goes wrong.  Running our tests in this fashion will allow us to test against real world scenarios.  This is our first line of defense in detecting real world errors.  

Conclusion

You don’t have to be a Netflix or Airbnb company to take advantage of devops practices. Fortune 500 companies and government agencies are adopting these patterns so that they can recover from failure more quickly, deploy more often, deploy more quickly.  The prerequisite to practicing these modern devops techniques is Infrastructure as Code.  If your organization is looking to begin capitalizing on the benefits of modern clouds but does not know where to start, codifying infrastructure should be the first step.  

Are you looking for guidance transitioning your legacy apps to AWS?  Stelligent can help!  Stelligent Migrate is our service in which we help facilitate the migration of your enterprise workloads to AWS.  If you have any questions or are interested in how Stelligent can help, please reach out to sales@stelligent.com!

WordPress, Mu, and You

Today, we’re going to give you the “Hello, world” of web stacks and explain how you could run WordPress with mu. This is the fifth post in Stelligent’s blog series on mu, where we’ll lay out a path for using mu to run WordPress and try to hit a target that’s often hard to find in that part of our world: managing a WordPress stack with infrastructure as code, utilizing a continuous delivery pipeline. In that vein, we’re going to keep the feature list as short and simple as possible. To make this easier to follow, we’re going to keep the feature list short and simple. Look for future posts in the Stelligent blog where we’ll talk about adding more functionality.

Why would I want to run WordPress with mu?

mu will take care of a lot of nice things for you. It creates a CodePipeline stack that will manage your deployment workflow, and you’ll have test and production environments out of the box. You can inspect changes in your test instance before you let them continue through the pipeline.  It will manage your databases in RDS, so your backend data layer gets its own high availability and scaling that’s independent of your other resources; your test and production data are also kept separate. Each environment will run in Amazon’s EC2 Container Service, behind a load balancer, so the pipeline can roll out new containers without bringing the old ones down. When it’s all done, you can even export CloudFormation templates for your infrastructure and manage them on your own — you’re never locked into using mu!

We’re going to deploy a Docker container from the official WordPress image.  How do we get there? mu will manage a CodePipeline stack that orchestrates everything: it creates a custom image with your web data and the official image, then uses ECS to deploy a test instance for you to inspect, and then — if you approve it — rolls it out to production. Because we’re using mu, all of this will be done through code, and we’ll use GitHub to make that code available to CodePipeline.

Stelligent created mu to simplify using microservices on AWS. It gives you a simple, flexible front-end to some really sophisticated services that can seem bewildering until you’ve gained a lot of experience with them. All those advanced products from AWS are themselves hiding the complexity of running very robust services across data centers around the world. There is vast machinery hidden behind this relatively simple command-line interface. You benefit from all of this with a really simple process for deploying your WordPress site, and you get to do it with tools native to most developers: a code editor and a command line.

Getting Started

Let’s get down to the nitty gritty. To get started, fork our repository into your own GitHub account then clone it to your workstation. Click the “Fork” button in the top right corner of the page at stelligent/mu-wordpress:

Credit to GitHub; taken from their

Next, clone the new repo from your account to your local workstation:

git clone _your_fork_of_mu-wordpress_
cd mu-wordpress

In this directory, you’ll find a few files. Edit mu’s main config file, mu.yml, changing pipeline.source.repo to point to your own GitHub account instead of “stelligent”:

pipeline:
  source:
    provider: GitHub
    repo: _your_github_username_/mu-wordpress

Commit your changes and push them back up to your GitHub account:

git add mu.yml
git commit -m 'first config' && git push

You’re ready to start up your pipeline:

mu pipeline up

mu will ask you for a GitHub token. CodePipeline uses it to watch your repo for changes so that it can automatically deploy them. Create a new token in your own GitHub account and grant it the “admin:repo_hook” and “admin” permissions. Save it somewhere, like a nice password manager, and then provide it at mu’s prompt.

Now you can watch your pipeline get deployed:

mu pipeline logs -f

Give it a little time – it will probably take about 10 minutes for the pipeline to deploy all the resources. When it’s done, you’ll have your first environment, “test”:

mu env list

You should see a table like this, but it won’t say CREATE_COMPLETE under each environment until they’re done.

+-------------+-----------------------+---------------------+-----+
| ENVIRONMENT |         STACK         |       STATUS        | ... |
+-------------+-----------------------+---------------------+-----+
| test        | mu-cluster-test       | CREATE_COMPLETE     | ... |
+-------------+-----------------------+---------------------+-----+

On to WordPress!

Now that you have a test environment, you can initialize WordPress and make sure it’s working. Start by inspecting “test”:

mu env show test

You’ll see a block at the top that includes “Base URL”:

Environment:    test


Cluster Stack:  mu-cluster-test (UPDATE_IN_PROGRESS)
VPC Stack:      mu-vpc-test (UPDATE_COMPLETE)
Bastion Host:   1.2.3.4
Base URL:       http://mu-cl-some-long-uuid.us-east-1.elb.amazonaws.com

Append “/wp-admin” to that and load the URL in your browser:

http://mu-cl-some-long-uuid.us-east-1.elb.amazonaws.com/wp-admin

Follow the instructions there to set up a WordPress admin user and initialize the database.

First WordPress admin initialization page

Deploy to “production”

If it looks to you like it’s working, load up your CodePipeline console. You can find the URL for it by asking mu for information on your service:

mu service show | head

The first line of output will give you the URL for your pipeline stack:

Pipeline URL:   https://console.aws.amazon.com/codepipeline/home?region=...

Load that page in your web browser. Find the “Production” stage. The first step there is “Approve”, and you’ll see a button labeled “Review”. Fill that in – any text will do – and click the “Approve” button:

Approve or reject a revision in CodePipeline

CodePipeline will deploy a new copy of the container it build into your “prod” environment. When it’s done, you can do the same thing you did with test to initialize WordPress and inspect your new web site. When you’re ready, try these commands to watch it go and get the site URL:

mu pipeline logs -f
mu env show prod |head

Behind the scenes

If you’ve made it this far, I’ll bet you’re wondering what mu is actually doing for you. Let’s take a look. mu brings together a handful of Amazon products. It will:

Those are the pieces. How do they all fit together?

  • GitHub stores your infrastructural code.
  • mu acts as your front-end to AWS by generating and applying CloudFormation templates, orchestrated by CodePipeline.
  • CodeBuild combines your content with the official WordPress Docker container, storing the new image in Amazon ECR, then deploying it through ECS.
  • An ECS cluster is run for each environment we define: in this case, “test” and “prod”.
  • An AWS ALB sits in front of each cluster.
  • Your WordPress database will be provided by an Amazon RDS cluster, one for each environment. Each runs Aurora, Amazon’s highly optimized clone of MySQL.

 

Architectural diagram of mu-wordpress

Let’s talk about how your information flows through those pieces. AWS CodePipeline orchestrates all the other services in AWS. It’s used to give you a continuous delivery pipeline:

AWS CodePipeline pipeline created by mu

  1. It watches your GitHub repo for changes and automatically applies them shortly after you push.
  2. AWS CodeBuild uses buildspec.yml to run any custom steps you add there.
  3. AWS CodeBuild generates your own Docker image by combining the results of the last step with the official WordPress image and storing it in Amazon ECR.
  4. Your container is deployed to your “test” environment.
  5. You manually inspect your container and approve or reject it.
  6. If you approve it, your container is deployed to your “prod” environment.


Wrapping it up

If you want to customize your WordPress site, add content to the html directory of your repo. Anything there will be installed with WordPress as the container is built, so you have a simple way to add content to your site’s wp-content directory by adding new plugins or themes to your own codebase. Take a look at your mu.yml file, too – it includes comments that explain the settings used and suggest some things you might want to change, like the instance size deployed to each environment.

We’ve shown a fast and simple way to get WordPress running at AWS. There’s a lot of tech running behind the scenes, but mu makes it easy to get started and navigate that complexity. There’s a lot more we could do: optimizing WordPress and making your stack robust are extensive topics that deserve their own articles, and much has been written about them elsewhere. Look for future posts on the Stelligent blog as we improve our mu-wordpress code, too.

Additional Resources

Are you passionate about working with the latest AWS technologies? Are you interested in helping build tools to help automate deployments? Do you want to engage in spirited debate about how to pronounce Greek letters? Stelligent is hiring!

Vault : A tool for managing secrets.

Over the last few weeks, we’ve been reminded once again of why Cyber Security is of the utmost importance. Not only can security breaches cause a denial of service but they can also cause loss of intellectual property, espionage and many other embarrassing incidents. As a member of IT staff, instead of relying just on security team, every member should also be proactive in identifying security gaps.

In this blog post, we will take one small proactive measure that can go a long way: database passwords.

We’ve seen this often: mission critical application store passwords to mission critical database in a config file, which is copied around on multiple servers and checked-in into the source code repository. More often than not, the password in this config file is unencrypted. This setup not only makes access to your mission critical data vulnerable to outside threats but also to a disgruntled employee, or other insider threats.

There is a better solution, one which addresses both insider and outsider threats. We’ll address the solution to both of these problems using Hashicorp Vault. Although Vault has many use cases, in this blog we’ll address the specific use case of managing database passwords and best practices in doing so.

Let us first quickly discuss the key features of Vault and how we’re going to leverage them in our solution.

  • Secret Storage: Vault can store secrets in memory (non-persistent) or use third party (persistent) store. In either case, secrets are encrypted before storing. In our case, we’ll use Hashicorp Consul for the persistent secret store.
  • Dynamic Secrets: Vault allows to create on-demand secrets. We will create passwords dynamically to connect to a database; in our case, this will be a MySQL database.
  • Leasing, Renewal and Revocation: Vault allows password rotation based on lease associated with them. In a case of intrusion, Vault allows key revocation for system lockdown. For our demonstration, we will renew the lease with every database connection. However, in production one can set the renewal period based on their security and organizational policies.

Now that we understand Vault features, let’s look at how we can use it in a scenario where a Java application wants to connect and read data from MySQL database. Before we start using Vault we need to set it up and configure it. This section is typically done by system administrators. However, with Vault there is one major difference: the almighty “unseal” process.

Vault Setup

Depending on your system download and install Vault (install instructions) and Consul (install instructions).

Follow the step by step instructions provided below for Vault setup. Vault init (step-3) is done only once when the server is started for the first time with a secret storage (Consul) that has never been used before.

The Vault init step in incredibly important and as a best practice, it must be done in presence of a few other stakeholders in your organization. Vault init steps outputs unseal keys and initial root token which should be distributed among stakeholders. Vault splits the root token into multiple unseal keys using “Shamir’s Secret Sharing” algorithm.

Once initialized, a quorum is needed to read configuration information from vault. The number of unseal keys needed to establish a quorum can be set by passing the parameter “key-shares” and the maximum number of unseal keys generated by Vault can be set by passing parameter “key-threshold”. So in our case, three out of five stakeholders are needed to form a quorum. The root token should be destroyed because, if needed, Vault can issue one-time-root token by using the unseal keys.

In essence by chunking the root token (aka root password) into multiple unseal keys and distributing those within multiple individuals we’re protecting against a single individual hijacking our corporate data.

MySQL Database Staging

We need to create a user in MySQL database which will be used by Vault to login and dynamically create users based on access policies and lease time we set in Vault. In our case, we’ll create user “vaultadmin”.

Vault Database (MySQL Secret Backend plugin) Setup

Secret backend help store and generate secrets dynamically. In our case we’ll use database secret backend and MySQL plugin to create database credentials dynamically based on configured access control policies.

Now that Vault is setup and configured, we need create a mechanism to let our application authenticate to Vault so that it is able to read the database credentials. For our purpose we will use token-based authentication:

Spring-Boot Application Setup

We should use the authentication token created above in the spring boot bootstrap.properties file.

The source code for application can be found at https://github.com/stelligent/MySQL-Vault. The source code uses spring cloud libraries, specifically artifacts “spring-cloud-starter-vault-config” and “Spring-cloud-vault-config-databases” to connect to Vault and MySQL database. 

Application to Database Authentication Workflow

vault-mysql-workflow

  1. Application uses token and role to authenticate to Vault.
  2. Vault validates the token and role against roles and policies stored in secret storage. If validated, the workflow continues; otherwise access is rejected.
  3. Vault secret/database backend connects to MySQL database using connection information from step 2 of Vault Database Setup.
  4. Vault secret/database issues SQL statements to create user in MySQL database based on configuration in step 3 of Vault Database Setup.
  5. Vault passes the username/password information to application. Application can then access the database until the lease time expires. You can set the lease to expire per your organization’s password rotation policy. Once a lease expires, the credentials won’t be valid and the application will need to retake steps 1 through 5.

Using Vault it’s possible to have applications use dynamically generated encrypted passwords with a key rotation policy which is good for security and compliance. Moreover Vault itself provides protection against single user accessing corporate secrets. Using Hashicorp Vault is much more secure  than passing passwords around in configuration files.

Did you find this post interesting? Are you passionate about working with the latest AWS technologies? If so, Stelligent is hiring and we would love to hear from you!

DevOps in AWS Radio: Serverless (Episode 8)

In this episode, Paul Duvall and Brian Jakovich cover recent DevOps in AWS news and speak with Mike Roberts and John Chapin from Symphonia about Serverless architectures, DevOps, and AWS.

Here are the show notes:

DevOps in AWS News

Episode Topics

  1. Pros and Cons of Serverless architectures
  2. Symphonia’s Serverless speciality
  3. How is DevOps and Continuous Delivery fit into Serverless
  4. Continuous Experimentation and Serverless architectures
  5. Types of applications or services are most suitable or not suitable for Serverless
  6. O’Reilly report: “What is Serverless?”
  7. Serverless architectures resources and people in the space
  8. Vendor lockin

Additional Resources

About DevOps in AWS Radio

On DevOps in AWS Radio, we cover topics around applying DevOps principles and practices such as Continuous Delivery in the Amazon Web Services cloud. This is what we do at Stelligent for our customers. We’ll bring listeners into our roundtables and speak with engineers who’ve recently published on our blog and we’ll also be reaching out to the wider DevOps in AWS community to get their thoughts and insights.

The overall vision of this podcast is to describe how listeners can create a one-click (or “no click”) implementation of their software systems and infrastructure in the Amazon Web Services cloud so that teams can deliver software to users whenever there’s a business need to do so. The podcast will delve into the cultural, process, tooling, and organizational changes that can make this possible including:

  • Automation of
    • Networks (e.g. VPC)
    • Compute (EC2, Containers, Serverless, etc.)
    • Storage (e.g. S3, EBS, etc.)
    • Database and Data (RDS, DynamoDB, etc.)
  • Organizational and Team Structures and Practices
  • Team and Organization Communication and Collaboration
  • Cultural Indicators
  • Version control systems and processes
  • Deployment Pipelines
    • Orchestration of software delivery workflows
    • Execution of these workflows
  • Application/service Architectures – e.g. Microservices
  • Automation of Build and deployment processes
  • Automation of testing and other verification approaches, tools and systems
  • Automation of security practices and approaches
  • Continuous Feedback systems
  • Many other Topics…

Service discovery for microservices with mu

mu is a tool that makes it simple and cost-efficient for developers to use AWS as the platform for running their microservices.  In this fourth post of the blog series focused on the mu tool, we will use mu to setup Consul for service discovery between multiple microservices.  

Why do I need service discovery?

One of the biggest benefits of a microservices architecture is that the services can be deployed independently of one another.  However, this presents a new challenge in that it becomes difficult for clients to know the list of containers to use when invoking the service.  Here are three different approaches to address this challenge:

  • Load balancer per microservice: Create a load balancer for every microservice and add/remove containers to the load balancer as deployments and scaling events occur.  The endpoint address of the load balancer is then shared with clients through some manual process.

cloudcraft - Microservices - multip.png

There are three concerns with this approach.  First, the endpoint address of the load balancer must never change or else all the clients will be broken and require updates to take the new endpoint address.  This can be addressed via DNS CNAME records, but still requires that the name chosen for the record must not change.  Second, there is the additional cost of a load balancer for every microservice.  Finally, there is additional latency introduced with adding a load balancer between each microservice invocation.

  • Shared load balancer: Create a load balancer that is shared by all microservices in an environment.  The load balancer must have rules for each microservice to route requests by URI patterns.

ms-architecture-3

The concern with this approach is that all traffic is now flowing through a single load balancer which can become a constraint in scaling the entire system.  Additionally, the load balancer becomes a shared resource amongst all the microservice teams, potentially impacting a team’s ability to operate independently of other teams.

  • Client load balancer: Load balancing from within the client is an approach in which the client has an awareness of all the containers in-service for a given microservice.  The client can then load balance between the containers when invoking the microservice.  This approach requires a system to provide service registration and service discovery.   

cloudcraft - mu-bananaservice-v3

The benefit with this approach is there are no longer load balancers between each microservice request so all the concerns with those prior approaches are addressed.  However, a new type microservice, an edge service, will need to be deployed to allow clients outside the microservice environment (that do not have access to service discovery) to invoke the service.

The preferred approach is the third approach which uses service discovery and client side load balancing within the microservice environment and edge routing with traditional load balancing for clients outside the microservice environment.  This approach provides the lowest latency and most loosely coupled solution for microservice invocation.

Let mu help!

The environment that mu creates for your microservice can manage the provisioning of Consul for service discovery and registration of your microservices.  Consul is a sort of phonebook for microservices.  It provides APIs for services to register their endpoints and for clients to lookup the endpoints.

Let’s demonstrate this by adding an additional milkshake service to the invoke the banana service from the first post.  Additionally, we will create a zuul router service to provide an edge service via Netflix’s Zuul.  Zuul is a proxy service that serves as the front door for all requests from outside the microservice environment.  Zuul will use Consul for service discovery to determine where best to route the incoming request.  Additionally, Zuul provides an excellent location to enforce policies such as authentication, authorization or logging on all incoming requests.

Enabling Consul and Edge Router

The first thing we will want to do is set up our edge router with Zuul.  This is just a matter of adding the @EnableZuulProxy and @EnableDiscoveryClient annotations to the Spring Boot application:

@SpringBootApplication
@EnableDiscoveryClient
@EnableZuulProxy
public class ZuulRouterApplication {

   public static void main(String[] args) {
     SpringApplication.run(ZuulRouterApplication.class, args);
   }
}

Zuul is configured via the application.yml file in src/main/resources.  For each service that we want exposed via the edge router, we add URI path patterns:

spring:
  application:
    name: zuul-router
zuul:
  routes:
    milkshake-service:
      path: /milkshakes/**
      stripPrefix: false
    banana-service:
      path: /bananas/**
      stripPrefix: false

In order to enable Consul in your environment, you need to update the environment definition in the mu.yml file.  Additionally, you need to configure Spring Cloud Consul to connect to the docker host ip address for service discovery.  We will also want to configure Spring Cloud to not register with Consul, since mu will already configure the Registrator agent on your ECS container instances:

 environments:
 - name: dev
   cluster:
     maxSize: 5
   discovery:
     provider: consul
 - name: production

service:
  name: zuul-router
  port: 8080
  pathPatterns:
  - /*
  environment:
    SPRING_CLOUD_CONSUL_HOST: 172.17.0.1
    SPRING_CLOUD_CONSUL_DISCOVERY_REGISTER: 'false'
  pipeline:
    source:
      provider: GitHub
      repo: cplee/zuul-router
    build:
      image: aws/codebuild/java:openjdk-8

Create Milkshake Service

Now we can create a new service to manage the creation of milkshakes.  The service looks very similar to the banana service, with the exception of declaring a Spring RestTemplate annotated with @LoadBalanced to enable client side loadbalancing via Ribbon.

 

@SpringBootApplication
@EnableDiscoveryClient
public class MilkshakeApplication {

  @LoadBalanced
  @Bean
  RestTemplate restTemplate(){
     return new RestTemplate();
  }
}

Now we can use the RestTemplate to make calls directly to the banana service.  Ribbon will do a lookup in Consul for a service named banana-service and replace it in the URL with one of the container’s IP and port:

@Component
public class BananaProvider implements FlavorProvider {

  @Autowired
  private RestTemplate restTemplate;

  private List<Map<String,Object>> getAll() {
    ParameterizedTypeReference<List<Map<String, Object>>> typeRef =
            new ParameterizedTypeReference<List<Map<String, Object>>>() {};

    ResponseEntity<List<Map<String, Object>>> exchange =
            this.restTemplate.exchange("http://banana-service/bananas",HttpMethod.GET,null, typeRef);

    return exchange.getBody();
  } 

Try it out!

After we have deployed all three services, we can use mu to confirm that all are running as expected.

~ ❯❯❯ mu env show dev                                                                                                                                                                                                       

Environment:    dev
Cluster Stack:  mu-cluster-dev (UPDATE_COMPLETE)
VPC Stack:      mu-vpc-dev (UPDATE_COMPLETE)
Bastion Host:   35.164.117.25
Base URL:       http://mu-cl-EcsEl-144KXQMIRY9WI-1411768500.us-west-2.elb.amazonaws.com

Container Instances:
+---------------------+----------+--------------+------------+-----------+--------+---------+-----------+-----------+
|    EC2 INSTANCE     |   TYPE   |     AMI      |     AZ     | CONNECTED | STATUS | # TASKS | CPU AVAIL | MEM AVAIL |
+---------------------+----------+--------------+------------+-----------+--------+---------+-----------+-----------+
| i-08e3edc8c644f0534 | t2.micro | ami-62d35c02 | us-west-2b | true      | ACTIVE |       3 |       604 |       139 |
| i-05bc14a67e53889e1 | t2.micro | ami-62d35c02 | us-west-2a | true      | ACTIVE |       3 |       604 |       139 |
| i-0b56a0d9572531e9e | t2.micro | ami-62d35c02 | us-west-2c | true      | ACTIVE |       3 |       604 |       139 |
| i-05b2188a5c575fbeb | t2.micro | ami-62d35c02 | us-west-2b | true      | ACTIVE |       1 |       624 |       739 |
+---------------------+----------+--------------+------------+-----------+--------+---------+-----------+-----------+

Services:
+-------------------+---------------------------+------------------+---------------------+
|      SERVICE      |         IMAGE             |      STATUS      |     LAST UPDATE     |
+-------------------+---------------------------+------------------+---------------------+
| milkshake-service | milkshake-service:9e4bcd9 | CREATE_COMPLETE  | 2017-05-12 11:33:05 |
| zuul-router       | zuul-router:3d4795c       | UPDATE_COMPLETE  | 2017-05-12 12:09:47 | 
| banana-service    | banana-service:3b62124    | UPDATE_COMPLETE  | 2017-05-12 11:32:55 |
+-------------------+---------------------------+------------------+---------------------+

We can then use curl to get a list of all the bananas available via the banana-service:

curl -s http://mu-cl-EcsEl-144KXQMIRY9WI-1411768500.us-west-2.elb.amazonaws.com/bananas | jq
[
  {
    "pickedAt": null,
    "peeled": null,
    "links": [
      {
        "rel": "self",
        "href": "http://mu-cl-ecsel-144kxqmiry9wi-1411768500.us-west-2.elb.amazonaws.com/bananas/9"
      }
    ]
  }
]

Next we try to create a milkshake using the milkshake-service:

~ ❯❯❯ curl -s -d "{}" -H "Content-Type: application/json" http://mu-cl-EcsEl-144KXQMIRY9WI-1411768500.us-west-2.elb.amazonaws.com/milkshakes\?flavor\=Banana | jq                                                                         
{
  "timestamp": "2017-05-15T19:12:56.640+0000",
  "status": 500,
  "error": "Internal Server Error",
  "exception": "org.springframework.web.client.HttpClientErrorException",
  "message": "429 Not enough bananas to make the shake.",
  "path": "/milkshakes"
}

Looks like there aren’t enough bananas to create a milkshake.  Let’s create another one:

~ ❯❯❯ curl -s -d "{}" -H "Content-Type: application/json" http://mu-cl-EcsEl-144KXQMIRY9WI-1411768500.us-west-2.elb.amazonaws.com/bananas

~ ❯❯❯ curl -s http://mu-cl-EcsEl-144KXQMIRY9WI-1411768500.us-west-2.elb.amazonaws.com/bananas | jq                                                                                                                         
[
  {
    "pickedAt": null,
    "peeled": null,
    "links": [
      {
        "rel": "self",
        "href": "http://mu-cl-ecsel-144kxqmiry9wi-1411768500.us-west-2.elb.amazonaws.com/bananas/9"
      }
    ]
  },
  {
    "pickedAt": null,
    "peeled": null,
    "links": [
      {
        "rel": "self",
        "href": "http://mu-cl-ecsel-144kxqmiry9wi-1411768500.us-west-2.elb.amazonaws.com/bananas/10"
      }
    ]
  }
]

Now let’s try again creating a milkshake:

~ ❯❯❯ curl -s -d "{}" -H "application/json" http://mu-cl-EcsEl-144KXQMIRY9WI-1411768500.us-west-2.elb.amazonaws.com/milkshakes\?flavor\=Banana | jq                                                                      
{
  "id": 3,
  "flavor": "Banana"
}

This time it worked, and if we query the list of bananas again, we see that 2 have been deleted for the milkshake:

~ ❯❯❯ curl -s http://mu-cl-EcsEl-144KXQMIRY9WI-1411768500.us-west-2.elb.amazonaws.com/bananas | jq                                                                                                                        
[]

Conclusion

Decomposing a monolithic application into microservices presents an interesting challenge in enabling services to invoke one another while still keeping the services loosely coupled.  Using a client side load balancer like Ribbon along with a service discovery tool like Consul provide an excellent solution to this challenge.  As demonstrated in this post, mu makes it simple to enable service discovery in your microservice environment to help achieve this solution.  Head over to stelligent/mu on GitHub and get started!

Additional Resources

Did you find this post interesting? Are you passionate about working with the latest AWS technologies? If so, Stelligent is hiring and we would love to hear from you!

Cloud Custodian Cleans Up Your Cloud Clutter

AWS allows you to build enormous and complex cloud infrastructures in a matter of hours. With the ability to create resources so easily, sometimes it can be hard to manage all those resources. If only there were a simple but powerful tool that could manage it all. Cloud Custodian (a.k.a C7N) is a Python CLI tool that gives you powerful account management capabilities with a simple config file. Cloud Custodian can help you manage your AWS account using a simple policy config file and time-based or event-based Lambdas. The config files (YAML formatted) allow you to define policies for everything from tag compliance to backups. Define policies for a wide variety of management activities, including garbage collection to encryption.

Cloud computing has made creating and managing web resources insanely easy, quite possibly too easy. You can now spin up quite a few computing, database, and storage resources with the click of a button or the stroke of a return key. However, if you use a company account, you likely spin up those resources often for demonstration and testing purposes, without considering the cost or clutter you might be creating along with it. This was “the problem” at Capital One when they created this very powerful tool for managing the cleanup of your superfluous cloud resources. Capital One started developing Cloud Custodian in July 2015 and open-sourced the tool in April 2016.

Cloud Custodian’s feature-set has grown exponentially with it’s popularity because they’re very good about responding to feature requests. It’s now grown to the point where there’s not much in the AWS world you can’t do with it. Here’s a short list of some things you might be surprised it can do:

  • Encryption
  • Backups
  • Garbage Collection
  • Unused Resources
  • Off-hours
  • Tag Compliance
  • SG Compliance

Odds are though, you’re considering Cloud Custodian for it’s namesake: cleaning up your AWS account; resource/cost management during off-hours; and overall garbage collection. True to it’s name, this is where Cloud Custodian excels. With a relatively simple configuration file you can tidy and trim your AWS account and keep it that way as you grow your business.

Here’s a very basic example custodian.yml file that stops all EC2 instances tagged with Custodian:

policies:
  - name: stop-instances
    resource: ec2
    filters:
      - "tag:Custodian": present
    actions:
      - stop

Cloud Custodian is great for mid to large sized companies that give a large number of their employees full access to a company AWS account. Naturally, their account quickly becomes cluttered with dozens of CloudFormation stacks, VPCs, old test instances, and Lambda functions. Here at Stelligent we have an AWS Labs account for exploring and testing in AWS. We use Cloud Custodian to clean up old testing resources based on age and resource tags. 

Your Cloud Custodian Strategy

singlenodedeploy.pngSource

As of today, there’s not much Custodian can’t do in your AWS account so it’s good to explore what Cloud Custodian can do for you before deciding on your overall strategy. Here are four common use-cases:

  • Automatic clean-up: Using the mode property, you can run actions in response to a variety of CloudWatch EventsRead more
  • Monitoring your environment: This is one of my favorite features. Custodian generates CloudWatch metrics by default so it’s easy to throw together dashboards that give you full visibility into what is being managed by Custodian and what isn’t. It’s hard to get good visibility in a vast system like AWS. Read more
  • Stopping Resources during Off-hours: Custodian makes it very easy to set up Off-hours for your resources based on tags. Below is an example. Read more

    policies:
      - name: offhours-stop
        resource: ec2
        filters:
          - type: offhour
            tag: downtime
            onhour: 8
            offhour: 20
        actions:
          - stop
  • Tag-compliance: One of the most common use cases for Custodian is tag-compliance. You can manage tag-compliance policies for your entire account in a single config file. You can even check it into version control. And, if you’re really ambitious, you can create a pipeline to watch your version control system that runs custodian for you so you don’t have to activate virtualenv on your personal machine every time you want to make a change. Read more

Prereqs and Pro Tips

Cloud Custodian is very well documented, so if you’re excited to start taking out the digital trash in your AWS account there’s no better place to start than their website and documentation. There are a few things to keep in mind before diving head-first into the cloud equivalent of the custodial arts:

Prereqs

At Stelligent we’re a tiny bit obsessed with one-command solutions. I have to admit, I cringed a little when I saw that Cloud Custodian took 3 or more commands to install and run. In the Getting Started section of the docs, Custodian requires you to have python, pip and virtualenv installed before you can even install Cloud Custodian. Then, once you activate virtualenv to install and run it the first, you’ll need to re-activate the virtualenv every time you want to run it in the future. That’s why I recommend using a Pipeline or CloudFormation template which brings us directly to our pro tips:

Pro Tip #1 – Minimalist Custodian

The easiest way to get started cleaning up your AWS account with Custodian is to go through your account and tag everything you want to keep with something like “NoCustodian”. Then, set

policies:
  - name: stop-instances
    resource: ec2
    filters:
      - "tag:Custodian": present
    actions:
      - stop

Click the button below to launch an example CloudFormation Stack that boots an EC2 instance and then uses Custodian to stop the instance.

Pro Tip #2 – Don’t piss off your co-workers

The first thing you’ll be tempted to do when implementing Cloud Custodian is terminate all the old and un-used resources in your account. Just be sure all the relevant parties in your company know what you’ll be terminating and when.

Pro Tip #3 – Use a Pipeline

Setup a CodePipeline that allows you to keep your custodian.yml file in source control and re-run it with CodeBuild every time you commit a change.

In Summary

If you need better visibility and automated management of your AWS account, Cloud Custodian has lots of helpful features that are easy to manage in a single config file. If you aren’t already a python developer, I recommend setting up a CloudFormation template or Automated Pipeline to manage changes to your. You can use the launch button in Pro Tip #1 to see an example of Custodian in a CloudFormation template. Keep a look out for a future blog post with a detailed example of a fully-automated Custodian Pipeline.

Links
Custodian Website
Custodian Docs
Custodian GitHub

Videos
AWS This Is My Architecture: Cloud Custodian
Cloud Custodian @ AWS re:Invent
Cloud Custodian @ Serverlessconf


Stelligent is hiring! Do you enjoy working on complex problems like figuring out ways to automate all the things as part of a deployment pipeline? Do you believe in the “one-button everything” mantra? If your skills and interests lie at the intersection of DevOps automation and the AWS cloud, check out the careers page on our website.

Microservice databases with mu

mu is a tool that makes it simple and cost-efficient for developers to use AWS as the platform for running their microservices.  In this third post of the blog series focused on the mu tool, we will use mu to manage microservice databases in the pipeline we built in the first post.  

Why should my microservice manage the database?

As discussed in prior posts, adopting a microservice architecture can increase a team’s ability to deliver software faster through decoupling and team autonomy.  By decomposing an application into microservices and then giving teams complete ownership of their microservices, the teams can then make decisions and implement changes independent of other teams and their microservices.

Unless the same approach is taken to decompose the databases that support the microservices, the benefits of microservices will be limited by the cross team dependencies on shared databases. When your microservices share a database then in effect you’ve used the database as an API between the services.  This type of architecture causes tight coupling between services and likely will require regression testing and even deployment of multiple services at the same time.

Martin Fowler, in his post titled Microservices, says “Microservices prefer letting each service manage its own database.”  By decomposing all the way down into the database you can realize the benefits of agility that microservices has to offer.

decentralised-data
Source: https://martinfowler.com/articles/microservices.html

Let mu help!

blog1The continuous delivery pipeline that mu creates for your microservice can manage the provisioning of a database.  Additionally, the details about the database can be injected into your service as environment variables.

Let’s demonstrate this by adding a database to the microservice pipeline we created in the first post for the banana service.

Define the database

Previously, the banana service was using an embedded H2 database.  This won’t work in a production environment so we need an RDS database instance that the microservice can use.  Adding a database for a service with mu is as simple as adding a couple lines to your mu.yml file:

service:
  name: banana-service
  port: 8080
  pathPatterns:
  - /bananas
  database:
    name: banana

By default, this will create an RDS database instance of size db.t2.small with the aurora engine.  Next we need to reference the database from our microservice.  We can pass the database URL and credentials via environment variables:

service:
  name: banana-service
  port: 8080
  pathPatterns:
  - /bananas
  database:
    name: banana

  environment:
    SPRING_DATASOURCE_USERNAME: ${DatabaseMasterUsername}
    SPRING_DATASOURCE_PASSWORD: ${DatabaseMasterPassword}
    SPRING_DATASOURCE_URL: jdbc:mysql://${DatabaseEndpointAddress}:${DatabaseEndpointPort}/${DatabaseName}

This approach does have the disadvantage of passing database credentials as environment variables.  This presents a security issue as any IAM user/role with access to ECS task API would be able to discover the credentials.

AWS has recently announced IAM database authentication that can be utilized to obtain temporary database credentials from the microservice via an AWS API call.  Although we will save the details for a future blog post, for now it’s worth mentioning that mu can configure the database for IAM database authentication to work around this issue of passing credentials as environment variables.  This would be accomplished with a mu.yml like this:

service:
  name: banana-service
  port: 8080
  pathPatterns:
  - /bananas
  database:
    name: banana
    instanceClass: db.t2.medium
    iamAuthentication: true

  environment:
       SPRING_DATASOURCE_URL: jdbc:mysql://${DatabaseEndpointAddress}:${DatabaseEndpointPort}/${DatabaseName}

The configuration of the tables and the data in the database is managed with Liquibase. When the service is started, Liquibase creates/updates the database tables and data. This is accomplished by creating the a file named db.changelog-master.yaml  in src/main/resources/db/changelog/

Now we can commit and push our changes to cause a new run of the pipeline to occur:

$ git add --all && git commit -m "add database" && git push

We see our pipeline is green, so we have confidence that the new database is working properly with the microservice.

Conclusion

Realizing the benefits of microservices requires decomposing not just the application, but also the databases that support it.  As demonstrated in this post, mu makes it simple to manage your database and wire them up to your microservices.  The goal is that mu empowers you to implement microservice best practices in your application.

In the upcoming posts in this blog series, we will look into:

  • Service Discovery – use mu to enable service discovery via `Consul` to allow for inter-service communication
  • Additional Use Cases – deploy applications other than microservices via mu, like a wordpress stack

Until then, head over to stelligent/mu on GitHub and get started!

Additional Resources

Did you find this post interesting? Are you passionate about working with the latest AWS technologies? If so, Stelligent is hiring and we would love to hear from you!