Puppet configuration: The ultimate AWS automation

Automation projects can slip down the priority list. If your organization is ready to tackle AWS automation, this guide will help ease the process.


Amazon Web Services (AWS) customers make a five-step journey to cloud automation mastery. They start with the pointy-clicky AWS web console, and finish by coding declarative models of their cloud computing systems.

1: Ease in with the AWS web console

The easiest way to get familiar with the many AWS services is to repeatedly poke things in the AWS web console to see what happens. The AWS console is stuffed full of features, and more appear every year. An AWS customer can look around the many dashboards, and figure out which parts are useful for their project. The AWS console is not just for beginners — even seasoned veterans continue to use the AWS control panel for manual work.

AWS security groups restrict access to services. The EC2 dashboard in the AWS console has a Security Groups page. Figure A shows the creation of a security group called sgMail, allowing mail requests from the internet. Come on in, spammers!

Figure A

Image: Screenshot by Nick Hardiman

2: Get in touch with the AWS CLI

The next stage is to use the AWS command line tools. Moving from the web to the command line can feel like walking around with your eyes closed. Once you get the hang of them, the CLI tools are quicker and more flexible to use than the web console.

Viewing that sgMail security group using the command line tools looks like this.

nick:~ $ aws ec2 describe-security-groups --group-names sgMail
SECURITYGROUPS	Mail access from Internet	sg-40199d36	sgMail	243894605340
nick:~ $ 

Brief and to the point. I'm only interested in the facts. Only the facts, please.

3: Get serious with the AWS API

After that, use the AWS API. This is where the learning curve is steepest. An AWS customer needs to have a few things in place before they can start writing code and using the API. These things include:

  • basic knowledge of a programming language such as Java, Node.js, or Ruby;
  • an add-on library for that language to interface with the AWS API — AWS is so popular there are AWS SDKs for Python (called Boto), PHP, Ruby, and many other languages; and
  • a development environment for writing code, which could be as simple as a text editor or as complex as Microsoft's Visual Studio.

4: Automation frenzy!

Once an AWS customer is over the hump of getting to grips with the AWS API, it's playtime. Along comes the irresistible urge to automate everything! Creation of a high-availability cluster, package deployment, security updates — all must be automated using the power of code.

In fact, programmers have been automating AWS for as long as there has been AWS. Many configuration management tools help automate all the layers of the technology stack.

In fact, there are so many automation tools that integrating them into one working production line is a complicated job.

Automation is usually done using imperative languages.

After going through some installation and configuration pain, a Python developer can use boto to work with AWS security groups. Commands to view that sgMail group look like this.

nick:~ $ python
Python 2.7.6 (default, Sep  9 2014, 15:04:36) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto.ec2
>>> conn = boto.ec2.connect_to_region('eu-west-1')
>>> rs = conn.get_all_security_groups()
>>> print rs
[SecurityGroup:sgMail, SecurityGroup:default]
>>> sg = rs[0]
>>> sg.rules
[IPPermissions:tcp(25-25), IPPermissions:tcp(110-110), IPPermissions:tcp(143-143), IPPermissions:tcp(465-465), IPPermissions:tcp(993-993), IPPermissions:tcp(995-995)]
>>> quit()
nick:~ $

A reference, an object, a method, a list operator, and that crazy interactive Python ">>>" prompt — it's getting tricky now. And, yes, pretty ugly too — those pretty pastel colors of the console are now a distant memory.

5: Level up with declarative code

The Puppet, Chef, and Ansible automation applications have their own configuration languages that help their users describe models of how things are required to be. They also have clever engines that read those models and change computing environments to look them.

These configuration languages are declarative. Computer languages like the Python example above are imperative — they are useful for writing step-by-step procedures. Declarative languages describe what is required, but not the steps to get there. The SQL "select" statement is declarative — it may order a server to "SELECT records FROM table," but the steps to find the table, use an index, display results, and so on are hidden.

Puppet Labs brought out a new module to provision AWS infrastructure. Using one of their modules means getting up to speed with Puppet. Puppet automation has a learning curve just like writing code does - a customer must get to grips with these tasks to use Puppet:

  • installing a Puppet application and a Ruby interpreter;
  • using the RubyGems package manager; and
  • writing a manifest.

After setting up the new Puppet AWS module, an AWS customer can describe that sgMail group using a manifest like this.

nick:~ $ cat /etc/puppet/manifests/aws-manifest.pp
ec2_securitygroup { 'sgMail':
  region      => 'eu-west-1',
  ensure      => present,
  description => 'Mail access from Internet',
  ingress     => [{
    protocol   => 'tcp',
    port       => 25,
    cidr       => '',
    protocol   => 'tcp',
    port       => 110,
    cidr       => '',
    protocol   => 'tcp',
    port       => 143,
    cidr       => '',
    protocol   => 'tcp',
    port       => 465,
    cidr       => '',
    protocol   => 'tcp',
    port       => 993,
    cidr       => '',
    protocol   => 'tcp',
    port       => 995,
    cidr       => '',
nick:~ $

A customer can then make Puppet talk to AWS and make sure this security group exists — either manually with the command puppet apply aws-manifest.pp, or automatically using a Puppet master.

How far has your organization travelled?

Not all organizations complete the journey. Some organizations won't go beyond step four.

Enterprise customers already have a working production line by this point, so the automation projects can slip down the priority list. An organization may only decide to put in the extra time and expense to get from "it works" to "it works well" if there is a business driver caused by an audit failure, operational cost, shrinking market share, or other pain.

About Nick Hardiman

Nick Hardiman builds and maintains the infrastructure required to run Internet services. Nick deals with the lower layers of the Internet - the machines, networks, operating systems, and applications. Nick's job stops there, and he hands over to the ...

Editor's Picks

Free Newsletters, In your Inbox