Cloud automation: Everything you need to know to build a web service

Nick Hardiman's cloud automation series guides you through how to build a simple web service using Ubuntu, AWS, cloud-init, and Puppet.


Image: iStock/Nastco

This article is the first in a seven-part cloud automation series that will guide you through the build process for a simple web service. This big how-to uses Ubuntu 14.04 LTS, Amazon Web Services (AWS) Command Line Interface (CLI), cloud-init, and Puppet to deliver an Apache server.

If you're into DevOps, this tutorial is about the "Ops" bit; building this system is all about infrastructure, OS, and application configuration. It's not the "Dev" bit of object-oriented coding, unit tests, and continuous integration.

The architecture

The architecture is straightforward:

  • one Apache server on one machine, and
  • one Puppet server on another machine.

I didn't include any High Availability (HA) infrastructure, auto-scaling, or developer tools--I'm not even adding other LAMP components.

The technical stack is simple:

  • AWS provided the foundation I build on. I let AWS take care of the hardware, networking, and virtualization layers.
  • The next layer is the operating system. I use a free OS from Ubuntu.
  • Applications go on top. Apache and its supporting packages (binaries, libraries, and SSL certificate) come from the Ubuntu repositories.

The build process

The process is broken down into these steps.

  1. Design the cloud technology stack
  2. Install the new AWS CLI toolkit 
  3. Choose an AWS region 
  4. Add AWS security groups
  5. Work with cloud-init
  6. Create the Puppet master
  7. Create the Puppet agent

One Amazon EC2 machine hosts the Puppet master, and a second machine hosts Apache. I'm going to use cloud-init to create a Puppet master service. The Apache build happens automatically--the Puppet master installs it. The Apache machine is a puppet controlled by the puppet master (how creepy does that sound?).

Why complicate things with cloud-init and Puppet?

Why build two machines when one will do? And why add in configuration management technologies you can do without?

Manually building one machine is easier than getting your head around cloud-init and a Puppet server. If you have built a few LAMP servers in your time, you know how streamlined this process has become over the years. Run through these steps, and you're done.

  1. Pick the right AMI.
  2. Rent an EC2 machine.
  3. Log in.
  4. Run one command - apt-get install apache.

It's easy work, but it's manual work--it's not making the most of the cloud. The cloud opens up new possibilities that we did not have on-premise: the power to automatically build customer services, automatically configure them, and scale them to fit demand. These configuration management tools are part of making that happen.

The benefit of doing the complicated work now is that it reduces how much pain you might feel in the future. All customer services change over time. The more automation you can add, the more time you can spend sweating the big stuff and the quicker you can adapt to change.

Also, many enterprises want to move test, dev, and batch services to AWS. It doesn't hurt to be the guy with the skills to make that happen.

It might sound simple, but it requires a lot of work

This way of working isn't a time saver. I still have to manually build the Puppet master using esoteric cloud technology so, in fact, it's harder work. The advantage comes later, when my Puppet master controls many puppets, providing many customer services.

I make this system using AWS tools and a bunch of building blocks: EC2 machines, security groups, a keypair, cloud-config, and a Puppet manifest. You're going to see a fair number of labels with the prefix "p-" (an abbreviation of "Puppet-related kinda thing") on all these new things.

More AWS how-tos

AWS auto-scaling

  1. Getting to know Amazon's auto-scaling command line tools
  2. Auto-scaling an EC2 service: Creating a new AMI
  3. Installing and checking Amazon Auto Scaling and CloudWatch tools
  4. Add an auto-scaling group and policy for Amazon EC2 machines
  5. How to add Cloudwatch monitors to auto-scale your Amazon Web Service
  6. AWS auto-scaling: Add notification and test to see what happens

AWS CloudFormation

  1. How to be a SaaS vendor on a shoestring with Amazon Web Services
  2. First steps to SaaS with AWS CloudFormation templates
  3. Use AWS CloudFormation to create a highly available cluster
  4. Anatomy of an AWS CloudFormation template
  5. Modifying AWS CloudFormation templates: Polish your code


By Nick Hardiman

Nick Hardiman builds and maintains the infrastructure required to run Internet services. Nick deals with the lower layers of the Internet - the machines, networks, operating systems, and applications. Nick's job stops there, and he hands over to the ...