Cloud

AWS EC2 cloud automation tricks that will amaze you

Nick Hardiman uses cloud-init to do clever cloud instance initialization things when firing up a new Amazon Web Services EC2 machine.

cloud_021414.jpg

I'm automating a simple web service. Using the new Amazon Web Services (AWS) Command Line Interface (CLI) tools, I will build two Amazon EC2 machines before your eyes! One machine hosts a Puppet master and the other hosts a puppet agent and Apache.

I'm using some cloud automation tricks. I use cloud-init to create a Puppet master service, and then the Puppet master will create the Apache service.

I put the instructions (the cloud-config data) in these two files.

  • p-master-user-data.yml - cloud-init information for my Puppet master.
  • p-agent-user-data.yml - cloud-init information for my Puppet agent.

Ubuntu uses an application called cloud-init to prepare a new cloud machine for use. Cloud-init can do all sorts of clever cloud instance initialization things when you fire up a new EC2 machine. Cloud-init instructions can be written in many formats such as cloud-config data, a shell script, or even python code.

The ephemeral drive /dev/xdb2

If you have started an Ubuntu-based cloud machine, you may have already noticed one of the things that cloud-init does. On the Trusty OS, cloud-init creates a second drive - an ephemeral one - called /dev/xdb2.

Ephemeral means short-lived; in the cloud world, ephemeral means "do not rely on this sticking around." This ephemeral drive and its contents disappear if the machine is stopped. And you ain't getting your stuff back.

Feeding cloud-init instructions to AWS EC2

The aws ec2 run-instances command -- the one that starts a new machine -- has an option called -user-data. If you give this option a bunch of cloud-init instructions, they are copied over to the new machine and carried out by cloud-init.

The cloud-init instructions that set up the Puppet master are whacky-looking YAML code. Every sysadmin since the UNIX epoch understands shell scripts, but few can look at cloud-config data without having palpitations. Here are both versions for installing a web server.

The cloud-config data version

#cloud-config
# Install the web server
packages:
 - apache2

The shell script version

#!/bin/bash
# Install the web server
apt-get –y install apache2

p-master-user-data.yml - cloud-init information for my Puppet master

These abbreviated instructions trigger activity. The instructions:

  • add security patches and bug fixes,
  • install and configure Puppet, and then
  • reboot the machine.

The reboot is a good idea because there's a good chance the upgrades included a new kernel.

#cloud-config
# 
# add security patches and bug fixes
package_update: true
package_upgrade: true
# 
# install puppet
packages:
 - puppetmaster
# 
# configure puppet
write_files:
 - content: |
        *
   path: /etc/puppet/autosign.conf
   owner: root:root
   permissions: '0644'
# 
 - content: |
        package { 'apache2':
          ensure => present,
        }
   path: /etc/puppet/manifests/site.pp
   owner: root:root
   permissions: '0644'
# 
# clean up
final_message: 'The Puppet master is ready.'
power_state:
 mode: reboot
 message: Rebooting
 timeout: 30

These instructions are going to be handed over to the aws ec2 run-instances command. Stick these instructions in a text file so they are ready to go.

1. Edit a new text file called p-master-user-data.yml.

2. Copy these cloud-config instructions into the file.

3. Save your work.

Cloud-init will follow these instructions. When it deals with that write_files section, cloud-init creates two files on the Puppet master.

  • /etc/puppet/autosign.conf - Puppet security configuration
  • /etc/puppet/manifests/site.pp - a Puppet manifest

The file /etc/puppet/autosign.conf

This file tells the Puppet master to carry out basic puppet autosigning. The Puppet master allows requests from any Puppet agent. The system is only protected by the p-master-group security group (see my previous post).

The autosign file contains a whitelist that matches anything. The only content is a wildcard (*).

The file /etc/puppet/manifests/site.pp

This file contains Puppet instructions (a manifest in Puppet-speak). It is downloaded by the Puppet agent.

The contents look like this.

package { 'apache2':
  ensure => present,
}

p-agent-user-data.yml - cloud-init information for my Puppet agent

The application on each Puppet that sends requests to the Puppet master is called the Puppet agent.

The Puppet agent looks for a Puppet master named puppet (no, not puppetmaster -- that would be too obvious). These cloud-conf instructions associate an IPv4 address with that puppet name and the Puppet master's address.

#cloud-config
# 
# add security patches and bug fixes
package_update: true
package_upgrade: true
# 
# install puppet
packages:
 - puppet
# 
# associate an IPv4 address with that puppet name 
# be careful not to add another line after every reboot
bootcmd:
 - grep puppet /etc/hosts || echo 10.104.240.212 puppet >> /etc/hosts
 - puppet agent --enable
#
# clean up
final_message: 'The Puppet agent is ready.'
power_state:
 mode: reboot
 message: Rebooting
 timeout: 30

Store these instructions in another text file.

1. Edit a new text file called p-agent-user-data.yml.

2. Copy these cloud-config instructions into the file.

3. Save your work.

How it will all come together

This is what I'll be doing later when I fire up the instances.

The new AWS CLI command doesn't accept a user-data file. When you launch a new EC2 machine, you have to stick all those lines from a .yml file on the end of the run-instances command.

aws ec2 run-instances --image-id ami-50b64527 --count           1 --instance-type t1.micro --key-name p-keypair --security-groups p-master-group --user-data “#cloud-config
# 
# add security patches and bug fixes
…
 mode: reboot
 message: Rebooting
 timeout: 30”

It's very long and very confusing. It's neater to stick all those cloud-config commands into a variable, like this.

4. Stick the file contents into a variable.

5. Launch the Puppet master.

aws ec2 run-instances \
  --image-id        ami-50b64527 \
  --count           1 \
  --instance-type   t1.micro \
  --key-name        p-keypair \
  --security-groups p-master-group \
  --user-data       "$my_user_data"

That very long aws ec2 run-instances line a couple of paragraphs up is broken up into several lines here, and spaced out to make it look like a table. The sysadmins will recognize the backslash \ on the end of each line is tech shorthand for "ignore this line break" -- or, if you are really pedantic (and what good sysadmin isn't?), "escape the meaning of the next character." These many lines are treated as if they are one line.

We've talked the talk. Next we walk the walk.

After all the preparation with installing the tool, picking an AWS region, and getting the configuration together, it's finally time to make something happen. Next time we run that run-instances command to create the Puppet master, and check our work.

Catch up on previous installments in this series

About

Nick Hardiman builds and maintains the infrastructure required to run Internet services. Nick deals with the lower layers of the Internet - the machines, networks, operating systems, and applications. Nick's job stops there, and he hands over to the ...

0 comments

Editor's Picks