Ansible Quickstart

Included is a collection of Ansible Groups, put together to build the following:

  1. A load balancer
  2. 1-n web nodes (to sit behind the load balancer)
  3. A build server setup to automate deployments

Let's see how to use this.

Install Ansible

On Macintosh, we can install Ansible doing the following:

# Using Brew (recommended)
# This assumes you have Homebrew installed already
brew update
brew install ansible

# Using Python only
sudo easy_install pip
sudo pip install -U pip
sudo pip install ansible

On RedHat/CentOS, we can do the following after adding the EPEL repository:

# Install using package manager
sudo yum install epel-release
sudo yum install ansible

# Or install pip via package manager
sudo yum install epel-release
sudo yum install python-pip
sudo pip install -U pip
sudo pip install Ansible

# Or via Python only, if
# easy_install is on the server
sudo easy_install pip
sudo pip install -U pip
sudo pip install ansible

On Debian/Ubuntu you can install via pip:

# Via package manager
sudo apt-get install -y python-pip
sudo pip install -U pip
sudo pip install ansible

# Or via Python only
sudo easy_install pip
sudo pip install -U pip
sudo pip install ansible

On Ubuntu (not Debian), you can also install it via a PPA repository:

sudo apt-get install -y software-properties-common
sudo add-apt-repository -y ppa:ansible/ansible
sudo apt-get update
sudo apt-get install ansible

Setup Servers and Hosts

Assuming you have 3+ new servers, rename the hosts.dist file to hosts and fill out the servers. Depending on what platform you use and usernames you are given, adjust the SSH user to log in as.

Let's pretend we have 4 servers and know their IP addresses. We can fill out the hosts file like this:

[load_balancer]
142.54.87.123  ansible_ssh_user=ubuntu

[web]
142.54.87.124  ansible_ssh_user=ubuntu
142.54.87.125  ansible_ssh_user=ubuntu

[build_server]
142.54.87.126  ansible_ssh_user=ubuntu

On AWS, the provided Ubuntu image comes with a sudo user named "ubuntu". We'll use that user when provisioning the server with Ansible.

SSH Keys

We'll need to create some SSH keys to be used in various ways:

  1. An SSH key used to allow your work station to access all servers.
  2. An SSH key used on each Web Server to access GitHub.
  3. An SSH key used by the Build Server to access each Web Server.

We'll create these three SSH keys and then add them to our Ansible variables as needed.

Local Key

We'll make an SSH keypair to access each server. We could also use an already-existing key on your local computer, such as ~/.ssh/id_rsa.

This key will be used so allow you to login to each server, useful for maintenance or debugging.

I'm assuming you're using a Mac or *nix variety computer.

Make a new key:

cd ~/.ssh
ssh-keygen -t rsa -b 4096 -C you@your-domain.com -f id_myapp

You can create this with or without a password. Adding a password is always more secure. Note the filename created here via -f. We'll be adding the public key (e.g.id_myapp.pub) file content to the authorized_keys file for each user on each server so we can access the servers.

The private key we generate here will stay within the ~/.ssh directory of our local computer.

GitHub Key

We'll make another key locally, but this one will be used by our web servers instead of our local computer.

cd ~/.ssh
ssh-keygen -t rsa -b 4096 -C you@your-domain.com -f id_github

This SSH key must NOT be given a password. We won't have the opportunity to provide a password with automated deployments.

The private key generated (~/.ssh/id_github) will be added to each web server. The public key (~/.ssh/id_github.pub) will be added to GitHub for the repository we are deploying. This allows each web server to access GitHub, which will be required for this deployment scenario.

Build Server Key

We'll make our final key. This SSH key will be used by the Build Server to access the Web Servers and run tasks on them.

cd ~/.ssh
ssh-keygen -t rsa -b 4096 -C you@your-domain.com -f id_internal

The private key generated (~/.ssh/id_internal) will be added to the Build Server's user responsible for listening for webhooks and kicking off deploys.

The public key (~/.ssh/id_internal.pub) will be added to each Web Server authorized_keys file. This gives the Build Server the ability to run tasks on the Web Servers.

Only the user used to run and deploy the application will allow this access. For example, if user appuser runs the application, then /home/appuser/.ssh/authorized_keys is where the id_internal public key is added (it's the user the Build Server has access to run tasks as).

Ansible Variables

We're going to populate the group_vars needed to run the roles and get started. This will use some information we haven't yet retrieved along with the SSH keys created in the steps above.

Each group variable file directly corresponds to a group defined in the hosts file. The all file is a special one in that it provides variables for every host group.

Files to populate:

  1. group_vars/all
  2. group_vars/build_server
  3. group_vars/web, if using a PHP application with Composer

All

The variables in group_vars/all are available to all servers. Here is a rundown of the variables to add:

---
# Deployed Application Details

# Usernames to add to each server

# User that will run and
# deploy the application
appuser_username: appuser

# An admin user which will
# be allowed to run sudo commands
admin_username: admin

# The name of our application
# (machine-readable)
app_name: serialapp

# User passwords for our two users
# Password used here: serialapp
#   generated via `mkpasswd --method=SHA-512`
#   @link http://docs.ansible.com/ansible/faq.html#how-do-i-generate-crypted-passwords-for-the-user-module
admin_password:   '$6$gYhFEkelmCcCCf$gbpiYXbYxM8bgi9A.pCmkwNyG8YExw9xHb9zL8c6yrbVhyyTukbcBNuKvo6IPKZINdlJMZesN74f/QN5LngZD.'
appuser_password: '$6$gYhFEkelmCcCCf$gbpiYXbYxM8bgi9A.pCmkwNyG8YExw9xHb9zL8c6yrbVhyyTukbcBNuKvo6IPKZINdlJMZesN74f/QN5LngZD.'


# Your local public key, e.g. ~/.ssh/id_rsa.pub
# This will be added to the "authorized_keys" file on
# every server, so we can SSH into those servers from our
# local machines (e.g. your laptop)
local_authorized_key: '~/.ssh/id_myapp.pub'

# GitHub Webhook secret, must be known to GitHub and to our web listener
# which listens for webhooks sent from GitHub.
# >>> This MUST BE added to the WebHook configuration within your GitHub repository
webhook_secret: 'abcdefghijklmnopqrstuvwxyz1234567890'

# The GitHub private key that exists on each web server.
# The corresponding Public key should be added to Github for the
# deployed repository. This is added to the Web servers so they
# can access the github repository during deployment
# >>> This is the content of `id_github`
github_private_key: |
    -----BEGIN RSA PRIVATE KEY-----
    here we put in our long, multi-line ssh private key content
    -----END RSA PRIVATE KEY-----

# The public key that should be added to Github
# for the deployed repository.
# >>> This is the content of `id_github.pub`
github_public_key: ssh-rsa the-longish-ssh-public-key-generated

# The private key belonging to the Build Server. The corresponding
# public key is added to each Web server's "authorized_keys" file,
#  allowing the Build server the access needed to run tasks on them.
# >>> This is the content of `id_internal`
internal_private_key: |
    -----BEGIN RSA PRIVATE KEY-----
    here we put in our long, multi-line ssh private key content
    -----END RSA PRIVATE KEY-----

# The public key added to each Web server's "authoried_keys" file,
# allowing access to those servers from the Build server
# >>> This is the content of `id_internal.pub`
internal_public_key: ssh-rsa the-longish-ssh-public-key-generated

Build Server

The variables in group_vars/build_server are available to the Build Server only. Here is a rundown of the variables to add:

---
# AWS Access Keys. Make sure these are not available
# anywhere that can be searched or scraped by bots.
# Restrict access to these keys to the minimum possible access
aws_access_key_id: ABCDEFGHIJKLMNOPQRST
aws_secret_access_key: AbcDefGhIJKLmnopQRsTUvWxyZ12345678900987
# AWS defaults, used in each server's ~/.aws directory/config files
aws_region: us-east-1
aws_output: json
# The SQS url of the queue "pipe". Our build server will push to and pull from
# this queue in order to facilitate deployments.
sqs_url: 'https://sqs.us-east-1.amazonaws.com/123456789009/deploy-queue'

# URL for Webhook API given by Slack
slack_url: 'https://hooks.slack.com/services/F12345678/ABCDEFGHI/abcdefghIjKLmnopqRSTUvwX'

# The GitHub repository to be deployed. This can be public or private
# as we'll use SSH keys to allow access to the repository
git_repo: 'git@github.com:Some-User-Or-Organization/some-repository.git'

AWS Queue Permissions

You'll need to make a new SQS queue for this - the videos cover how exactly. The main point is making a new SQS queue and noting down it's URL and name.

Finally you'll need to make a new user and give them a set of credentials (the access key and secret key). The permissions to give that user for the SQS queue will be similar to the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1425446243123",
            "Effect": "Allow",
            "Action": [
                "sqs:ChangeMessageVisibility",
                "sqs:DeleteMessage",
                "sqs:GetQueueAttributes",
                "sqs:GetQueueUrl",
                "sqs:ReceiveMessage",
                "sqs:SendMessage"
            ],
            "Resource": [
                "arn:aws:sqs:us-east-1:012345678987:queue-name-here"
            ]
        }
    ]
}

The Resource you use will have a unique id number and name, and possibly a different region depending on what AWS region you use.

Web

This example has a github_token variable in the group_vars/web file. This is used by Composer to access GitHub and install library dependencies for PHP applications.

You can generate a token as per GitHub's docs.

---
# Generate a GitHub access token for Composer
# to use as per
# https://help.github.com/articles/creating-an-access-token-for-command-line-use/
github_token=1234567890987654321

Security

As the variables in the Group Vars are sensitive, you should consider not adding these to a repository on the internet. However, if you do, definitely make use of Ansible Vault to encrypt them.

You'll need a password to use to encrypt/decrypt variables within these files.

Here's how to do that:

# Encrypt "all" variable file, and create a password
ansible-vault encrypt group_vars/all
> CREATE PASSWORD AS PROMPTED

# Encrypt "build_server" variable file, and create a password
# Use the same password as used above
ansible-vault encrypt group_vars/build_server
> CREATE PASSWORD AS PROMPTED

# Encrypt any other file with sensitive variables
# Use the same password as used above
# In this case, we encrypt the "web" variable file
#   which contains a GitHub access token
ansible-vault encrypt group_vars/web
> CREATE PASSWORD AS PROMPTED

Once encrypted, you can edit these files using ansible-vault as well:

ansible-vault edit group_vars/all
> ENTER PASSWORD AS PROMPTED

This will open the files in the editor of your choice, as defined by the $EDITOR environment variable (usually nano or vim).

This file will be changed if you open the file but make no changes, as the file is re-encrypted every time it is opened (and then closed) using the ansible-vault tool. Directly editing the file will show an encrypted hash of the file.

Run Ansible

Once these are all set (or to test if we messed something up) we can finally run Ansible and get started!

cd ~/path/to/deploy-dist

# Check our Yaml Syntax
ANSIBLE_SSH_ARGS='-o IdentitiesOnly=yes -o ControlMaster=auto -o ControlPersist=60s' \
ansible-playbook --private-key=~/.ssh/fideloperllc.pem \
-i ./hosts \
--syntax-check \
--ask-vault-pass \
provision.yml

# Run Everything!
ANSIBLE_SSH_ARGS='-o IdentitiesOnly=yes -o ControlMaster=auto -o ControlPersist=60s' \
ansible-playbook --private-key=~/.ssh/fideloperllc.pem \
-i ./hosts \
--ask-vault-pass \
provision.yml

# Or, Limit runs by host
# In this example, we run only load_balancer tasks
ANSIBLE_SSH_ARGS='-o IdentitiesOnly=yes -o ControlMaster=auto -o ControlPersist=60s' \
ansible-playbook --private-key=~/.ssh/fideloperllc.pem \
-i ./hosts --limit=load_balancer \
--ask-vault-pass \
provision.yml

Finally we can run Ansible. The previous commands do the following:

First we set ANSIBLE_SSH_ARGS. The important argument here is IdentitiesOnly yes, which help to ensure we use only the SSH key we have available to access our servers. In my case of using AWS, this is the private key they have you download to access any and all servers you create. You might access this via user root and a password for Linode/Digital Ocean server that doesn't have a key added when you generate a server.

Then we run the Ansible command with the following flags:

  • --private-key=~/.ssh/fideloperllc.pem - Define the private key used to access these servers. If you don't have a key pre-loaded onto the servers via your provider, see the documentation directly following this
  • -i ./hosts - Use the hosts file we generated
  • --syntax-check - Optionally, don't run the commands but instead just check the Yaml syntax
  • --ask-vault-pass - Ask the Vault password used to encrypt the Group variables.

If you do not have an SSH key available on the server, you're likely given credentials to access the server via the root user, using a password. You can run Ansible that way by:

  1. Adjusting the hosts file to set the SSH user
  2. Telling ansible-playbook to ask for a user password

In the Hosts file, tell Ansible to log into each server using user root:

[load_balancer]
142.54.87.123  ansible_ssh_user=root

[web]
142.54.87.124  ansible_ssh_user=root
142.54.87.125  ansible_ssh_user=root

[build_server]
142.54.87.126  ansible_ssh_user=root

The following command tells Ansible to prompt you for the user's password. This will be the password for user root:

ansible-playbook \
    -i ./hosts \
    --ask-pass \
    --ask-vault-pass \
    provision.yml

GitHub Notes

It was mentioned in the above configuration, but to be clear - Setup within GitHub involves three things:

  1. Copying the id_github.pub file content into GitHub for access to our repository
  2. Configuring the Webhook (setting the IP address of your server at port 8080 and set GitHub to use the /webhook URI)
    • If the IP address of the server is 142.54.87.126, the build server URL from GitHub would be http://142.54.87.126:8080/webhook.
  3. Optionally creating a GitHub Token for Composer to use to grab PHP libraries without hitting GitHub API limits

Stand-alone Fabric

We use Python Fabric in this video series to create a deployment process. A stand-alone Fabfile (fabfile.py) is available with these files, which you can add to any project of your own.

By default this will assume you're deploying a PHP project (and even more specifically, a Laravel project), but it of course can be modified to suit your needs.

There are multiple variables to fill in, found within the brackets.

Setup

If you use the fabfile.py as a stand-alone file, rather than the automated server, some setup will be needed on the production server where the web application will live.

This will basically mirror the setup in the Ansible "app" role provided here. The following is a bash script of the basic steps. We'll pretend the user that runs the application will be "appuser" and the application named "serialapp".

# Before running this,create the application user (perhaps "appuser") and install server dependencies. In this script, we'll need PHP and Composer.


## Add the github SSH key and a configuration to automatically use it
# This assumes there's an id_github file created on the production
# server to use with GitHub.
# See above for notes on creating that.
USER_SSH_CONFIG="
Host github.com
    HostName github.com
    IdentityFile ~/.ssh/id_github
    User git
    IdentitiesOnly yes
"

echo $USER_SSH_CONFIG >> ~/.ssh/config


# Assumes composer is installed
# on the production server
# And you've generated a GitHub token
# This assumes composer is installed
mkdir -p ~/.composer
composer --global config github-oauth.github.com 1234567890

# Generate "Persistent" directories
# for files that persist across deployments
mkdir -p ~/serialapp/persist
mkdir -p ~/serialapp/persist/storage/app
mkdir -p ~/serialapp/persist/storage/framework/cache
mkdir -p ~/serialapp/persist/storage/framework/sessions
mkdir -p ~/serialapp/persist/storage/framework/views
mkdir -p ~/serialapp/persist/storage/logs

# Create a .env file to be used with the laravel applicatin
# if you're following along this example
APP_ENV_FILE="
APP_ENV=production
APP_DEBUG=false
APP_KEY=1234567890098765432112345678900987654321

DB_HOST=localhost
DB_DATABASE=homestead
DB_USERNAME=homestead
DB_PASSWORD=secret

CACHE_DRIVER=file
SESSION_DRIVER=file
QUEUE_DRIVER=sync
"

echo $APP_ENV_FILE > ~/serialapp/persist/.env

These should be run on the production server. Then we can run Fabric locally to connect and run tasks on it.