If you have not had the chance to work with Salt yet, it is a very exciting new configuration management system that is easy to get up and running, powerful enough to support distributed command execution and complex configuration management, and scalable enough to support thousands of servers simultaneously.

Recently, I wrote about how a de facto containerization standard will enable a whole new generation of management tools.  Back in January, SaltStack announced several awesome new features in the Salt 2014.1.0 (Hydrogen) release, including support for the life-cycle management of Docker containers.  SaltStack is very early in getting Docker support, and Docker itself still does not consider itself production-ready (tell that to Yelp and Spotify), but together these tools offer an out-of-the box solution for getting started with immutable infrastructure.

The deployment and management of an application across multiple virtual machines, using multiple public clouds, is a use case that would have been considered categorically “hard” just a year ago.  Companies like Google, Facebook, and Ning spent many years developing this kind of orchestration technology internally in order to deal with their scale challenges. Today, using Docker containers, together with Salt-Cloud, and a few Salt States, we can do this from scratch in a few tens-of-minutes of effort.  And, because we are using Salt’s declarative configuration management, we can scale this pattern to actually operate our production environment.

Use Case

The core use case is one or more application containers which we want to deploy on one or more virtual machines, using one or more public cloud providers.

For the sake of simplicity, we will restrict this use case to:

• Assume some familiarity with Salt
• Assume some familiarity with Docker
• Assume that you have a Salt master already installed
• Assume that you want to do this on a single public cloud, using Digital Ocean (since adding new clouds to Salt-Cloud is dead simple)
• Simulate a real-world application with a dummy apache service

Demo Use Case

In order to simulate a real-world application, will create a Docker container with the Apache web server.  Conceptually, this container could be a front-end proxy, a middle tier web service, a database, or virtually any other type of service we might need to deploy in our production application. To do that, we simply create a Dockerfile in a directory in the normal way, build the container, and push the container to the Docker repository.

Step 1: Create a Dockerfile

[email protected]:/some/dir/apache# cat Dockerfile
# A basic apache server. To use either add or bind mount content under /var/www
FROM ubuntu:12.04

MAINTAINER Kimbro Staken version: 0.1

RUN apt-get update && apt-get install -y apache2 && apt-get clean && rm -rf /var/lib/apt/lists/*

ENV APACHE_RUN_USER www-data
ENV APACHE_RUN_GROUP www-data
ENV APACHE_LOG_DIR /var/log/apache2

EXPOSE 80

CMD ["/usr/sbin/apache2", "-D", "FOREGROUND"]

Step 2: Build the container

[email protected]:/some/dir/apache# docker build -t jthomason/apache .
Uploading context  2.56 kB
Uploading context
Step 0 : FROM ubuntu:12.04
 ---> 1edb91fcb5b5
Step 1 : MAINTAINER Kimbro Staken version: 0.1
 ---> Using cache
 ---> 534b8974c22c
Step 2 : RUN apt-get update && apt-get install -y apache2 && apt-get clean && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 7d24f67a5573
...
<A WHOLE LOT OF STUFF HAPPENS>
...
Successfully built 527ad6962e09

Step 3: Push the container

[email protected]:/tmp/apache# docker push jthomason/apache
The push refers to a repository [jthomason/apache] (len: 1)
...
<A WHOLE LOT OF STUFF HAPPENS>
...
Pushing tag for rev [527ad6962e09] on {https://registry-1.docker.io/v1/repositories/jthomason/apache/tags/latest}

With our demo application successfully pushed to the Docker registry, we are now ready to proceed with orchestrating its deployment.  As mentioned previously, we assume you have a Salt master installed somewhere. If not, you’ll need to follow the documentation to get a Salt master installed.  The next step then is to configure Salt-Cloud for your choice of public cloud provider.  Configuring Salt-Cloud is simple.  We need to create an SSH key pair that Salt-Cloud will use to install the Salt Minion on newly created VMs, add that keypair to our choice of public cloud provider, and create a Salt-Cloud configuration file with the API credentials for our public cloud.

Step 4: Create an SSH Key Pair

[email protected]:/etc/salt/keys# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa): digital-ocean-salt-cloud
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in id_dsa.
Your public key has been saved in digital-ocean-salt-cloud.pub.
The key fingerprint is:
06:8f:6f:e1:97:5a:5a:48:ce:09:f3:b6:33:42:48:9a [email protected]
The key's randomart image is:
+--[ DSA 1024]----+
|                 |
|                 |
|      .          |
|    .  +         |
|   + .+ S        |
|  E . [email protected] + .     |
|     .  @ =      |
|      .ooB       |
|       .+o       |
+-----------------+
[email protected]:/etc/salt/keys#

Step 5: Upload SSH Key Pair

With Digital Ocean now enabled with our SSH keys, next steps before provisioning time are to configure salt-cloud with Digital Ocean’s API key for our account, and define the profiles for virtual machine sizes, geographic locations, and images for Digital Ocean.  The salt-cloud configuration for credentials are kept on the salt-master in /etc/salt/cloud.providers.d/, while the profiles for each public cloud are kept in /etc/salt/cloud.profiles.d/.  See the salt-cloud documentation for more details on configuration options.

Step 6: Configure Salt-Cloud

do:
  provider: digital_ocean
  # Digital Ocean account keys
  client_key: <YOUR KEY HERE>
  api_key: <YOUR API KEY HERE>
  ssh_key_name: digital-ocean-salt-cloud.pub
  # Directory & file name on your Salt master
  ssh_key_file: /etc/salt/keys/digital-ocean-salt-cloud

# Official distro images available for Arch, CentOS, Debian, Fedora, Ubuntu

ubuntu_512MB_sf1:
  provider: do
  image: Ubuntu 12.04.4 x64
  size: 512MB
#  script: Optional Deploy Script Argument
  location: San Francisco 1
  script: curl-bootstrap-git
  private_networking: True

ubuntu_1GB_ny2:
  provider: do
  image: Ubuntu 12.04.4 x64
  size: 1GB
#  script: Optional Deploy Script Argument
  location: New York 2
  script: curl-bootstrap-git
  private_networking: True

ubuntu_2GB_ny2:
  provider: do
  image: Ubuntu 12.04.4 x64
  size: 2GB
#  script: Optional Deploy Script Argument
  location: New York 2
  script: curl-bootstrap-git
  private_networking: True

It is now time to configure Salt to provision our application container.  To do that, we need to create two Salt states, one to provision Docker on a newly created VM, and another to provision the application container.  Salt states are Salt’s declarative configuration states, which are executed on target hosts by the salt-minion.  States are an incredibly rich feature of Salt, one that could hardly be covered with any sufficient level of detail in a tutorial like this.  This example is not a particularly smart or optimal use of states, but it is simple. You’ll want to read up on Salt states to develop the best practices for your environment.

Step 7: Create a Salt state for Docker

docker-python-apt:
  pkg.installed:
    - name: python-apt

docker-python-pip:
  pkg.installed:
    - name: python-pip

docker-python-dockerpy:
  pip.installed:
    - name: docker-py
    - repo: git+https://github.com/dotcloud/docker-py.git
    - require:
      - pkg: docker-python-pip

docker-dependencies:
   pkg.installed:
    - pkgs:
      - iptables
      - ca-certificates
      - lxc

docker_repo:
    pkgrepo.managed:
      - repo: 'deb http://get.docker.io/ubuntu docker main'
      - file: '/etc/apt/sources.list.d/docker.list'
      - key_url: salt://docker/docker.pgp
      - require_in:
          - pkg: lxc-docker
      - require:
        - pkg: docker-python-apt
      - require:
        - pkg: docker-python-pip

lxc-docker:
  pkg.latest:
    - require:
      - pkg: docker-dependencies

docker:
  service.running

This first salt state defines the dependencies and configuration for installing Docker on

Step 8: Create Salt state for the application container

apache-image:
   docker.pulled:
     - name: jthomason/apache
     - require_in: apache-container

apache-container:
   docker.installed:
     - name: apache
     - hostname: apache
     - image: jthomason/apache
     - require_in: apache

apache:
   docker.running:
     - container: apache
     - port_bindings:
            "80/tcp":
                HostIp: ""
                HostPort: "80"

Now that configuration is complete, we are now ready to provision 1..n virtual machines, each with a running instance of our application container.  Before we do that, let us first verify that the Salt master is actually working. We know there is at least one agent that should be talking to this salt-master at this point, which is the agent running on the salt-master itself.

Step 9: Verify that Salt is working

[email protected]:~# salt '*' test.ping
salt.garply.org:
True
[email protected]:~#

Satisfied that everything is in working order with the salt installation, we can now provision our first virtual machine with an instance of our container using salt-cloud.

Step 10: Provision a VM with an instance of the container

[email protected]:# salt-cloud --profile ubuntu_512MB_sf1 one.garply.org
[INFO    ] salt-cloud starting
[INFO    ] Creating Cloud VM one.garply.org
[INFO    ] Rendering deploy script: /usr/lib/python2.7/dist-packages/salt/cloud/deploy/curl-bootstrap-git.sh
...
<A WHOLE LOT OF STUFF HAPPENS>
...
one.garply.org:
    ----------
    backups_active:
        False
    created_at:
        2014-04-23T19:23:12Z
    droplet:
        ----------
        event_id:
            22373933
        id:
            1521385
        image_id:
            3101045
        name:
            one.garply.org
        size_id:
            66
    id:
        1521385
    image_id:
        3101045
    ip_address:
        107.170.230.112
    locked:
        True
    name:
        one.garply.org
    private_ip_address:
        None
    region_id:
        3
    size_id:
        66
    status:
        new

After the salt-cloud run completes, it emits a YAML blob containing information about the newly created VM instance.  Let’s use the IP address of the instance to see if our application is running

Step 11: Verify application is running 

Great success!

We have established the basic setup and management pattern for our infrastructure.  Adding additional public clouds is easy, thanks to salt-cloud, providing a single control interface for our entire application infrastructure.   But where to go from here?    A starting point is to consider how salt states can be used to manage VM and container life-cycles, in the context of the overall continuous integration and deployment process.   I plan to share some of my thoughts on that specifically in a future post.  Obviously, there is a lot of thought to be given to your specific objectives, since that will ultimately determine the deployment and operations architecture for your application.  However, Salt is an incredibly powerful tool that, when combined with Docker, provides a declarative framework for managing the application life-cycle in the immutable infrastructure paradigm right out of the box.  That versatility puts a whole lot of miles behind you, allowing you to focus on other core challenges with application deployment and operations.