on Jason Wilder's Blog

A Simple Way To Dockerize Applications

Mon, 13 Oct 2014 00:00:00 UTC

Dockerizing an application is the process of converting an application to run within a Docker container. While dockerizing most applications is straight-forward, there are a few problems that need to be worked around each time.

Two common problems that occur during dockerization are:

Making an application use environment variables when it relies on configuration files
Sending application logs to STDOUT/STDERR when it defaults to files in the container’s file system

This post introduces a new tool: dockerize that simplifies these two common dockerization issues.

The Problems

Configuration

Many applications use configuration files to control how they work. Different runtime environments have different values for various sections of a file. For example, database connection details for a development environment would be different than a production environment. Similarly, API keys and other sensitive details would be different across environments.

There are a few ways to handle these environmental differences with docker containers:

Embed all environment details in the image and use a control environment variable to indicate which file to use at run time. (e.g. APP_CONFIG=/etc/dev.config)
Use volumes to bind mount the configuration data at run time
Use wrapper scripts that modify configuration data with tools like sed that environment variable

Embedding all environment details is not ideal because environmental changes should not require a rebuild of an image. It’s also less secure since sensitive data such as API keys, login credentials, etc.. for all environments are stored in the image. Comprimising a development environment could leak production details. Having these kinds of details in any image should really be avoided.

Using volumes keep these details out of the image but makes deployment more complicated since you can no longer just deploy the image. You must also coordinate configuration file changes along with the image.

Injecting environment variables into custom files is not always trivial as well. You can sometimes craft a sed command or write some custom scripts to it but it’s repetitive work. This does produce an image that works well in a docker ecosystem though.

Logging

Docker containers that log to STDOUT and STDERR are easier to troubleshoot, monitor and integrate into a centralized logging system. Logs can be acessed directly with the docker logs command and through the docker logs API calls. There are also many tools that can automatically pull docker logs and ship them off if they log to STDOUT and STDERR.

Unfortunately, many applications log to one or more files on the file system by default. While this can usually be worked around, it’s tedious to figure out the nuances of each applications logging configuration.

Using Dockerize

dockerize is a small Golang application that simplifies the dockerization process by:

Generating configuration files using templates and the containers environment variables at startup
Tailing arbitrary log files to STDOUT and STDERR
Starting a process to run within the container

An Example

To demonstrate how it works, we’ll walk through dockerizing a generic nginx container with dockerize. We start with:

FROM ubuntu:14.04

# Install Nginx.
RUN echo "deb http://ppa.launchpad.net/nginx/stable/ubuntu trusty main" > /etc/apt/sources.list.d/nginx-stable-trusty.list
RUN echo "deb-src http://ppa.launchpad.net/nginx/stable/ubuntu trusty main" >> /etc/apt/sources.list.d/nginx-stable-trusty.list
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C300EE8C
RUN apt-get update
RUN apt-get install -y nginx

RUN echo "daemon off;" >> /etc/nginx/nginx.conf

EXPOSE 80

CMD nginx

Next we’ll install dockerize and run nginx through it:

FROM ubuntu:14.04

# Install Nginx.
RUN echo "deb http://ppa.launchpad.net/nginx/stable/ubuntu trusty main" > /etc/apt/sources.list.d/nginx-stable-trusty.list
RUN echo "deb-src http://ppa.launchpad.net/nginx/stable/ubuntu trusty main" >> /etc/apt/sources.list.d/nginx-stable-trusty.list
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C300EE8C
RUN apt-get update
RUN apt-get install -y wget nginx

RUN echo "daemon off;" >> /etc/nginx/nginx.conf

RUN wget https://github.com/jwilder/dockerize/releases/download/v0.0.1/dockerize-linux-amd64-v0.0.1.tar.gz
RUN tar -C /usr/local/bin -xvzf dockerize-linux-amd64-v0.0.1.tar.gz

ADD dockerize /usr/local/bin/dockerize

EXPOSE 80

CMD dockerize nginx

Nginx logs to two different log files under /var/log/nginx by default. It would be nice have the nginx access and error log streamed to the console if your run this container interactively or if you docker logs nginx so you can see what’s happening.

We can fix that by passing -stdout <file> and -stderr <file> as command-line options. These can also be passed multiple times if there are several files to tail.

CMD dockerize -stdout /var/log/nginx/access.log -stderr /var/log/nginx/error.log nginx

Now when you run the container, nginx logs are available via docker logs nginx.

To demonstrate the templating, we’ll make this a into a more generic proxy server than can be configured using environment variables. We’ll define the environment variable PROXY_URL to be a URL of a site to proxy.

PROXY_URL="http://jasonwilder.com"

When the container is started with this variable, dockerize will use it to generate an nginx server location path.

Here’s the template:

{% raw %}

server {
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;

    root /usr/share/nginx/html;
    index index.html index.htm;

    # Make site accessible from http://localhost/
    server_name localhost;

    location / {
      access_log off;
      proxy_pass {{ .Env.PROXY_URL }};
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

{% endraw %}

Then our final Dockerfile would look like:

FROM ubuntu:14.04

# Install Nginx.
RUN echo "deb http://ppa.launchpad.net/nginx/stable/ubuntu trusty main" > /etc/apt/sources.list.d/nginx-stable-trusty.list
RUN echo "deb-src http://ppa.launchpad.net/nginx/stable/ubuntu trusty main" >> /etc/apt/sources.list.d/nginx-stable-trusty.list
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C300EE8C
RUN apt-get update
RUN apt-get install -y wget nginx

RUN echo "daemon off;" >> /etc/nginx/nginx.conf

RUN wget https://github.com/jwilder/dockerize/releases/download/v0.0.1/dockerize-linux-amd64-v0.0.1.tar.gz
RUN tar -C /usr/local/bin -xvzf dockerize-linux-amd64-v0.0.1.tar.gz

ADD default.tmpl /etc/nginx/sites-available/default.tmpl

EXPOSE 80

CMD dockerize -template /etc/nginx/sites-available/default.tmpl:/etc/nginx/sites-available/default -stdout /var/log/nginx/access.log -stderr /var/log/nginx/error.log nginx

The -template <src>:<dest> options indicates that the template at /etc/nginx/sites-available/default.tmpl should be generated and written to /etc/nginx/sites-available/default. Multiple templates can be specified as well.

Run this container with:

$ docker run -p 80:80 -e PROXY_URL="http://jasonwilder.com" --name nginx -d nginx

You can then access http://localhost and it will proxy to this site.

This is a simplistic example but it can easily be extended using the embedded split function and range statement to handle multiple proxy values or other options. There are also a few other template functions available.

Conclusion

While this example is somewhat simplistic, many applications need some shims to make them run well within docker. dockerize is a generic utility to help with this process.

You can find the code at jwilder/dockerize.

Projects

Fri, 10 Oct 2014 00:00:00 UTC

Some projects I develop and maintain:

docker-gen - Generate configuration files, scripts and run commands when docker containers are started and stopped.
nginx-proxy - Automated virtual host proxying using nginx for Docker containers.
docker-squash - Squash docker images to make them smaller.
docker-discover & docker-register - Service registration and discovery containers for docker using Etcd and Haproxy.
dockerize - A simple way to dockerize applications.
gofana - Self-contained grafana server supporting HTTPS, Auth, and dashboard storage.
influxdb - Distributed time-series database.
galayx - Docker microservice PaaS.
shuttle - Dynamic HTTP(S)/TCP/UDP Proxy
mongodb-tools - Tools for analyzing mongo DB indexes and collections

Squashing Docker Images

Tue, 19 Aug 2014 00:00:00 UTC

A common problem when building docker images is that they can get big quickly. A base image can be a tens to hundreds of MB in size. Installing a few packages and running a build can easily create a image that is a 1GB or larger. If you build an application in your container, build artifacts can stick around and end up getting deployed.

Large images are problematic when you start publishing images to a registry. More layers creates more requests and larger layers take longer to transfer. Unfortunately, deleting things in later layers does not actually remove them from the image due to the way AUFS layers work.

There are a few options to address this problem but this post will show you how you can squash your images to make them smaller without requiring big changes to your development and deployment workflow.

docker-squash

Since this is a problem that affects me currently, I created a tool to squash images before pushing them to a registry. docker-squashis a standalone Go application that works similarly to the idea described in 332. It’s intended to be used as a publishing tool in your workflow and would be run before pushing to a registry.

The way it works is that you save, squash and load an image with something like docker save <ID> | docker-squash -t <TAG> [-from <ID>] | docker load.

The resulting image has all of the layers beneath the initial FROM layer squashed into a single layer. The other layers defining PORT, etc.. are retained as well.

The default options retains the base image layer so that it does not need to be repeatedly transferred when pushing and pulling updates to the image.

Example

I have a simple Go test image called jwilder/whoami that I’ll use as an example. When you run it, it listens on a port 8080 and returns the hostname of the container over HTTP.

Starting Image

Viewing it’s history shows that it’s pretty big (423.7MB) for just simple 20 line Go app.

$ docker images jwilder/whoami
REPOSITORY          TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
jwilder/whoami      latest              63e174c2ca3d        29 minutes ago       423.7 MB

Viewing the history shows how the size is broken down between the layers:

$ docker history jwilder/whoami:latest
IMAGE               CREATED             CREATED BY                                      SIZE
63e174c2ca3d        14 seconds ago      /bin/sh -c #(nop) CMD [/app/http]               0 B
e4eea4411c00        14 seconds ago      /bin/sh -c #(nop) EXPOSE map[8000/tcp:{}]       0 B
c50f2b65cab3        14 seconds ago      /bin/sh -c #(nop) ENV PORT=8000                 0 B
589338fba5eb        15 seconds ago      /bin/sh -c go build -o http                     7.031 MB
651626d6e364        15 seconds ago      /bin/sh -c #(nop) WORKDIR /app                  0 B
8dfc0bb00563        16 seconds ago      /bin/sh -c #(nop) ADD dir:78239d85b32dd28e4cb   21.8 kB
fc294d2b22cb        17 seconds ago      /bin/sh -c apt-get update && apt-get install    191.3 MB
c4ff7513909d        3 days ago          /bin/sh -c #(nop) CMD [/bin/bash]               0 B
cc58e55aa5a5        3 days ago          /bin/sh -c apt-get update && apt-get dist-upg   32.67 MB
0ea0d582fd90        3 days ago          /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/   1.895 kB
d92c3c92fa73        3 days ago          /bin/sh -c rm -rf /var/lib/apt/lists/*          0 B
9942dd43ff21        3 days ago          /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic   194.5 kB
1c9383292a8f        3 days ago          /bin/sh -c #(nop) ADD file:c1472c26527df28498   192.5 MB
511136ea3c5a        14 months ago

Breaking this down:

225.4MB - 511136ea3c5a..c4ff7513909d is the ubuntu:14.04 base image.
191.3MB - fc294d2b22cb is installing the Go SDK (golang-go)
7MB - 589338fba5eb builds my app

Clean Up

I don’t want to ship the Go SDK with my image. There is also a bunch of left-over apt-get cache data, as well as some extra packages I don’t need. I’ll remove them in a new container.

$ docker run -it jwilder/whoami:latest /bin/bash
root@5fe8a50718c3:/app# apt-get purge -y man  perl-modules vim-common vim-tiny \
> libpython3.4-stdlib:amd64 python3.4-minimal xkb-data \
> libx11-data eject python3 locales golang-go
...
$ root@5fe8a50718c3:/app# apt-get clean autoclean
$ root@5fe8a50718c3:/app# apt-get autoremove -y
$ root@5fe8a50718c3:/app# rm -rf /var/lib/{apt,dpkg,cache,log}/
$ root@5fe8a50718c3:/app# exit

Squash The Image

Next I need to create a image from that container:

$ docker commit 5fe8a50718c3
49b5a7a88d5353fe77204ad5591a3ef100fc2807a9d6dce979fd1b17a73a68d6

Then I’ll save, squash and load it. I’m tagging the new image with -t jwilder/whoami:squash:

$ docker save 49b5a7a88d5 | sudo docker-squash -t jwilder/whoami:squash | docker load

If you run docker-squash with the -verbose option, you can see what it’s actually doing to the image.

$ docker save 49b5a7a88d5 | sudo docker-squash -t jwilder/whoami:squash -verbose | docker load
Loading export from STDIN using /tmp/docker-squash683466637 for tempdir
Loaded image w/ 15 layers
Extracting layers...
  -  /tmp/docker-squash683466637/49b5a7a88d5353fe77204ad5591a3ef100fc2807a9d6dce979fd1b17a73a68d6/layer.tar
  -  /tmp/docker-squash683466637/651626d6e364ccc22ac990ba95cd0aab9256c56055087cc9a5a1790cea5250b9/layer.tar
  -  /tmp/docker-squash683466637/c50f2b65cab3b74f9bdb6f616b36f132b9a182ed883d03f11173e32fa39ab599/layer.tar
  -  /tmp/docker-squash683466637/d92c3c92fa73ba974eb409217bb86d8317b0727f42b73ef5a05153b729aaf96b/layer.tar
  -  /tmp/docker-squash683466637/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/layer.tar
  -  /tmp/docker-squash683466637/1c9383292a8ff4c4196ff4ffa36e5ff24cb217606a8d1f471f4ad27c4690e290/layer.tar
  -  /tmp/docker-squash683466637/589338fba5eb5cc32a25a036975a5e0938f12eff0dc70b661363c13ef1a192a5/layer.tar
  -  /tmp/docker-squash683466637/63e174c2ca3d53e2b7639a440940e16e15c1970e6ad16f740ffdcc60e59e0324/layer.tar
  -  /tmp/docker-squash683466637/9942dd43ff211ba917d03637006a83934e847c003bef900e4808be8021dca7bd/layer.tar
  -  /tmp/docker-squash683466637/0ea0d582fd9027540c1f50c7f0149b237ed483d2b95ac8d107f9db5a912b4240/layer.tar
  -  /tmp/docker-squash683466637/8dfc0bb00563dab615dfcc28ab3e338089f5b1d71d82d731c18cbe9f7667435f/layer.tar
  -  /tmp/docker-squash683466637/c4ff7513909dedf4ddf3a450aea68cd817c42e698ebccf54755973576525c416/layer.tar
  -  /tmp/docker-squash683466637/cc58e55aa5a53b572f3b9009eb07e50989553b95a1545a27dcec830939892dba/layer.tar
  -  /tmp/docker-squash683466637/e4eea4411c0065f8b0c7cf6be31dd58daa5ac04d8c64d54537cbfce2eb8e3413/layer.tar
  -  /tmp/docker-squash683466637/fc294d2b22cb53cb2440ff6fece18813ee7363f5198f5e20346abfcf7cce54fe/layer.tar
Inserted new layer 27935276f797 after 1c9383292a8f
  -  511136ea3c5a
  -  1c9383292a8f /bin/sh -c #(nop) ADD file:c1472c26527df28498744f9e9e8a8304c
  -> 27935276f797 /bin/sh -c #(squash) from 1c9383292a8f
  -  9942dd43ff21 /bin/sh -c echo '#!/bin/sh' > /usr/sbin/policy-rc.d  && echo
  -  d92c3c92fa73 /bin/sh -c rm -rf /var/lib/apt/lists/*
  -  0ea0d582fd90 /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/\1/g' /etc/apt/
  -  cc58e55aa5a5 /bin/sh -c apt-get update && apt-get dist-upgrade -y && rm -
  -  c4ff7513909d /bin/sh -c #(nop) CMD [/bin/bash]
  -  fc294d2b22cb /bin/sh -c apt-get update && apt-get install -y golang-go
  -  8dfc0bb00563 /bin/sh -c #(nop) ADD dir:78239d85b32dd28e4cb1d81ace7ffd32b8
  -  651626d6e364 /bin/sh -c #(nop) WORKDIR /app
  -  589338fba5eb /bin/sh -c go build -o http
  -  c50f2b65cab3 /bin/sh -c #(nop) ENV PORT=8000
  -  e4eea4411c00 /bin/sh -c #(nop) EXPOSE map[8000/tcp:{}]
  -  63e174c2ca3d /bin/sh -c #(nop) CMD [/app/http]
  -  49b5a7a88d53 /bin/bash
Squashing from 27935276f797 into 27935276f797
  -  Deleting whiteouts
  -  Rewriting child history
  -  Removing 9942dd43ff21. Squashed. (/bin/sh -c echo '#!/bin/sh' > /usr/sbin/policy-...)
  -  Removing d92c3c92fa73. Squashed. (/bin/sh -c rm -rf /var/lib/apt/lists/*)
  -  Removing 0ea0d582fd90. Squashed. (/bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/\1...)
  -  Removing cc58e55aa5a5. Squashed. (/bin/sh -c apt-get update && apt-get dist-upgra...)
  -  Replacing c4ff7513909d w/ new layer 72391e640b52 (/bin/sh -c #(nop) CMD [/bin/bash])
  -  Removing fc294d2b22cb. Squashed. (/bin/sh -c apt-get update && apt-get install -y...)
  -  Removing 8dfc0bb00563. Squashed. (/bin/sh -c #(nop) ADD dir:78239d85b32dd28e4cb1d...)
  -  Replacing 651626d6e364 w/ new layer bd7b4b11874a (/bin/sh -c #(nop) WORKDIR /app)
  -  Removing 589338fba5eb. Squashed. (/bin/sh -c go build -o http)
  -  Replacing c50f2b65cab3 w/ new layer e4af8871b961 (/bin/sh -c #(nop) ENV PORT=8000)
  -  Replacing e4eea4411c00 w/ new layer 6803497b6a61 (/bin/sh -c #(nop) EXPOSE map[8000/tcp:{}])
  -  Replacing 63e174c2ca3d w/ new layer 40b8c7c33bba (/bin/sh -c #(nop) CMD [/app/http])
  -  Removing 49b5a7a88d53. Squashed. (/bin/bash)
Tarring up squashed layer 27935276f797
Removing extracted layers
Tagging 40b8c7c33bba as jwilder/whoami:squash
Tarring new image to STDOUT
Done. New image created.
  -  40b8c7c33bba Less than a second /bin/sh -c #(nop) CMD [/app/http] 3.072 kB
  -  6803497b6a61 Less than a second /bin/sh -c #(nop) EXPOSE map[8000/tcp:{}] 3.072 kB
  -  e4af8871b961 Less than a second /bin/sh -c #(nop) ENV PORT=8000 3.072 kB
  -  bd7b4b11874a Less than a second /bin/sh -c #(nop) WORKDIR /app 3.072 kB
  -  72391e640b52 Less than a second /bin/sh -c #(nop) CMD [/bin/bash] 3.072 kB
  -  27935276f797 1 seconds /bin/sh -c #(squash) from 1c9383292a8f 39.49 MB
  -  1c9383292a8f 3 days /bin/sh -c #(nop) ADD file:c1472c26527df28498744f9e9e8a83... 201.6 MB
  -  511136ea3c5a 14 months  1.536 kB
Removing tempdir /tmp/docker-squash683466637

My squashed layer is down from ~198MB to 39.5MB. Roughly 80% smaller. I should be able to get it down to ~7MB if I squash some of the apt-get updates my build pulled in with the upstream ubuntu:14.04 base image and use a custom base image.

If I was to create a custom base image, I would squash that entire image down to a single layer using -from root and update my Dockerfile to use it as the FROM image.

This is what -from root looks like with my example images:

$ docker save 49b5a7a88d5 | sudo docker-squash -t jwilder/whoami:squash -verbose -from root | docker load
Loading export from STDIN using /tmp/docker-squash627981871 for tempdir
Loaded image w/ 15 layers
Extracting layers...
  -  /tmp/docker-squash627981871/d92c3c92fa73ba974eb409217bb86d8317b0727f42b73ef5a05153b729aaf96b/layer.tar
  -  /tmp/docker-squash627981871/cc58e55aa5a53b572f3b9009eb07e50989553b95a1545a27dcec830939892dba/layer.tar
  -  /tmp/docker-squash627981871/1c9383292a8ff4c4196ff4ffa36e5ff24cb217606a8d1f471f4ad27c4690e290/layer.tar
  -  /tmp/docker-squash627981871/63e174c2ca3d53e2b7639a440940e16e15c1970e6ad16f740ffdcc60e59e0324/layer.tar
  -  /tmp/docker-squash627981871/8dfc0bb00563dab615dfcc28ab3e338089f5b1d71d82d731c18cbe9f7667435f/layer.tar
  -  /tmp/docker-squash627981871/c4ff7513909dedf4ddf3a450aea68cd817c42e698ebccf54755973576525c416/layer.tar
  -  /tmp/docker-squash627981871/0ea0d582fd9027540c1f50c7f0149b237ed483d2b95ac8d107f9db5a912b4240/layer.tar
  -  /tmp/docker-squash627981871/9942dd43ff211ba917d03637006a83934e847c003bef900e4808be8021dca7bd/layer.tar
  -  /tmp/docker-squash627981871/c50f2b65cab3b74f9bdb6f616b36f132b9a182ed883d03f11173e32fa39ab599/layer.tar
  -  /tmp/docker-squash627981871/49b5a7a88d5353fe77204ad5591a3ef100fc2807a9d6dce979fd1b17a73a68d6/layer.tar
  -  /tmp/docker-squash627981871/589338fba5eb5cc32a25a036975a5e0938f12eff0dc70b661363c13ef1a192a5/layer.tar
  -  /tmp/docker-squash627981871/651626d6e364ccc22ac990ba95cd0aab9256c56055087cc9a5a1790cea5250b9/layer.tar
  -  /tmp/docker-squash627981871/e4eea4411c0065f8b0c7cf6be31dd58daa5ac04d8c64d54537cbfce2eb8e3413/layer.tar
  -  /tmp/docker-squash627981871/fc294d2b22cb53cb2440ff6fece18813ee7363f5198f5e20346abfcf7cce54fe/layer.tar
  -  /tmp/docker-squash627981871/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/layer.tar
Inserted new layer 6996f41c1688 after 511136ea3c5a
  -  511136ea3c5a
  -> 6996f41c1688 /bin/sh -c #(squash) from 511136ea3c5a
  -  1c9383292a8f /bin/sh -c #(nop) ADD file:c1472c26527df28498744f9e9e8a8304c
  -  9942dd43ff21 /bin/sh -c echo '#!/bin/sh' > /usr/sbin/policy-rc.d  && echo
  -  d92c3c92fa73 /bin/sh -c rm -rf /var/lib/apt/lists/*
  -  0ea0d582fd90 /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/\1/g' /etc/apt/
  -  cc58e55aa5a5 /bin/sh -c apt-get update && apt-get dist-upgrade -y && rm -
  -  c4ff7513909d /bin/sh -c #(nop) CMD [/bin/bash]
  -  fc294d2b22cb /bin/sh -c apt-get update && apt-get install -y golang-go
  -  8dfc0bb00563 /bin/sh -c #(nop) ADD dir:78239d85b32dd28e4cb1d81ace7ffd32b8
  -  651626d6e364 /bin/sh -c #(nop) WORKDIR /app
  -  589338fba5eb /bin/sh -c go build -o http
  -  c50f2b65cab3 /bin/sh -c #(nop) ENV PORT=8000
  -  e4eea4411c00 /bin/sh -c #(nop) EXPOSE map[8000/tcp:{}]
  -  63e174c2ca3d /bin/sh -c #(nop) CMD [/app/http]
  -  49b5a7a88d53 /bin/bash
Squashing from 6996f41c1688 into 6996f41c1688
  -  Deleting whiteouts
  -  Rewriting child history
  -  Removing 1c9383292a8f. Squashed. (/bin/sh -c #(nop) ADD file:c1472c26527df2849874...)
  -  Removing 9942dd43ff21. Squashed. (/bin/sh -c echo '#!/bin/sh' > /usr/sbin/policy-...)
  -  Removing d92c3c92fa73. Squashed. (/bin/sh -c rm -rf /var/lib/apt/lists/*)
  -  Removing 0ea0d582fd90. Squashed. (/bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/\1...)
  -  Removing cc58e55aa5a5. Squashed. (/bin/sh -c apt-get update && apt-get dist-upgra...)
  -  Replacing c4ff7513909d w/ new layer 09a62007c3f3 (/bin/sh -c #(nop) CMD [/bin/bash])
  -  Removing fc294d2b22cb. Squashed. (/bin/sh -c apt-get update && apt-get install -y...)
  -  Removing 8dfc0bb00563. Squashed. (/bin/sh -c #(nop) ADD dir:78239d85b32dd28e4cb1d...)
  -  Replacing 651626d6e364 w/ new layer b4f0dec85412 (/bin/sh -c #(nop) WORKDIR /app)
  -  Removing 589338fba5eb. Squashed. (/bin/sh -c go build -o http)
  -  Replacing c50f2b65cab3 w/ new layer cd499c2d09ef (/bin/sh -c #(nop) ENV PORT=8000)
  -  Replacing e4eea4411c00 w/ new layer 653dfab45562 (/bin/sh -c #(nop) EXPOSE map[8000/tcp:{}])
  -  Replacing 63e174c2ca3d w/ new layer f7f7eb6aae54 (/bin/sh -c #(nop) CMD [/app/http])
  -  Removing 49b5a7a88d53. Squashed. (/bin/bash)
Tarring up squashed layer 6996f41c1688
Removing extracted layers
Tagging f7f7eb6aae54 as jwilder/whoami:squash
Tarring new image to STDOUT
Done. New image created.
  -  f7f7eb6aae54 Less than a second /bin/sh -c #(nop) CMD [/app/http] 3.072 kB
  -  653dfab45562 Less than a second /bin/sh -c #(nop) EXPOSE map[8000/tcp:{}] 3.072 kB
  -  cd499c2d09ef Less than a second /bin/sh -c #(nop) ENV PORT=8000 3.072 kB
  -  b4f0dec85412 Less than a second /bin/sh -c #(nop) WORKDIR /app 3.072 kB
  -  09a62007c3f3 Less than a second /bin/sh -c #(nop) CMD [/bin/bash] 3.072 kB
  -  6996f41c1688 2 seconds /bin/sh -c #(squash) from 511136ea3c5a 111.9 MB
  -  511136ea3c5a 14 months  1.536 kB
Removing tempdir /tmp/docker-squash627981871

That gets my full build down to a single layer of 106.2MB:

REPOSITORY          TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
jwilder/whoami      squash              f7f7eb6aae54        About a minute ago   106.2 MB
jwilder/whoami      latest              63e174c2ca3d        29 minutes ago       423.7 MB

Since I typically have base images already loaded onto my docker hosts, I would continue to use the default squash settings and retain my parent base image so that I’m only tranferring the changes for each image.

Conclusion

Squashing images with docker-squash can reduce image sizes significantly. Since it only needs to be run before publishing to a registry, the regular docker build caching is not changed and you don’t lose any of the benefits of using Docker. Similarly, it does not require complex Dockerfile setups to get good results so you can start using it on existing projects with little effort.

If you want to try it out or learn more about it works, you can get it from github.

Docker Service Discovery Using Etcd and Haproxy

Tue, 15 Jul 2014 00:00:00 UTC

In a previous post, I showed a way to create an automated nginx reverse proxy for docker containers running on the same host. That setup works fine for front-end web apps, but is not ideal for backend services since they are typically spread across multiple hosts.

This post describes a solution to the backend service problem using service discovery for docker containers.

The architecture we’ll build is modelled after SmartStack, but uses etcd instead Zookeeper and two docker containers running docker-gen and haproxy instead of nerve and synapse .

How It Works

Similar to SmartStack, we have components to serve as a registry (etcd), a registration side-kick process (docker-register), discovery side-kick process (docker-discover), some backend services (whoami) and finally a consumers (ubuntu/curl).

The registration and discovery components work as appliances alongside the the application containers so there is no embedded registration or discovery code in the backend or consumer containers. They just listen on ports or connect to other local ports.

Service Registry - Etcd

Before anything can be registered, we need some place to track registration entries (i.e. IP and ports of services). We’re using etcd because it has a simple programming model for service registration and supports TTLs for keys and directories.

Usually, you would run 3 or 5 etcd nodes but I’m just using one to keep things simple.

There is no reason why we could not use Consul or any other storage option that supports TTL expiration.

To start etcd:

$ docker run -d --name etcd -p 4001:4001 -p 7001:7001 coreos/etcd

Service Registration - docker-register

Registering service containers is handled by the jwilder/docker-register container. This container registers other containers running on the same host in etcd. Containers we want registered must expose a port. Containers running the same image on different hosts are grouped together in etcd and will form a load-balanced cluster. How containers are groups is somewhat arbitrary and I’ve chosen the container image name for this walkthrough. In a real deployment, you would likely want to group things by environment, service version, or other meta-data.

(The current implementation only supports one port per container and assumes it is TCP currently. There is no reason why multiple ports and types could not be supported as well as different grouping attributes.)

docker-register uses docker-gen along with a python script as a template. It dynamically generates a script that, when run, will register each container’s IP and PORT under a /backends directory.

docker-gen takes care of monitoring docker events and calling the generated script on an interval to ensure TTLs are kept up to date. If docker-register is stopped, the registrations expire.

To start docker-register, we need to pass in the host’s external IP where other hosts can reach it’s containers as well as the address of your etcd host. docker-gen requires access to the docker daemon in order to call it’s API so we bind mount the docker unix socket into the container as well.

$ HOST_IP=$(hostname --all-ip-addresses | awk '{print $1}')
$ ETCD_HOST=w.x.y.z:4001
$ docker run --name docker-register -d -e HOST_IP=$HOST_IP -e ETCD_HOST=$ETCD_HOST -v /var/run/docker.sock:/var/run/docker.sock -t jwilder/docker-register

Service Discovery - docker-discover

Discovering services is handled by the jwilder/docker-discover container. docker-discover polls etcd periodically and generates an haproxy config with listeners for each type of registered service.

For example, containers running jwilder/whoami are registered under /backends/whoami/<id> and are exposed on host port 8000.

Other containers that need to call the jwilder/whoami service, can send requests to docker bridge IP:8000 or host IP:8000.

If any of the backend services goes down, haproxy health checks remove it from the pool and will retry the request on a healthy host. This ensure that backend services can be started and stopped as needed as well as handling inconsistencies in the the registration information while ensuring minimal client impact.

Finally, stats can be viewed by accessing port 1936 on the docker-discover container.

To run docker-discover:

$ ETCD_HOST=w.x.y.z:4001
$ docker run -d --net host --name docker-discover -e ETCD_HOST=$ETCD_HOST -p 127.0.0.1:1936:1936 -t jwilder/docker-discover

We’re using --net host so that the container uses the host’s network stack. When this container binds port 8000, it’s actually binding on the host’s network. This simplifies the proxy setup.

AWS Demo

We’ll run the full thing on four AWS hosts: an etcd host, a client host and two backend hosts. The backend service is a simple Golang HTTP server that returns it’s hostname (container ID).

Etcd Host

First we start our etcd registry:

$ hostname --all-ip-addresses | awk '{print $1}'
10.170.71.226

$ docker run -d --name etcd -p 4001:4001 -p 7001:7001 coreos/etcd

Our etcd address is 10.170.71.226. We’ll use that on the other hosts. If we were running this is a live environment, we could assign an EIP and DNS address to it to make it easier to configure.

Backend Hosts

Next, we start the the services and docker-register on each host. The service is configured to listen on port 8000 in the container and we let docker publish it on an random host port.

On backend host 1:

$ docker run -d -p 8000:8000 --name whoami -t jwilder/whoami
736ab83847bb12dddd8b09969433f3a02d64d5b0be48f7a5c59a594e3a6a3541
$ docker run --name docker-register -d -e HOST_IP=$(hostname --all-ip-addresses | awk '{print $1}') -e ETCD_HOST=10.170.71.226:4001 -v /var/run/docker.sock:/var/run/docker.sock -t jwilder/docker-register
77a49f732797333ca0c7695c6b590a64a7d75c14b5ffa0f89f8e0e21ae47ae3e

$ docker ps
CONTAINER ID        IMAGE                            COMMAND                CREATED             STATUS              PORTS                     NAMES
736ab83847bb        jwilder/whoami:latest            /app/http              48 seconds ago      Up 47 seconds       0.0.0.0:49153->8000/tcp   whoami
77a49f732797        jwilder/docker-register:latest   "/bin/sh -c 'docker-   28 minutes ago      Up 28 minutes                                 docker-register

On backend host 2:

$ docker run -d -p 8000:8000 --name whoami -t jwilder/whoami
4eb0498e52076275ee0702d80c0d8297813e89d492cdecbd6df9b263a3df1c28
$ docker run --name docker-register -d -e HOST_IP=$(hostname --all-ip-addresses | awk '{print $1}') -e ETCD_HOST=10.170.71.226:4001 -v /var/run/docker.sock:/var/run/docker.sock -t jwilder/docker-register
832e77c83591cb33bba53859153eb91d897f5a278a74d4ec1f66bc9b97deb221

$ docker ps
CONTAINER ID        IMAGE                            COMMAND                CREATED             STATUS              PORTS                     NAMES
4eb0498e5207        jwilder/whoami:latest            /app/http              59 seconds ago      Up 58 seconds       0.0.0.0:49154->8000/tcp   whoami
832e77c83591        jwilder/docker-register:latest   "/bin/sh -c 'docker-   34 minutes ago      Up 34 minutes                                 docker-register

Client Host

On the client host, we need to start docker-discover and a client container. For the client container, I’m using Ubuntu Trusty and will make some curl requests.

First start docker-discover:

$ docker run -d --net host --name docker-discover -e ETCD_HOST=10.170.71.226:4001 -p 127.0.0.1:1936:1936 -t jwilder/docker-discover

Then, start a sample client container and pass in a HOST_IP. We’re using the eth0 address but could also use docker0 IP. We’re passing this in as an environment variable since it is configuration that will vary between deploys.

$ docker run -e HOST_IP=$(hostname --all-ip-addresses | awk '{print $1}') -i -t ubuntu:14.04 /bin/bash
$ root@2af5f52de069:/# apt-get update && apt-get -y install curl

Then, make some requests to the whoami service port 8000 to see them load-balanced.

$ root@2af5f52de069:/# curl $HOST_IP:8000
I'm 4eb0498e5207
$ root@2af5f52de069:/# curl $HOST_IP:8000
I'm 736ab83847bb
$ root@2af5f52de069:/# curl $HOST_IP:8000
I'm 4eb0498e5207
$ root@2af5f52de069:/# curl $HOST_IP:8000
I'm 736ab83847bb

We can start some more instances on the backends:

$ docker run -d -p :8000 --name whoami-2 -t jwilder/whoami
$ docker run -d -p :8000 --name whoami-3 -t jwilder/whoami

$ docker ps
CONTAINER ID        IMAGE                            COMMAND                CREATED             STATUS              PORTS                     NAMES
5d5c12c96192        jwilder/whoami:latest            /app/http              3 seconds ago       Up 1 seconds        0.0.0.0:49156->8000/tcp   whoami-2
bb2a408b8ec5        jwilder/whoami:latest            /app/http              21 seconds ago      Up 20 seconds       0.0.0.0:49155->8000/tcp   whoami-3
4eb0498e5207        jwilder/whoami:latest            /app/http              2 minutes ago       Up 2 minutes        0.0.0.0:49154->8000/tcp   whoami
832e77c83591        jwilder/docker-register:latest   "/bin/sh -c 'docker-   36 minutes ago      Up 36 minutes                                 docker-register

And make some requests again on the client hosts:

$ root@2af5f52de069:/# curl $HOST_IP:8000
I'm 736ab83847bb
$ root@2af5f52de069:/# curl $HOST_IP:8000
I'm 4eb0498e5207
$ root@2af5f52de069:/# curl $HOST_IP:8000
I'm bb2a408b8ec5
$ root@2af5f52de069:/# curl $HOST_IP:8000
I'm 5d5c12c96192
$ root@2af5f52de069:/# curl $HOST_IP:8000
I'm 736ab83847bb

Finally, we can shutdown some some containers and routes will be updated. This kills everything on backend2.

$ docker kill 5d5c12c96192 bb2a408b8ec5 4eb0498e5207

$ root@2af5f52de069:/# curl $HOST_IP:8000
I'm 736ab83847bb
$ root@2af5f52de069:/# curl $HOST_IP:8000
I'm 67c3cccbb8ba
$ root@2af5f52de069:/# curl $HOST_IP:8000
I'm 736ab83847bb
$ root@2af5f52de069:/# curl $HOST_IP:8000
I'm 67c3cccbb8ba

If we wanted to see how haproxy is balancing traffic or monitor for errors, we can access the client’s host port 1936 in a web browser.

Wrapping Up

While there are a lot of different ways to implement service discovery, SmartStack’s sidekick style of registration and proxying keeps application code simple and easy to integrate in a distributed environment and really fits well with Docker containers.

Similarly, Docker’s event and container APIs facilitate service registration and discovery with registration services such as etcd.

The code for docker-register and docker-discover is on github. While both are functional there is a lot that can be improved. Please feel fee to submit or suggest improvements.

Automated Nginx Reverse Proxy for Docker

Tue, 25 Mar 2014 00:00:00 UTC

A reverse proxy server is a server that typically sits in front of other web servers in order to provide additional functionality that the web servers may not provide themselves.

For example, a reverse proxy can provide SSL termination, load balancing, request routing, caching, compression or even A/B testing.

When running web services in docker containers, it can be useful to run a reverse proxy in front of the containers to simplify depoyment.

Why Use A Reverse Proxy With Docker

Docker containers are assigned random IPs and ports which makes addressing them much more complicated from a client perspsective. By default, the IPs and ports are private to the host and cannot be accessed externally unless they are bound to the host.

Binding the container to the hosts port can prevent multiple containers from running on the same host. For example, only one container can bind to port 80 at a time. This also complicates rolling out new versions of the container without downtime since the old container must be stopped before the new one is started.

A reverse proxy can help with these issues as well as improve availabilty by facilitating zero-downtime deployments.

Generating Reverse Proxy Configs

Setting up a reverse proxy configuration can be complicated when containers are started and stopped. Typically the configuration needs to be updated manually which is error prone and time consuming.

Fortunately, Docker provides a remote API to inspect containers and access their IP, Ports and other configuration meta-data. In addition, it also provides a real-time events API that can be used for notifications when containers are started and stopped. These APIs can be used to generate a reverse proxy config automatically.

docker-gen is a small utility that uses these APIs and exposes container meta-data to templates. Templates are rendered and an optional notification command can be run to restart the service.

Using docker-gen, we can generate Nginx config files automatically and reload nginx when they change. The same approach can also be used for docker log management.

Nginx Reverse Proxy for Docker

This example nginx template can be used to generate a reverse proxy configuration for docker containers using virtual hosts for routing. The template is implemented using the golang text/template package. It uses a custom groupBy template function to group the running containers by their VIRTUAL_HOST environment variable. This simplifies iterating over the containers to generate a load-balanced backend and also enables zero-downtime deployments.

{{ range $host, $containers := groupBy $ "Env.VIRTUAL_HOST" }}
upstream {{ $host }} {

{{ range $index, $value := $containers }}
    {{ with $address := index $value.Addresses 0 }}
    server {{ $address.IP }}:{{ $address.Port }};
    {{ end }}
{{ end }}

}

server {
    #ssl_certificate /etc/nginx/certs/demo.pem;
    #ssl_certificate_key /etc/nginx/certs/demo.key;

    gzip_types text/plain text/css application/json application/x-javascript
               text/xml application/xml application/xml+rss text/javascript;

    server_name {{ $host }};

    location / {
        proxy_pass http://{{ $host }};
        include /etc/nginx/proxy_params;
    }
}
{{ end }}

The template can be run with docker-gen using:

docker-gen -only-exposed -watch -notify "/etc/init.d/nginx reload" templates/nginx.tmpl /etc/nginx/sites-enabled/default

-only-exposed - Only use containers that have exposed ports.
-watch - After starting up, watch for docker container events and regenerate the template.
-notify "/etc/init.d/nginx reload" - Reload the nginx config after the template is generated.
templates/nginx.tmpl - The nginx template.
/etc/nginx/sites-enabled/default - Destination file.

This is the rendered template with two containers configured with VIRTUAL_HOST=demo1.localhost and one with VIRTUAL_HOST=demo2.localhost

upstream demo1.localhost {
    server 172.17.0.4:5000;
    server 172.17.0.3:5000;
}

server {
    #ssl_certificate /etc/nginx/certs/demo.pem;
    #ssl_certificate_key /etc/nginx/certs/demo.key;

    gzip_types text/plain text/css application/json application/x-javascript
               text/xml application/xml application/xml+rss text/javascript;

    server_name demo1.localhost;

    location / {
        proxy_pass http://demo.localhost;
        include /etc/nginx/proxy_params;
    }
}

upstream demo2.localhost {
    server 172.17.0.5:5000;
}

server {
    #ssl_certificate /etc/nginx/certs/demo.pem;
    #ssl_certificate_key /etc/nginx/certs/demo.key;

    gzip_types text/plain text/css application/json application/x-javascript
               text/xml application/xml application/xml+rss text/javascript;

    server_name demo2.localhost;

    location / {
        proxy_pass http://demo2.localhost;
        include /etc/nginx/proxy_params;
    }
}

Try It Out

I created a trusted build with this setup to make it easier to try it out:

Run nginx-proxy container:

$ docker run -d -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock -t jwilder/nginx-proxy

Start your containers with a VIRTUAL_HOST environment variables:

$ docker run -e VIRTUAL_HOST=foo.bar.com -t ...

If you need HTTPS, would like to run docker-gen in a separate container from nginx, Websocket support or other features, take a look at the github project for more information.

Conclusion

Generating nginx reverse proxy configs for docker containers can be automated using the Docker APIs and some basic templating. This can simplify deployments as well as improve availability.

While this works well for containers running on a single host, generating configs for remote hosts requires service discovery. Take a look at docker service discovery for a solution to that problem.

Update: There’s a few other posts with similar ideas and variations that are worth checking out:

Using Nginx, Confd, and Docker for Zero-Downtime Web Update - Brian Ketelsen
Docker Events - Michael Crosby
Haproxy As A Static Reverse Proxy for Docker Containers - Oskar Hane

Docker Log Management Using Fluentd

Mon, 17 Mar 2014 00:00:00 UTC

Docker is an open-source project to easily create lighweight, portable and self-sufficient containers for applications. Docker allows you to run many isolated applications on a single host without the weight of running virtual machines.

One of the problems with the current versions of docker is managing logs. Each container runs a single process and the output of that process is saved by docker to a location on the host.

There are a few operational issues with this currently:

This log file grows indefinitely. Docker logs each line as a JSON message which can cause this file to grow quickly and exceed the disk space on the host since it’s not rotated automatically.
The docker logs command returns all recorded logs each time it’s run. Any long running process that is a little verbose can be difficult to examine.
Logs under the containers /var/log or other locations are not easily visible or accessible.

Docker Logging Options

While logging in docker is evolving, there are several approaches to handling logs with docker currently:

Collection Inside Container - Each container starts up a log collection process in addition to the application that will be running. baseimage-docker uses runit along with syslog as an example.
Collection Outside Container - A single collection agent runs on the host and containers have a volume mounted from the host where they write their logs.
Collection In Separate Container - This is a slight variation of running the collection agent on the host. The collection agent is also run in a container and volumes from that container are bound to any application containers using the volumes-from docker run option. This Docker and logstash article has an example of this approach.

These approaches work but also have some drawbacks. If collection is performed inside the container, then each container is running duplicate processes that can waste resources. (Running multiple processes in a container seems to be a debated subject even though the docker docs use supervisor as an example.)

If collectionis run outside the container using volumes, you still need to make sure your application logs to those volumes and not stdout/stderr. This might not be possible with all applications. Finally, the containers running still have the container JSON log file that will grow unbounded too.

Using Fluentd With Docker

Another variation of collection outside the container can be done with a centralized logging agent and without binding volumes to the containers. This method works directly against the container’s JSON log file on the host.

When you run a container, the state of the container lives under /var/lib/docker/containers/<id>.

root@precise64:/var/lib/docker/containers/fe38c4124f36d0a5b2a38ea7dd58fe88ac92980286f1f6a7b7ed3ced7c994374# ls -la
total 44
drwx------  3 root root  4096 Mar 14 19:56 .
drwx------ 83 root root 12288 Mar 14 21:53 ..
-rw-r--r--  1 root root   106 Mar 14 19:56 config.env
-rw-r--r--  1 root root  1522 Mar 14 19:56 config.json
-rw-------  1 root root   241 Mar 14 19:56 fe38c4124f36d0a5b2a38ea7dd58fe88ac92980286f1f6a7b7ed3ced7c994374-json.log
-rw-r--r--  1 root root   126 Mar 14 19:56 hostconfig.json
-rw-r--r--  1 root root    13 Mar 14 19:56 hostname
-rw-r--r--  1 root root   181 Mar 14 19:56 hosts
drwxr-xr-x  2 root root  4096 Mar 14 19:56 root

The file fe38c4124f36d0a5b2a38ea7dd58fe88ac92980286f1f6a7b7ed3ced7c994374-json.log is the container log file. Each line is a JSON object and there is a line for every line of input and output from the container.

{"log":"root@c835298de6dd:/# ls\r\n","stream":"stdout","time":"2014-03-14T22:15:15.155863426Z"}
{"log":"bin  boot  dev\u0009etc  home  lib\u0009lib64  media  mnt  opt\u0009proc  root  run  sbin  selinux\u0009srv  sys  tmp  usr  var\r\n","stream":"stdout","time":"2014-03-14T22:15:15.194869963Z"}

fluentd is an open-source data collector that works natively with lines of JSON so you can run a single fluentd instance on the host and configure it to tail each container’s JSON file.

If you need to tail a log file somewhere on the containers file system, you can use the root subdirectory as well. All of the tailed files can then be forwarded to a centralized logging system

This is a sample fluentd.conf file that tails each container’s logs and sends them to stdout.

## File input
## read docker logs with tag=docker.container

<source>
  type tail
  format json
  time_key time
  path /var/lib/docker/containers/c835298de6dde500c78a2444036101bf368908b428ae099ede17cf4855247898/c835298de6dde500c78a2444036101bf368908b428ae099ede17cf4855247898-json.log
  pos_file /var/lib/docker/containers/c835298de6dde500c78a2444036101bf368908b428ae099ede17cf4855247898/c835298de6dde500c78a2444036101bf368908b428ae099ede17cf4855247898-json.log.pos
  tag docker.container.c835298de6dd
  rotate_wait 5
</source>

<source>
  type tail
  format json
  time_key time
  path /var/lib/docker/containers/965c22a2ad1e935cb1476772ebe1ebef0050559b4cbcc7775b936348e7822347/965c22a2ad1e935cb1476772ebe1ebef0050559b4cbcc7775b936348e7822347-json.log
  pos_file /var/lib/docker/containers/965c22a2ad1e935cb1476772ebe1ebef0050559b4cbcc7775b936348e7822347/965c22a2ad1e935cb1476772ebe1ebef0050559b4cbcc7775b936348e7822347-json.log.pos
  tag docker.container.965c22a2ad1e
  rotate_wait 5
</source>

<source>
  type tail
  format json
  time_key time
  path /var/lib/docker/containers/889fe291f590c2c2aa2852856687efbb3e6fdd2faeca84da4bc0be2263f37953/889fe291f590c2c2aa2852856687efbb3e6fdd2faeca84da4bc0be2263f37953-json.log
  pos_file /var/lib/docker/containers/889fe291f590c2c2aa2852856687efbb3e6fdd2faeca84da4bc0be2263f37953/889fe291f590c2c2aa2852856687efbb3e6fdd2faeca84da4bc0be2263f37953-json.log.pos
  tag docker.container.889fe291f590
  rotate_wait 5
</source>

<match docker.**>
  type stdout
</match>

In a live environment, the actual log contents could be sent to an elasticsearch cluster and viewed with kibana or graylog2. Alternatively, there are hosted services that can work with JSON as well.

Since containers ID are unwieldly to work with, I created a simple golang project called docker-gen that can generate arbitrary files using a template from the running docker container data. The example fluentd template in the project was used to generate the sample above.

Although not shown, docker-gen could also generate logrotate config files to rotate the container JSON files to avoid running out of disk space on the host. Hopefully, the docker project will address this in a future release.

Conclusion

This approach provides the following benefits:

The host is able to forward any container’s logs to a central log server using a single collection agent.
It does not require the applications to use syslog or write to a certain volume.
The host can access the container logs as well as any log files on the containers filesystem.
The host can rotate logs for the containers.

The one drawback to this approach is that it accesses the docker file system directly without using the API which means it could break in the future if a future docker release changes how it stores container logs on the host’s file system.

Open-Source Service Discovery

Tue, 04 Feb 2014 00:00:00 UTC

Service discovery is a key component of most distributed systems and service oriented architectures. The problem seems simple at first: How do clients determine the IP and port for a service that exist on multiple hosts?

Usually, you start off with some static configuration which gets you pretty far. Things get more complicated as you start deploying more services. With a live system, service locations can change quite frequently due to auto or manual scaling, new deployments of services, as well as hosts failing or being replaced.

Dynamic service registration and discovery becomes much more important in these scenarios in order to avoid service interruption.

This problem has been addressed in many different ways and is continuing to evolve. We’re going to look at some open-source or openly-discussed solutions to this problem to understand how they work. Specifically, we’ll look at how each solution uses strong or weakly consistent storage, runtime dependencies, client integration options and what the tradeoffs of those features might be.

We’ll start with some strongly consistent projects such as Zookeeper, Doozer and Etcd which are typically used as coordination services but are also used for service registries as well.

We’ll then look at some interesting solutions specifically designed for service registration and discovery. We’ll examine Airbnb’s SmartStack, Netflix’s Eureka, Bitly’s NSQ, Serf, Spotify and DNS and finally SkyDNS.

The Problem

There are two sides to the problem of locating services. Service Registration and Service Discovery.

Service Registration - The process of a service registering its location in a central registry. It usually register its host and port and sometimes authentication credentials, protocols, versions numbers, and/or environment details.
Service Discovery - The process of a client application querying the central registry to learn of the location of services.

Any service registration and discovery solution also has other development and operational aspects to consider:

Monitoring - What happens when a registered service fails? Sometimes it is unregistered immediately, after a timeout, or by another process. Services are usually required to implement a heartbeating mechanism to ensure liveness and clients typically need to be able to handle failed services reliably.
Load Balancing - If multiple services are registered, how do all the clients balance the load across the services? If there is a master, can it be deteremined by a client correctly?
Integration Style - Does the registry only provide a few language bindings, for example, only Java? Does integrating require embedding registration and discovery code into your application or is a sidekick process an option?
Runtime Dependencies - Does it require the JVM, Ruby or something that is not compatible with your environment?
Availability Concerns - Can you lose a node and still function? Can it be upgraded without incurring an outage? The registry will grow to be a central part of your architecture and could be a single point of failure.

General Purpose Registries

These first three registries use strongly consistent protocols and are actually general purpose, consistent datastores. Although we’re looking at them as service registries, they are typically used for coordination services to aid in leader election or centralized locking with a distributed set of clients.

Zookeeper

Zookeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. It’s written in Java, is strongly consistent (CP) and uses the Zab protocol to coordinate changes across the ensemble (cluster).

Zookeeper is typically run with three, five or seven members in the ensemble. Clients use language specific bindings in order to access the ensemble. Access is typically embedded into the client applications and services.

Service registration is implemented with ephemeral nodes under a namespace. Ephemeral nodes only exist while the client is connected so typically a backend service registers itself, after startup, with its location information. If it fails or disconnects, the node disappears from the tree.

Service discovery is implemented by listing and watching the namespace for the service. Clients receive all the currently registered services as well as notifications when a service becomes unavailable or new ones register. Clients also need to handle any load balancing or failover themselves.

The Zookeeper API can be difficult to use properly and language bindings might have subtle differences that could cause problems. If you’re using a JVM based language, the Curator Service Discovery Extension might be of some use.

Since Zookeeper is a CP system, when a partition occurs, some of your system will not be able to register or find existing registrations even if they could function properly during the partition. Specifically, on any non-quorum side, reads and writes will return an error.

Doozer

Doozer is a consistent, distributed data store. It’s written in Go, is strongly consistent and uses Paxos to maintain consensus. The project has been around for a number of years but has stagnated for a while and now has close to 160 forks. Unfortunately, this makes it difficult to know what the actual state of the project is and whether is is suitable for production use.

Doozer is typically run with three, five or seven nodes in the cluster. Clients use language specific bindings to access the cluster and, similar to Zookeeper, integration is embedded into the client and services.

Service registration is not as straightforward as with Zookeeper because Doozer does not have any concept of ephemeral nodes. A service can register itself under a path but if the service becomes unavailable, it won’t be removed automatically.

There are a number of ways to address this issue. One option might be to add a timestamp and heartbeating mechanism to the registration process and handle expired entries during the discovery process or with another cleanup processes.

Service discovery is similar to Zookeeper in that you can list all the entries under a path and then wait for any changes to that path. If you use a timestamp and heartbeat during registration, you would ignore or delete any expired entries during discovery.

Like Zookeeper, Doozer is also a CP system and has the same consequences when a partition occurs.

Etcd

Etcd is a highly-available, key-value store for shared configuration and service discovery. Etcd was inspired by Zookeeper and Doozer. It’s written in Go, uses Raft for consensus and has a HTTP+JSON based API.

Etcd, similar to Doozer and Zookeeper, is usually run with three, five or seven nodes in the cluster. Clients use a language specific binding or implement one using an HTTP client.

Service registration relies on using a key TTL along with heartbeating from the service to ensure the key remains available. If a services fails to update the key’s TTL, Etcd will expire it. If a service becomes unavailable, clients will need to handle the connection failure and try another service instance.

Service discovery involves listing the keys under a directory and then waiting for changes on the directory. Since the API is HTTP based, the client application keeps a long-polling connection open with the Etcd cluster.

Since Etcd uses Raft, it should be a strongly-consistent system. Raft requires a leader to be elected and all client requests are handled by the leader. However, Etcd also seems to support reads from non-leaders using this undocumented consistent parameter which would improve availabilty in the read case. Writes would still need to be handled by the leader during a partition and could fail.

Single Purpose Registries

These next few registration services and approaches are specifically tailored to service registration and discovery. Most have come about from actual production use cases while others are interesting and different approaches to the problem. Whereas Zookeeper, Doozer and Etcd could also be used for distributed coordination, these solutions don’t have that capability.

Airbnb’s SmartStack

Airbnb’s SmartStack is a combination of two custom tools, Nerve and Synapse that leverage haproxy and Zookeeper to handle service registration and discovery. Both Nerve and Synapse are written in Ruby.

Nerve is a sidekick style process that runs as a separate process alongside the application service. Nerve is reponsible for registering services in Zookeeper. Applications expose a /health endpoint, for HTTP services, that Nerve continuously monitors. Provided the service is available, it will be registered in Zookeper.

The sidekick model eliminates the need for a service to interact with Zookeeper. It simply needs a monitoring endpoint in order to be registered. This makes it much easier to support services in different languages where robust Zookeeper binding might not exist. This also provides many of benefits of the Hollywood principle.

Synapse is also a sidekick style process that runs as a separate process alongside the service. Synapse is responsible for service discovery. It does this by querying Zookeeper for currently registered services and reconfigures a locally running haproxy instance. Any clients on the host that need to access another service always accesses the local haproxy instance which will route the request to an available service.

Synapse’s design simplifies service implementations in that they do not need to implement any client side load balancing or failover and they do not need to depend on Zookeepr or its language bindings.

Since SmartStack relies on Zookeeper, some registrations and discovery may fail during a partition. They point out that Zookeepr is their “Achilles heel” in this setup. Provided a service has been able to discover the other services, at least once, before a partition, it should still have a snapshot of the services after the partition and may be able to continue operating during the partition. This aspect improves the availability and reliability of the overall system.

Update: If you’re intested in a SmartStack style solution for docker containers, check out docker service discovery.

Netflix’s Eureka

Eureka is Netflix’s middle-tier, load balancing and discovery service. There is a server component as well as a smart-client that is used within application services. The server and client are written in Java which means the ideal use case would be for the services to also be imlemented in Java or another JVM compatible language.

The Eureka server is the registry for services. They recommend running one Eureka server in each availability zone in AWS to form a cluster. The servers replicate their state to each other through an asynchronous model which means each instance may have a slightly, different picture of all the services at any given time.

Service registration is handled by the client component. Services embed the client in their application code. At runtime, the client registers the service and periodically sends heartbeats to renew its leases.

Service discovery is handled by the smart-client as well. It retrieves the current registrations from the server and caches them locally. The client periodically refreshes its state and also handles load balancing and failovers.

Eureka was designed to be very resilient during failures. It favors availabilty over strong consistency and can operate under a number of different failure modes. If there is a partition within the cluster, Eureka transitions to a self-preservation state. It will allow services to be discovered and registered during a partition and when it heals, the members will merge their state again.

Bitly’s NSQ lookupd

NSQ is a realtime, distributed messaging platform. It’s written in Go and provides an HTTP based API. While it’s not a general purpose service registration and discovery tool, they have implemented a novel model of service discovery in their nsqlookupd agent in order for clients to find nsqd instances at runtime.

In an NSQ deployment, the nsqd instances are essentially the service. These are the message stores. nsqlookupd is the service registry. Clients connect directly to nsqd instances but since these may change at runtime, clients can discover the available instances by querying nsqlookupd instances.

For service registration, each nsqd instance periodically sends a heartbeat of its state to each nsqlookupd instance. Their state includes their address and any queues or topics they have.

For discovery, clients query each nsqlookupd instance and merge the results.

What is interesting about this model is that the nsqlookupd instances do not know about each other. It’s the responsibility of the clients to merge the state returned from each stand-alone nsqlookupd instance to determine the overal state. Because each nsqd instance heartbeats its state, each nsqlookupd eventually has the same information provided each nsqd instance can contact all available nsqlookupd instances.

All the previously discussed registry components all form a cluster and use strong or weakly consistent consensus protocols to maintain their state. The NSQ design is inherently weakly consistent but very tolerant to partitions.

Serf

Serf is a decentralized solution for service discovery and orchestration. It is also written in Go and is unique in that uses a gossip based protocol, SWIM for membership, failure detection and custom event propogation. SWIM was designed to address the unscalability of traditional heart-beating protocols.

Serf consists of a single binary that is installed on all hosts. It can be run as an agent, where it joins or creates a cluster, or as a client where it can discover the members in the cluster.

For service registration, a serf agent is run that joins an existing cluster. The agent is started with custom tags that can identify the hosts role, env, ip, ports, etc. Once joined to the cluster, other members will be able to see this host and it’s metadata.

For discovery, serf is run with the members command which returns the current members of the cluster. Using the members output, you can discover all the hosts for a service based on the tags their agent is running.

Serf is a relatively new project and is evolving quickly. It is the only project in this post that does not have a central registry architectural style which makes it unique. Since it uses a asynchronous, gossip based protocol, it is inherently weakly-consistent but more fault tolerant and available.

Spotify and DNS

Spotify described their use of DNS for service discovery in their post In praise of “boring” technology. Instead of using a newer, less mature technology they opted to build on top of DNS. Spotify views DNS as a “distributed, replicated database tailored for read-heavy loads.”

Spotify uses the relatively unknown SRV record which is intended for service discovery. SRV records can be thought of as a more generalized MX record. They allow you to define a service name, protocol, TTL, priority, weight, port and target host. Basically, everything a client would need to find all available services and load balance against them if necessary.

Service registration is complicated and fairly static in their setup since they manage all zone files under source control. Discovery uses a number of different DNS client librarires and custom tools. They also run DNS caches on their services to minimize load on the root DNS server.

They mention at the end of their post that this model has worked well for them but they are starting to outgrow it and are investigating Zookeeper to support both static and dynamic registration.

SkyDNS

SkyDNS is a relatively new project that is written in Go, uses RAFT for consensus and also provides a client API over HTTP and DNS. It has some similarities to Etcd and Spotify’s DNS model and actually uses the same RAFT implementation as Etcd, go-raft.

SkyDNS servers are clustered together, and using the RAFT protocol, elect a leader. The SkyDNS servers expose different endpoints for registration and discovery.

For service registration, services use an HTTP based API to create an entry with a TTL. Services must heartbeat their state periodically. SkyDNS also uses SRV records but extends them to also support service version, environment, and region.

For discovery, clients use DNS and retrieve the SRV records for the services they need to contact. Clients need to implement any load balancing or failover and will likely cache and refresh service location data periodically.

Unlike Spotify’s use of DNS, SkyDNS does support dynamic service registration and is able to do this without depending on another external service such as Zookeeper or Etcd.

If you are using docker, skydock might be worth checking out to integrate your containers with SkyDNS automatically.

Overall, this is an interesting mix of old (DNS) and new (Go, RAFT) technology and will be interesting to see how the project evolves.

Summary

We’ve looked at a number of general purpose, strongly consistent registries (Zookeeper, Doozer, Etcd) as well as many custom built, eventually consistent ones (SmartStack, Eureka, NSQ, Serf, Spotify’s DNS, SkyDNS).

Many use embedded client libraries (Eureka, NSQ, etc..) and some use separate sidekick processes (SmartStack, Serf).

Interestingly, of the dedicated solutions, all of them have adopted a design that prefers availability over consistency.

Name	Type	AP or CP	Language	Dependencies	Integration
Zookeeper	General	CP	Java	JVM	Client Binding
Doozer	General	CP	Go		Client Binding
Etcd	General	Mixed (1)	Go		Client Binding/HTTP
SmartStack	Dedicated	AP	Ruby	haproxy/Zookeeper	Sidekick (nerve/synapse)
Eureka	Dedicated	AP	Java	JVM	Java Client
NSQ (lookupd)	Dedicated	AP	Go		Client Binding
Serf	Dedicated	AP	Go		Local CLI
Spotify (DNS)	Dedicated	AP	N/A	Bind	DNS Library
SkyDNS	Dedicated	Mixed (2)	Go		HTTP/DNS Library

(1) If using the consistent parameter, inconsistent reads are possible

(2) If using a caching DNS client in front of SkyDNS, reads could be inconsistent

Fluentd vs Logstash

Tue, 19 Nov 2013 00:00:00 UTC

Fluentd and Logstash are two open-source projects that focus on the problem of centralized logging. Both projects address the collection and transport aspect of centralized logging using different approaches.

This post will walk through a sample deployment to see how each differs from the other. We’ll look at the dependencies, features, deployment architecture and potential issues. The point is not to figure out which one is the best, but rather to see which one would be a better fit for your environment.

The example setup we’ll walk through is collecting web server logs on multiple hosts and archiving them to S3:

This type of architecture would be suitable for archival or processing with Hive or Pig.

Another common architecture is storing logs in ElasticSearch to make them searchable with Kibana or Graylog2. Setting that up is somewhat independent of using Logstash or Fluentd so I’ve left that out to keep things simple.

Installation Requirements

Logstash

Logstash is a JRuby based application which requires the JVM to run. Since it runs on the JVM, it can run anywhere the JVM does, which is usually means Linux, Mac OSX, and Windows. The package is shipped as single executable jar file which makes it very easy to install.

One of the downsides of depending on the JVM is that it’s memory footprint can be higher than you would want for transporting logs. Fortunately, Lumberjack can be run on individual hosts to collect and ship logs and Logstash can be run on the centralized log hosts.

Lumberjack is a Go based project with a much smaller memory and CPU footprint. Deployment is still straightforward as Logstash and is basically installing a single binary. The project provides deb and rpm packages to make it easier to deploy. An SSL certificates is required to setup authentication between Lumberjack and Logstash which is a little more complicated, but a nice benefit that encrypted transport is the default.

Fluentd

Fluentd is a CRuby application which requires Ruby 1.9.2 or later. There is an open-source version, fluentd, as well as a commercial version, td-agent. Fluentd runs on Linux and Mac OSX, but does not run on Windows currently.

For larger installs, they recommend using jemalloc to avoid memory fragmentation. This is included in the deb and rpm packages but needs to be installed manually if using the open-source version.

If you use the open-source version, you’ll need to install Fluentd from source or via gem install. Since Fluentd is primarily developed by a commercial company, their deb and rpm packages are configured to send data to their hosted centralized logging platform.

Apart from Ruby, they also recommend running ntpd which you should be running anyways.

Feature Comparison

Logstash supports a number of inputs, codecs, filters and outputs. Inputs are sources of data. Codecs essentially convert an incoming format into an internal Logstash representation as well as convert back out to an output format. These are usually used if the incoming message is not just a single line of text. Filters are processing actions on events and allow you to modify events or drop events as they are processed. Finally, outputs are destinations where events can be routed.

Fluentd is similar in that it has inputs and outputs and a matching mechanism to route log data between destinations. Internally, log messages are converted to JSON which provides structure to an unstructered log message. Messages can be tagged and then routed to different outputs.

Both projects have very similar capabilities and highlighting the difference between them from a feature standpoint is difficult. They both have plugin models that allow you to extend their functionality if needed. They also have rich repository of plugins already available.

Probably the most significant difference between Fluentd and Logstash is their design focus. Logstash emphasizes flexibility and interoperability whereas Fluentd prioritizes simplicity and robustness. This does not mean that Logstash is not robust or Fluentd is not flexible, rather each has prioritized features differently.

Fluentd has fewer inputs than Logstash, out of the box, but many of the inputs and outputs have built-in support for buffering, load-balancing, timeouts and retries. These types of features are necessary for ensuring data is reliably delivered.

For example, the out_forward plugin used to transfer logs from one fluentd instance to another has many robustness options that can be configured to ensure messages are delivered reliably.

Architecture comparison

From a deployment architecture standpoint, both frameworks are very similar. With Logstash, each web server would be configured to run Lumberjack and tail the web server logs. Lumberjack would forward the logs to a server running Logstash with a Lumberjack input. The Logstash server would also have an output configured using the S3 output. Since Lumberjack requires SSL certs, the log transfers would be encrypted from the web server to the log server.

With fluentd, each web server would run fluentd and tail the web server logs and forward them to another server running fluentd as well. This server would be configured to receive logs and write them to S3 using the S3 plugin. Fluentd does not support encryption so logs would be transferred unencrypted.

Update: @repeatedly pointed me to the fluent-plugin-secure-forward that some companies are using for encrypted transport.

Improving Availability

One central log server is a single point of failure. What happens if we wanted to have more than one central log server?

Lumberjack can be configured to use multiple servers but will only send logs to one until that one fails. If that happens, previously collected log data won’t be accessible until that host is resurrected. Essentially, it supports a master with hot-standby servers.

Fluentd on the other hand can forward two copies of the logs to each server if needed, load-balance between multiple hosts or have a master with a hot-standy in case of failure. There are lot of options for not only improving availabilty but also scalability if your log volume increases substantially. Keep in mind, that if you forward multiple copies, this could create duplicate logs in S3 which might need to be handled when you analyze them.

Potential Issues

The Logstash docs suggests using Redis as the receiving output if you run Logstash (not Lumberjack) on each host. This setup is based on Redis Lists and/or Pub/Sub which can lose messages if the receiver dies after removing the message from Redis and before it has had a chance to forward it along. Additionally, Redis would need to be configured with AOF to minimize the chance of lost messages if Redis were to fail.

There is a document describing the life of an event that discusses some of the failure modes and how Logstash addresses them. One important point is that outputs are responsible for retrying events in the case of errors. There are also internal, ephemeral queues within Logstash that can hold up to 20 events. Depending on the failure, there is a window for messages to be lost.

If you absolutely cannot risk losing messages, make sure you investigate all the failure modes and whether the plugins you are using are implemented correctly to handle them.

Update: LOGSTASH-1631 is a bug that demonstrates one way messages can be lost. It appears the internal messaging is going to be replaced with a more reliable implementation in the future.

Conclusion

Both Logstash and Fluentd are viable centralized logging frameworks that can transfer logs from multiple hosts to a central location. Logstash is incredibly flexible with many input and output plugins whereas fluentd provides fewer input and output sources but provides multiple options for reliably and robust transport.

Realtime Web Server Log Metrics

Mon, 22 Jul 2013 00:00:00 UTC

This is a sample config that uses nxlog to tail web access logs in Combined Log Format, pull out the status code and bytes sent and send them to statsd so they can be graphed using Graphite.

It’s a simple way to see if your web server is returning errors over time or how much data it’s sending. The same concept could be used for other log files.

Logster lets you do similar things but custom parsing is accomplished by writing Python plugins which can be a little more complicated than using configuration files.

nxlog works by defining inputs, processors, outputs and routes that tie the inputs to processers and finally to outputs.

For this example, I use the imfile module which will tail one or more files send each line along for processing. nxlog will remember the last line read by default so restarts or connectivity issues are handled gracefully. After each line is read, the line is passed to a _Exec statement that parses each field into variables.

The parsing could be done in the processor but because of how the routes are setup, I chose to do it during input so it only happens once.

# Tail access log in Combined Log Format and parse out the fields.
<Input in_nginx>
    Module  im_file
    File	"/var/log/nginx/access.log"
    Exec    if $raw_event =~ /^(\S+) (\S+) (\S+) \[([^\]]+)\] \"(\S+) (.+) HTTP.\d\.\d\" (\d+) (\d+) \"([^\"]+)\" \"([^\"]+)\"/\
                { \
                  $Hostname = $1; \
                  if $3 != '-' $AccountName = $3; \
                  $EventTime = parsedate($4); \
                  $HTTPMethod = $5; \
                  $HTTPURL = $6; \
                  $HTTPResponseStatus = $7; \
                  $HTTPBytesSent = $8; \
                  $HTTPReferer = $9; \
                  $HTTPUserAgent = $10; \
                }
</Input>

There are two processors configured http_status and http_bytes. Each one checks to make sure the required data has been parsed and then rewrites the event into a statsd protocol metric. Both processors create a counter metric that statsd will sum up and roll-over at every interval.

# Rewrite the log message to a statsd counter event using the HTTP status code.
<Processor http_status>
    Module      pm_null
    Exec        if defined($HTTPResponseStatus) { \
		          $raw_event = "http.status." + $HTTPResponseStatus + ":1|c"; \
                } else { \
                  drop(); \
                }
</Processor>

# Rewrite the log message to a statsd counter event using the bytes sent.
<Processor http_bytes>
    Module      pm_null
    Exec        if defined($HTTPBytesSent) { \
		          $raw_event = "http.bytes." + $HTTPBytes + ":" + $HTTPBytesSent + "|c"; \
                } else { \
                  drop(); \
                }
</Processor>

The output is a om_udp that forwards the re-written log event to a statsd instance.

# Statsd uses UDP
<Output out_statsd>
    Module	om_udp
    Host	127.0.0.1
    Port	8125
</Output>

Finally, two routes are created that originate from the same input file but are sent to separate processors and back to the same output. Since processors are normally chained together and I’m re-writing the event, chaining them together doesn’t work. Instead I need a new copy of the log message for each processor.

# Route nginx access log through status processor and out to statsd
<Route web_status_statsd>
    Path        in_nginx => http_status => out_statsd
</Route>

# Since we re-wrote the log event, define a new route for bytes sent
<Route web_bytes_statsd>
    Path        in_nginx => http_bytes => out_statsd
</Route>

And the whole thing put together:

# Tail access log in Combined Log Format and parse out the fields.
<Input in_nginx>
    Module  im_file
    File	"/var/log/nginx/access.log"
    Exec    if $raw_event =~ /^(\S+) (\S+) (\S+) \[([^\]]+)\] \"(\S+) (.+) HTTP.\d\.\d\" (\d+) (\d+) \"([^\"]+)\" \"([^\"]+)\"/\
                { \
                  $Hostname = $1; \
                  if $3 != '-' $AccountName = $3; \
                  $EventTime = parsedate($4); \
                  $HTTPMethod = $5; \
                  $HTTPURL = $6; \
                  $HTTPResponseStatus = $7; \
                  $HTTPBytesSent = $8; \
                  $HTTPReferer = $9; \
                  $HTTPUserAgent = $10; \
                }
</Input>

# Rewrite the log message to a statsd counter event using the HTTP status code.
<Processor http_status>
    Module      pm_null
    Exec        if defined($HTTPResponseStatus) { \
		          $raw_event = "http.status." + $HTTPResponseStatus + ":1|c"; \
                } else { \
                  drop(); \
                }
</Processor>

# Rewrite the log message to a statsd counter event using the bytes sent.
<Processor http_bytes>
    Module      pm_null
    Exec        if defined($HTTPBytesSent) { \
		          $raw_event = "http.bytes." + $HTTPBytes + ":" + $HTTPBytesSent + "|c"; \
                } else { \
                  drop(); \
                }
</Processor>

# Statsd uses UDP
<Output out_statsd>
    Module	om_udp
    Host	127.0.0.1
    Port	8125
</Output>

# Route nginx access log through status processor and out to statsd
<Route web_status_statsd>
    Path        in_nginx => http_status => out_statsd
</Route>

# Since we re-wrote the log event, define a new route for bytes sent
<Route web_bytes_statsd>
    Path        in_nginx => http_bytes => out_statsd
</Route>

Centralized Logging Architecture

Tue, 16 Jul 2013 00:00:00 UTC

In Centralized Logging, I covered a few tools that help with the problem of centralized logging. Many of these tools address only a portion of the problem which means you need to use several of them together to build a robust solution.

The main aspects you will need to address are: collection, transport, storage, and analysis. In some special cases, you may also want to have an alerting capability as well.

Collection

Applications create logs in different ways, some log through syslog, others log directly to files. If you consider a typical web application running on a linux hosts, there will be a dozen or more log files in /var/log as well as a few application specific logs in home directories or other locations.

If you are supporting a web based application and your developers or operations staff need access to log data quickly in order to troubleshoot live issues, you need a solution that is able to monitor changes to log files in near real-time. If you are using a file replication based approach where files are replicated to a central server on a fixed schedule, then you can only inspect logs as frequently as the replication runs. A one minute rsync cron job might not be fast enough when your site is down and you are waiting for the relevant log data to be replicated.

On the other hand, if you need to analyze log data offline for calculating metrics or other batch related work, a file replication strategy might be a good fit.

Transport

Log data can accumulate quickly on multiple hosts. Transporting it reliably and quickly to your centralized location may need additional tooling in order to effectively transmit it and ensure data is not lost.

Frameworks such as Scribe, Flume, Heka, Logstash, Chukwa, fluentd, nsq and Kafka are designed for transporting large volumes of data from one host to another reliably. Although each of these frameworks addresses the transport problem, they do so quite differently.

For example, Scribe, nsq and Kafka, require clients to log data via their API. Typically, application code is written to log directly to these sources which allows them to reduce latency and improve reliability. If you want to centralize typical log file data, you would need something to tail and stream the logs via their respective APIs. If you control the app that is logging the data you want to collect, these can be much more efficient.

Logstash, Heka, fluentd and Flume provide a number of input sources but also support natively tailing files and transporting them reliably. These are a better fit for more general log collection.

While rsyslog and Syslog-ng are typically thought of as the defacto log collector, not all applications use syslog.

Storage

Now that your log data is being transfered, it needs a destination. Your centralized storage system needs to be able to handle the growth in data over time. Each day will add a certain amount of storage that is relative to the number of hosts and processes that are generating log data.

How you store things depends on a few things:

How long should it be stored - If the logs are for long-term, archival purposes and do not require immediate analysis, S3, AWS Glacier, or tape backup might be a suitable option since they provide relatively low cost for large volumes of data. If you only need a few days or months worth of logs, storing them on some form distributed storage systems such as HDFS, Cassandara, MongoDB or ElasticSearch also works well. If you only need a few hours worth of retention for real-time analysis, Redis might work as well.
Your environments data volume. - A days worth of logs for Google is much different than a days worth of logs for ACME Fishing Supplies. The storage system you chose should allow you to scale-out horizontally if your data volume will be large.
How will you need to access the logs - Some storage is not suitable for real-time or even batch analysis. AWS Glacier or tape backup can take hours to load a file. These don’t work if you need log access for production troubleshooting. If you plan to do more interactive data analysis, storing log data in ElasticSearch or HDFS may allow you work with the raw data more effectively. Some log data is so large that it can only be analyzed in more batch oriented frameworks. The defacto standard is this case is Apache Hadoop along with HDFS.

Analysis

Once your logs are stored on a centralized storage platform, you need a way to analyze them. The most common approach is a batch oriented process that runs periodically. If you are storing log data in HDFS, Hive or Pig might help analyzing the data easier than writing native MapReduce jobs.

If you need a UI for analysis, you can store parsed log data in ElasticSearch and use a front-end such as Kibana or Graylog2 to query and inspect the data. The log parsing can be handled by Logstash, Heka or applications logging with JSON directly. This approach allows more real-time, interactive access to the data but is not really suited for a mass batch processing.

Alerting

The last component that is sometimes nice to have is the ability to alert on log patterns or calculated metrics based on log data. Two common uses for this are error reporting and monitoring.

Most log data is not interesting but errors almost always indicate a problem. It’s much more effective to have the logging system email or notify respective parties when errors ocurr instead of having someone repeatedly watch for the events. There are several services that solely provide application error logging such as Sentry or HoneyBadger. These can also aggregate repetitve exceptions which can give you and idea of how frequently an error is occuring.

Another use case is monitoring. For example, you may have hundreds of web servers and want to know if they start returning 500 status codes. If you can parse your web log files and record a metric on the status code, you can then trigger alerts when that metric crosses a certain threshold. Riemann is designed for detecting scenarios just like this.

Hopefully this helps provide a basic model for designing a centralized logging solution for your environment.

Optimizing MongoDB Indexes

Wed, 08 Feb 2012 00:00:00 UTC

Good indexes are an important part running a well performing application on MongoDB. MongoDB performs best when it can keep your indexes in RAM. Reducing the size of your indexes also leads to faster queries and the ability to manage more data with less RAM.

These are a few tips to reduce the size of your MongoDB indexes:

1) Determining Indexes Sizes

The first thing you should do is to understand the size of your indexes. You want to know the sizes before you make changes to confirm that the changes have actually reduced the size. Ideally, you are graphing your indexes over time with your monitoring tools.

Using the mongo shell you can run db.stats() to get database indexes stats:

> db.stats()
{
	"db" : "examples1",
	"collections" : 6,
	"objects" : 403787,
	"avgObjSize" : 121.9966467469235,
	"dataSize" : 49260660,
	"storageSize" : 66695168,
	"numExtents" : 20,
	"indexes" : 9,
	"indexSize" : 48524560,
	"fileSize" : 520093696,
	"nsSizeMB" : 16,
	"ok" : 1
}

indexes - The number of indexes in examples1 DB
indexSize - The size of the indexes in example1 DB

Since each collection has indexes, you can run db.collection.stats() to see them:

> db.address.stats()
{
	"ns" : "examples1.address",
	"count" : 3,
	"size" : 276,
	"avgObjSize" : 92,
	"storageSize" : 8192,
	"numExtents" : 1,
	"nindexes" : 2,
	"lastExtentSize" : 8192,
	"paddingFactor" : 1,
	"flags" : 1,
	"totalIndexSize" : 16352,
	"indexSizes" : {
		"_id_" : 8176,
		"_types_1" : 8176
	},
	"ok" : 1
}

totalIndexSize - The size of all indexes in the collection
indexSizes - A dictionary of index name and size

NOTE: all sizes returned by these commands are in bytes.

These commands are useful but they are tedious to use manually. To report on indexes stats, I wrote a utility, index-stats.py, that can be found in the mongodb-tools project on Github that makes things easier.

(virtualenv) mongodb-tools$ ./index-stats.py
Checking DB: examples2.system.indexes
Checking DB: examples2.things
Checking DB: examples1.system.indexes
Checking DB: examples1.address
Checking DB: examples1.typeless_address
Checking DB: examples1.user
Checking DB: examples1.typeless_user

Index Overview
+----------------------------+------------------------+--------+------------+
|         Collection         |         Index          | % Size | Index Size |
+----------------------------+------------------------+--------+------------+
| examples1.address          | _id_                   |   0.0% |      7.98K |
| examples1.address          | _types_1               |   0.0% |      7.98K |
| examples1.typeless_address | _id_                   |   0.0% |      7.98K |
| examples1.typeless_user    | _id_                   |  10.1% |      6.21M |
| examples1.typeless_user    | address_id_1           |  10.1% |      6.21M |
| examples1.typeless_user    | typeless_address_ref_1 |   5.9% |      3.62M |
| examples1.user             | _id_                   |  10.1% |      6.21M |
| examples1.user             | _types_1               |   6.9% |      4.24M |
| examples1.user             | _types_1_address_id_1  |  12.2% |      7.51M |
| examples1.user             | _types_1_address_ref_1 |  26.2% |     16.09M |
| examples2.things           | _id_                   |  10.1% |      6.21M |
| examples2.things           | _types_1               |   8.4% |      5.13M |
+----------------------------+------------------------+--------+------------+

Top 5 Largest Indexes
+-------------------------+------------------------+--------+------------+
|        Collection       |         Index          | % Size | Index Size |
+-------------------------+------------------------+--------+------------+
| examples1.user          | _types_1_address_ref_1 |  26.2% |     16.09M |
| examples1.user          | _types_1_address_id_1  |  12.2% |      7.51M |
| examples1.typeless_user | _id_                   |  10.1% |      6.21M |
| examples2.things        | _types_1               |   8.4% |      5.13M |
| examples1.user          | _types_1               |   6.9% |      4.24M |
+-------------------------+------------------------+--------+------------+

Total Documents: 600016
Total Data Size: 74.77M
Total Index Size: 61.43M
RAM Headroom: 2.84G
Available RAM Headroom: 1.04G

The output shows the total index size, each index size, and their relative sizes to each other. In addition, the Top 5 Largest indexes are reported across all your collections. This makes it easy to determine your largest indexes and the ones where reducing their size will provide most benefit.

RAM Headroom is your physical memory - index size. A positive value means you have RAM available for indexes to fit in memory.
Available RAM Headroom is free memory - index size. Since other processes consume memory on this system, I don’t have the total RAM Headroom available.

The RAM Headroom stat idea comes from the MongoDB monitoring service I use, ServerDensity.

From this output, I would focus on the examples1.user collection and the types_1_address_ref_1 and types_1_address_id_1 indexes first.

2) Remove Redundant Indexes

If you have been releasing code changes over a period of time, you’ll likely end up with redundant indexes. MongoDB can use the prefix of a compound index if all the component parts are not available. In the previous output,

| examples1.user          | _types_1               |   6.9% |      4.24M |

is redundant with

| examples1.user          | _types_1_address_ref_1 |  26.2% |     16.09M |
| examples1.user          | _types_1_address_id_1  |  12.2% |      7.51M |

Because _types_1 is the prefix to these two indexes. Dropping it would save 4.2M on the total index size and be one less index to update when user documents change.

To make it easier to find these indexes, you can run redundant-indexes.py from mongodb-tools:

(virtualenv)mongodb-tools$ ./redundant-indexes.py
Checking DB: examples2
Checking DB: examples1
Index examples1.user[_types_1] may be redundant with examples1.user[_types_1_address_ref_1]
Index examples1.user[_types_1] may be redundant with examples1.user[_types_1_address_id_1]
Checking DB: local

3) Compact Command

If you are running MongoDB 2.0+, you can run the compact command to defragment your collections and rebuild the indexes. The compact command locks the database so make sure you know where you are running it beforehand. If you are running with replica sets, the easiest thing to do is to run it on your secondaries, one at a time, fail-over the primary to new secondary and run compact on the old primary.

4) MongoDB 2.0 Index Improvements

If you are not running MongoDB 2.0 or later, upgrading and rebuilding your indexes should provide about a 25% savings.

See Index Performance Enhancements

5) Check Index Criteria

Another thing to check is your index criteria. You want the values that are indexed to be small and as selective as possible. Indexing values that do not help MongoDB find your data faster slow queries down and increase the index size. If you are using a mapping framework for your application, and it support defining indexes in the code, you should check to see what it’s actually indexing. For example MongoEngine for Python uses a “_types” field to identify subclasses in the same collection. This can add a lot of space and may not add to the selectivity of you indexes.

In my test data, my largest index is:

| examples1.user             | _types_1_address_ref_1 |  26.2% |     16.09M |

Looking at the data for it:

> db.user.findOne()
{
	"_id" : ObjectId("4f2ef95c89a40a11c5000002"),
	"_types" : [
		"User"
	],
	"address_id" : ObjectId("4f2ef95c89a40a11c5000000"),
	"address_ref" : {
		"$ref" : "address",
		"$id" : ObjectId("4f2ef95c89a40a11c5000000")
	},
	"_cls" : "User"
}

You can see that _types is an array with a value of User, the class name. Since I don’t have any subclasses of User in my code, indexing this value does not help the index selectivity. Another way of thinking about this is that each value in the index is going to have “User” as a prefix which adds a few extra bytes for value and does not increase the selectivity of the index.

Removing it in the code with:

class User(Document):
    meta {'index_types':False}

Changes the index to:

| examples1.user             | address_ref_1          |  16.8% |     12.39M |

About a 23% savings.

Digging in further, address_ref_1 is a ReferenceProperty to an Address object. The data above shows that it is a dictionary that contains the id of the reference field as well as the collection that it points to. If we change this ReferenceProperty to an ObjectIdProperty, which is what address_id, is, you can get additional savings:

| examples1.user             | address_id_1           |   9.5% |      6.21M |
| examples1.user             | address_ref_1          |  20.9% |     13.70M |

About a 53% savings. This changes the index value from being stored as a serialized dictionary to just and ObjectId which is likely highly optimized with MongoDB. Changing the property type does require code changes though and you also lose the automatic de-referencing capability provided by ReferenceProperties. It can produce significant savings though.

In total, we’ve reduced the original index by 61% by adjusting some index criteria and making some small code changes.

6) Delete/Move Old Data

In most applications, some data is accessed more frequently than others. If you have old data that won’t be accessed by your users, you may be able to purge it, move it to another un-indexed collection, or archive it somewhere outside of the DB. Ideally, you database contains and is indexing the working set of available data.

There are some other good optimization ideas that can be found here:

How do you tune your indexes?

Centralized Logging

Tue, 03 Jan 2012 00:00:00 UTC

Logs are a critical part of any system, they give you insight into what a system is doing as well what happened. Virtually every process running on a system generates logs in some form or another. Usually, these logs are written to files on local disks. When your system grows to multiple hosts, managing the logs and accessing them can get complicated. Searching for a particular error across hundreds of log files on hundreds of servers is difficult without good tools. A common approach to this problem is to setup a centralized logging solution so that multiple logs can be aggregated in a central location.

So what are your options?

File Replication

A simple approach is to setup file replication of your logs to a central server on a cron schedule. Usually rsync and cron are used since they are simple and straightforward to setup. This solution can work for a while but it doesn’t provide timely access to log data. It also doesn’t aggregate the logs and only co-locates them.

Syslog

Another option that you probably already have installed is syslog. Most people use rsyslog or syslog-ng which are two syslog implementations. These daemons allow processes to send log messages to them and the syslog configuration determines how the are stored. In a centralized logging setup, a central syslog daemon is setup on your network and the client logging dameons are setup to forward messages to the central daemon. A good write-up of this kind of setup can be found at: Centralized Logging Use Rsyslog

Syslog is great because just about everything uses it and you likely already have it installed on your system. With a central syslog server, you will likely need to figure out how to scale the server and make it highly-available.

syslog-ng
rsyslog

Distributed Log Collectors

A new class of solutions that have come about have been designed for high-volume and high-throughput log and event collection. Most of these solutions are more general purpose event streaming and processing systems and logging is just one use case that can be solved using them. All of these have their specific features and differences but their architectures are fairly similar. They generally consist of logging clients and/or agents on each specific host. The agents forward logs to a cluster of collectors which in turn forward the messages to a scalable storage tier. The idea is that the collection tier is horizontally scalable to grow with the increase number of logging hosts and messages. Similarly, the storage tier is also intended to scale horizontally to grow with increased volume. This is gross simplification of all of these tools but they are a step beyond traditional syslog options.

Scribe - Scribe is scalable and reliable log aggregation server used and released by Facebook as open source. Scribe is written in C++ and uses Thrift for the protocol encoding. Since it uses thrift, virtually any language can work with it.
Flume - Flume is an Apache project for collecting, aggregating, and moving large amounts of log data. It stores all this data on HDFS.
logstash - logstash lets you ship, parse and index logs from any source. It works by defining inputs (files, syslog, etc.), filters (grep, split, multiline, etc..) and outputs (elasticsearch, mongodb, etc..). It also provides a UI for accessing and searching your logs. See Getting Started
Chukwa - Chukwa is another Apache project that collects logs onto HDFS.
fluentd - Fluentd is similar to logstash in that there are inputs and outputs for a large variety of sources and destination. Some of it’s design tenets are easy installation and small footprint. It doesn’t provide any storage tier itself but allows you to easily configure where your logs should be collected.
kafka - Kafka was developed at LinkedIn for their activity stream processing and is now an Apache incubator project. Although Kafka could be used for log collection this is not it’s primary use case. Setup requires Zookeeper to manage the cluster state.
Graylog2 - Graylog2 provides a UI for searching and analyzing logs. Logs are stored in MongoDB and/or elasticsearch. Graylog2 also provides the GELF logging format to overcome some issues with syslog message: 1024 byte limit and unstructured log messages. If you are logging long stacktraces, you may want to look into GELF.
splunk - Splunk is commercial product that has been around for several years. It provides a whole host of features for not only collecting logs but also analyzing and viewing them.

Update: I wrote a post comparing Fluentd vs Logstash.

Hosted Logging Services

There are also several hosted “logging as a service” providers as well. The benefit of them is that you only need to configure your syslog forwarders or agents and they manage the collection, storage and access to the logs. All of the infrastructure that you have to setup and maintain is handled by them, freeing you up to focus on your application. Each service provide a simple setup (usuallysyslog forwarding based), an API and a UI to support search and analysis.

I go into more detail how all of these fit together in Centralized Logging Architecture.

Octopress Blogging System

Thu, 15 Dec 2011 00:00:00 UTC

After several years of maintaining a Wordpress blog, I’ve decided to switch to Octopress. Wordpress worked well for me at first but it seemed to have more functionality than I really needed or wanted. The breaking point happened last night when I tried to upgrade it and the templates I was using no longer worked. Digging into the code was going to be more effort than it was worth. So now I’ve moved on to something simpler.

Octopress uses a different approach then Wordpress for generating content. Octopress generates a static site whereas Wordpress needs MySQL and PHP. I like the simplicty of Octopress and it also makes it easy version control everything w/ git.

on Jason Wilder's Blog

A Simple Way To Dockerize Applications

The Problems

Configuration

Logging

Using Dockerize

An Example

Conclusion

Projects

Squashing Docker Images

Other Solutions

Using Small Base Images

Publishing Tools

Official Support

docker-squash

Example

Starting Image

Clean Up

Squash The Image

Conclusion

Docker Service Discovery Using Etcd and Haproxy

How It Works

Service Registry - Etcd

Service Registration - docker-register

Service Discovery - docker-discover

AWS Demo

Etcd Host

Backend Hosts

Client Host

Wrapping Up

Automated Nginx Reverse Proxy for Docker

Why Use A Reverse Proxy With Docker

Generating Reverse Proxy Configs

Nginx Reverse Proxy for Docker

Try It Out

Conclusion

Docker Log Management Using Fluentd

Docker Logging Options

Using Fluentd With Docker

Conclusion

Open-Source Service Discovery

The Problem

General Purpose Registries

Zookeeper

Doozer

Etcd

Single Purpose Registries

Airbnb’s SmartStack

Netflix’s Eureka

Bitly’s NSQ lookupd

Serf

Spotify and DNS

SkyDNS

Summary

Fluentd vs Logstash

Installation Requirements

Logstash

Fluentd

Feature Comparison

Architecture comparison

Improving Availability

Potential Issues

Conclusion

Realtime Web Server Log Metrics

Centralized Logging Architecture

Collection

Transport

Storage

Analysis

Alerting

Optimizing MongoDB Indexes

1) Determining Indexes Sizes

2) Remove Redundant Indexes

3) Compact Command

4) MongoDB 2.0 Index Improvements

5) Check Index Criteria

6) Delete/Move Old Data

Centralized Logging

File Replication

Syslog