Yet Another DevOps Blog

Introducing swarmt: local swarm clusters manager

2017-05-24T00:00:00+00:00

Swarmt is a small project I’ve started while dealing with the many swarm clusters
I’ve deployed on my laptop.

Moments when doing docker-machine ls gives that kind of output:

NAME         ACTIVE   DRIVER          STATE     URL   SWARM   DOCKER    ERRORS
anchore-vm   -        virtualbox      Stopped                 Unknown   
dagda-vm     -        virtualbox      Stopped                 Unknown   
swarm1m1     -        virtualbox      Stopped                 Unknown   
swarm1m2     -        virtualbox      Stopped                 Unknown   
swarm1w1     -        virtualbox      Stopped                 Unknown   
swarm1w2     -        virtualbox      Stopped                 Unknown   
swarm1w3     -        virtualbox      Stopped                 Unknown   
rancherm1    -        virtualbox      Stopped                 Unknown   
rancherm2    -        virtualbox      Stopped                 Unknown   
rancherw1    -        virtualbox      Stopped                 Unknown   
rancherw2    -        virtualbox      Stopped                 Unknown   
rancherw3    -        virtualbox      Stopped                 Unknown   
pxcm1        -        virtualbox      Stopped                 Unknown   
pxcm2        -        virtualbox      Stopped                 Unknown   
pxcw1        -        virtualbox      Stopped                 Unknown   
pxcw2        -        virtualbox      Stopped                 Unknown   
pxcw3        -        virtualbox      Stopped                 Unknown   
........

Until now I was using that kind of not so fancy shell tips:
for i in swarm1m1 swarm1m2 swarm1w1 swarm1w2 swarm1w3; do docker-machine $i stop;done

Here comes swarmt.sh to my rescue !

How does swarmt work

Basically swarmt wraps docker-machine with a little bit of [[IMG MAGIC]]

It looks for a configuration file which by default should be named swarmt.conf
you can use different configuration files (see examples below)

The file should contain these parameters:

project=[your project name]
smanager=[number of swarm managers you want]
sworker=[number of swarm workers you want]
mdriver=[docker-machine driver you want to use] (Virtualbox only , digital-ocean should follow)
mimage=[name of the image you want use]
dotoken=[digital-ocean token]
stackfile=[name of the stack file you want swarmt to load in the end] (optional)

Different options are available in swarmt:
init : create swarm nodes and bootstrap the cluster as defined in the configuration file
start: start an existing swarm cluster
stop : stop all swarm nodes
rm : stop and delete all swarm nodes
list : list and give status of all swarm nodes

Examples

Single swarm cluster

Let’s start with a very simple swarm cluster (i.e 1 manager and 2 workers)
for a mysql galera cluster.

Edit swarmt.conf as below:

project=swarmG
smanager=1
sworker=2
mdriver=virtualbox
mimage=https://releases.rancher.com/os/latest/rancheros.iso
dotoken=
stackfile=swarmG.yml

In this example, swarmG.yml doesn’t exist so swarmt won’t deploy any container

Time to fire up our swarm cluster:
./swarmt.sh init <=== yes ! that simple!

Few minutes later, you should have this message:
swarmG swarm cluster is up and running

Let’s see if it really works !

eval $(docker-machine env swarmGm1)
docker node ls

Should output:

swarmGm1     *        virtualbox      Running   tcp://192.168.99.100:2376           v17.05.0-ce   
swarmGw1     -        virtualbox      Running   tcp://192.168.99.101:2376           v17.05.0-ce   
swarmGw2     -        virtualbox      Running   tcp://192.168.99.102:2376           v17.05.0-ce

Awesome !

multiple swarm cluster

Now imagine, you want swarmtto manage, let’s say 2 swarm cluster configuration.

First, create myproject.conf :

myproject.conf:
project=myproject
smanager=1
sworker=2
mdriver=virtualbox
mimage=https://releases.rancher.com/os/latest/rancheros.iso
dotoken=
stackfile=

mycoolproject.conf:

mycoolproject.conf:
project=mycoolproject
smanager=2
sworker=3
mdriver=virtualbox
mimage=https://releases.rancher.com/os/latest/rancheros.iso
dotoken=
stackfile=

Let’s start clusters for these 2 projects:

./swarmt.sh -c myproject.conf init
./swarmt.sh -c mycoolproject.conf init

you should see:

myproject swarm cluster is up and running
mycoolproject swarm cluster is up and running

How about listing all swarm nodes?
./swarmt.sh list <== no need to specify any configuration file

Should output:

myproject swarm nodes:
myprojectm1     *        virtualbox      Running   tcp://192.168.99.100:2376           v17.05.0-ce   
myprojectw1     -        virtualbox      Running   tcp://192.168.99.101:2376           v17.05.0-ce   
myprojectw2     -        virtualbox      Running   tcp://192.168.99.102:2376           v17.05.0-ce  


mycoolproject swarm nodes:
mycoolprojectm1     *        virtualbox      Running   tcp://192.168.99.100:2376           v17.05.0-ce   
mycoolprojectm2     -        virtualbox      Running   tcp://192.168.99.100:2376           v17.05.0-ce   
mycoolprojectw1     -        virtualbox      Running   tcp://192.168.99.101:2376           v17.05.0-ce   
mycoolprojectw2     -        virtualbox      Running   tcp://192.168.99.102:2376           v17.05.0-ce 
mycoolprojectw3     -        virtualbox      Running   tcp://192.168.99.102:2376           v17.05.0-ce

Now let’s stop mycoolproject swarm nodes:
./swarmt.sh -c mycoolproject.conf stop

Should output:

Stopping "mycoolprojectm1"...
Machine "mycoolprojectm1" was stopped.
Stopping "mycoolprojectm2"...
Machine "mycoolprojectm2" was stopped.
Stopping "mycoolprojectw1"...
Machine "mycoolprojectw1" was stopped.
Stopping "mycoolprojectw2"...
Machine "mycoolprojectw2" was stopped.
Stopping "mycoolprojectw3"...
Machine "mycoolprojectw3" was stopped.


===================================
 mycoolproject swarm cluster is halted 
===================================

Need to start again your cluster ? here you are:
./swarmt.sh -c mycoolproject.conf start

Should output:

.......
 
mycoolprojectw3 swarm node is starting :
Starting "mycoolprojectw3"...
(mycoolprojectw3) Check network to re-create if needed...
(mycoolprojectw3) Waiting for an IP...
Machine "mycoolprojectw3" was started.
Waiting for SSH to be available...
Detecting the provisioner...
Started machines may have new IP addresses. You may need to re-run the `docker-machine env` command.
 
 
===================================
 swarm1 swarm cluster is ready 
===================================

As you can see swarmt can be pretty useful when dealing with multiple clusters setup.
Checkout my github repository for more about swarmt.
As always PR / Feature request / Issues are welcome :)

Have fun !

How to make your Docker images secrets enabled

2017-05-24T00:00:00+00:00

Integrated into Docker swarm, Docker secrets gives a complete and secure way to manage sensitive data shared with your containers.
No more environment variables or worth plain text files with username and password shared or baked into your containers.

Everything in Docker secrets have been built based on 2 concepts:

security
simplicity

Here’s a quick review of how it works (courtesy of Docker blog):

The following diagram provides a high-level view of how the Docker swarm mode architecture is applied to securely deliver a new type of object to our containers: a secret object.

In Docker, a secret is any blob of data, such as a password, SSH private key, TLS Certificate, or any other piece of data that is sensitive in nature. When you add a secret to the swarm (by running docker secret create), Docker sends the secret over to the swarm manager over a mutually authenticated TLS connection, making use of the built-in Certificate Authority that gets automatically created when bootstrapping a new swarm.

Secrets creation is dead simple:
$ echo "My super secret password" | docker secret create super_secret -

Now we need to share our newly created secret with a service

nodes cannot request the secrets themselves, and will only gain access to the secrets when provided to them by a manager – strictly for the services that require them.

2 methods are available:

using CLI:
docker service create --name="redis" --secret="super_secret" redis:alpine

using compose (3.1+):

version: "3.1"
services:
redis:
  image: redis:latest
  deploy:
    replicas: 1
  secrets:
    - super_secret
secrets:
super_secret:
  external: true

What does it mean to our services ?

Sharing a secret to a specific service will create in the service container’s filesystem, the file :
/run/secrets/[secret name] in which sensitive data is stored.

Here comes our main topic: How to make your docker images secrets enabled ??

As explained, secrets are stored in files shared securely with our services.
Which means our Docker images must be customized to read the needed files.

Here’s a very simple example using percona-xtradb-cluster Docker image.

Percona Docker images needs at least a mysql root password which is shared using the cough environment variable cough: MYSQL_ROOT_PASSWORD
docker run -e MYSQL_ROOT_PASSWORD=my-secret-pw -d [percona:tag]

To make our image secrets enabled, we just have to give the ability to read our secret file.

Easy as one!, two!, three!

One: edit the file entrypoint.sh

Two: add MYSQL_ROOT_PASSWORD_FILE as a valid variable:

if [ -z "$MYSQL_ROOT_PASSWORD" -a -z "$MYSQL_ALLOW_EMPTY_PASSWORD" -a -z "$MYSQL_RANDOM_ROOT_PASSWORD" -a -z "$MYSQL_ROOT_PASSWORD_FILE" ];  then
        echo >&2 'error: database is uninitialized and password option is not specified '
        echo >&2 '  You need to specify one of MYSQL_ROOT_PASSWORD, MYSQL_ROOT_PASSWORD_FILE,  MYSQL_ALLOW_EMPTY_PASSWORD or MYSQL_RANDOM_ROOT_PASSWORD'
        exit 1
fi

Three: make entrypoint.sh read our secret file

if [ ! -z "$MYSQL_ROOT_PASSWORD_FILE" -a -z "$MYSQL_ROOT_PASSWORD" ]; then
  MYSQL_ROOT_PASSWORD=$(cat $MYSQL_ROOT_PASSWORD_FILE)
fi

Build your image and you’re ready to go!

Here’s an example using docker-compose:

Create the mysql root password secret:
echo R00tT00r | docker secret create percona-secret -

create the appropriate docker-compose.yml:

version: "3.1"
services:
redis:
  image: percona-server:[your special tag]
  deploy:
    replicas: 1
  secrets:
    - percona-secret
secrets:
percona-secret:
  external: true

deploy your percona secrets enabled stack:
docker stack deploy -c docker-compose.yml percona

Done!

Look for part two, where I will show more patterns for using docker secret

Have fun !

Building Percona PXC Cluster on Swarm Mode

2017-04-22T00:00:00+00:00

As part of a personal project, i had to build a Mysql Galera Cluster.
Being a Percona Server fan for several years, i decide to use PXC which stands for Percona XtraDB Cluster(Percona Galera cluster implementation)

Introduction

This blog post decribe how to build a PXC cluster on top of docker Swarm Mode (1.13+).

Requirements:

Docker tools:

Docker Engine: 1.13+ (17-04-ce recommended)
Docker Compose: 1.11+ (1.12 recommended)
Docker Machine: 0.9+ (0.10 recommended)

Docker images:

OS:

RancherOS

Deploying a Swarm Cluster

First step, let’s deploy a Swarm cluster.
This setup uses virtualbox, but D.O, GCE or AWS are fine too.

The very simple shell script below will do the job
Feel free to hack, forking my repo : github-pxc-swarm
(Pull Request highly appreciated by the way)

#!/bin/bash

#Deploy RancherOS Virtual Machines 
#Switch to latest Docker Engine available
#Switch to Debian console
for i in pxcm1 pxcw1 pxcw2 pxcw3;
    do docker-machine create -d virtualbox --virtualbox-boot2docker-url https://releases.rancher.com/os/latest/rancheros.iso $i;
    docker-machine ssh $i "sudo ros engine switch docker-17.04.0-ce";
    docker-machine ssh $i "sudo ros console switch debian -f";
    sleep 15;
    docker-machine ssh $i "sudo apt update -qq && sudo apt install -qqy ca-certificates";
done

# Initialize Swarm Manager and tokens
docker-machine ssh pxcm1 "docker swarm init \
        --listen-addr $(docker-machine ip pxcm1) \
            --advertise-addr $(docker-machine ip pxcm1)"

export worker_token=$(docker-machine ssh pxcm1 "docker swarm \
    join-token worker -q")

export manager_token=$(docker-machine ssh pxcm1 "docker swarm \
    join-token manager -q")

# Initialize Swarm Workers and add them to the cluster
docker-machine ssh pxcw1 "docker swarm join \
        --token=${worker_token} \
            --listen-addr $(docker-machine ip pxcw1) \
                --advertise-addr $(docker-machine ip pxcw1) \
                    $(docker-machine ip pxcm1)"

docker-machine ssh pxcw2 "docker swarm join \
        --token=${worker_token} \
            --listen-addr $(docker-machine ip pxcw2) \
                --advertise-addr $(docker-machine ip pxcw2) \
                    $(docker-machine ip pxcm1)"

docker-machine ssh pxcw3 "docker swarm join \
        --token=${worker_token} \
            --listen-addr $(docker-machine ip pxcw3) \
                --advertise-addr $(docker-machine ip pxcw3) \
                    $(docker-machine ip pxcm1)"

Let’s see how our cluster is doing :)

eval "$(docker-machine env pxcm1)" 
docker node ls

Output example:

ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS  
bd1oyur0ia0nkw2ru6mrrqvi3    pxcw1     Ready   Active          
jgymwuqmxlyl2g9ig6pgxkp1p    pxcw2     Ready   Active          
n783lei0zipbcryj9r75hmu2k *  pxcm1     Ready   Active        Leader  
zmjv99aho5cv0nysfferqy6qf    pxcw3     Ready   Active

Deploying PXC cluster

This PXC cluster setup uses proxySQL and Etcd.

ProxySQL as its name suggests, will act as a proxy to your sql queries (Galera doesn’t come with VIP mecanism built-in)
Etcd wil be used for nodes discovery, each galera node will register itself into your Etcd instance

As we are deploying our cluster on top of Docker Swarm, PXC instances will be hosted on workers, whereas ProxySQL and Etcd will be hosted on the manager.
This is done using placement contraints feature of docker-compose.
We’d like to use docker secret management feature, but current images doesn’t “support” it for now

All services need their own environmental variables, let’s put them in separated files.

galera.env:

DISCOVERY_SERVICE=galera_etcd:2379
CLUSTER_NAME=galera-15
MYSQL_ROOT_PASSWORD=s3cr3TL33tP@ssw0rd

etcd.env:

ETCD_DATA_DIR=/opt/etcd/data
ETCD_NAME=etcd-node-01 
ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379,http://0.0.0.0:4001
ETCD_ADVERTISE_CLIENT_URLS=http://galera_etcd:2379,http://galera_etcd:4001
ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://galera_etcd:2380
ETCD-INITIAL-CLUSTER=etcd0=http://galera_etcd:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1

proxysql.env:

CLUSTER_NAME=galera-15
ETCD_HOST=galera_etcd
DISCOVERY_SERVICE=galera_etcd:2379
MYSQL_ROOT_PASSWORD=s3cr3TL33tP@ssw0rd
MYSQL_PROXY_USER=proxyuser
MYSQL_PROXY_PASSWORD=s3cr3TL33tPr0xyP@ssw0rd

We want to use docker-compose right ?

docker-compose.yml:

version: '3.1'

services:
  proxy:
    image: perconalab/proxysql
    networks:
      - galera
    ports:
      - "3306:3306"
      - "6032:6032"
    env_file: proxysql.env
    deploy:
      mode: replicated
      replicas: 1
      labels: [APP=proxysql]
     # service restart policy
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
      # service update configuration
      update_config:
        parallelism: 1
        delay: 10s
        failure_action: continue
        monitor: 60s
        max_failure_ratio: 0.3
      # placement constraint - in this case on 'worker' nodes only
      placement:
        constraints: [node.role == manager]

  etcd:
    image: quay.io/coreos/etcd
    command:  etcd
    volumes:
      - /usr/share/ca-certificates/:/etc/ssl/certs
    env_file: etcd.env
    networks:
      - galera
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]

  percona-xtradb-cluster:
    image: percona/percona-xtradb-cluster:5.7
    networks:
      - galera
    env_file: galera.env
    deploy:
      mode: global
      labels: [APP=pxc]
     # service restart policy
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
      # service update configuration
      update_config:
        parallelism: 1
        delay: 10s
        failure_action: continue
        monitor: 60s
        max_failure_ratio: 0.3
      # placement constraint - in this case on 'worker' nodes only
      placement:
        constraints: [node.role == worker]

networks:
  galera:
    # Use a custom driver
    driver: overlay
    internal: true
    ipam:
      driver: default
      config:
      - subnet: 10.20.1.0/24

Let’s start the real thing:
docker stack deploy -c docker-compose.yml galera

Ouput example:

Creating network galera_galera
Creating service galera_percona-xtradb-cluster
Creating service galera_proxy
Creating service galera_etcd

Check how things are going:
docker stack ps galera

Output example (few minutes later):

ID                  NAME                                                      IMAGE                                NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
wxiavo62e7j5        galera_percona-xtradb-cluster.zmjv99aho5cv0nysfferqy6qf   percona/percona-xtradb-cluster:5.7   pxcw3               Running             Running 25 minutes ago                       
f7fn4zxpxzn3        galera_percona-xtradb-cluster.jgymwuqmxlyl2g9ig6pgxkp1p   percona/percona-xtradb-cluster:5.7   pxcw2               Running             Running 25 minutes ago                       
5yk2kvfnaipj        galera_percona-xtradb-cluster.bd1oyur0ia0nkw2ru6mrrqvi3   percona/percona-xtradb-cluster:5.7   pxcw1               Running             Running 25 minutes ago                       
tvt76rukcml6        galera_etcd.1                                             quay.io/coreos/etcd:latest           pxcm1               Running             Running 25 minutes ago                       
1bo8rf1s088z        galera_proxy.1                                            perconalab/proxysql:latest           pxcm1               Running             Running 25 minutes ago

One last thing to do is to register our galera nodes into proxySQL. Easy as one, two, three :)

– One:

eval "$(docker-machine env pxcm1)" 
docker ps

Output example:

docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS                NAMES
ade1183337e5        quay.io/coreos/etcd   "etcd"              2 hours ago         Up 31 minutes                            galera_etcd.1.tvt76rukcml6h7h24vaef79cz
34129b98cd75        perconalab/proxysql   "/entrypoint.sh "   2 hours ago         Up 31 minutes       3306/tcp, 6032/tcp   galera_proxy.1.1bo8rf1s088zgmsl9ho16wtk7

– Two:

docker exec -i [name of the proxySQL container] add_cluster_nodes.sh

Output example:

docker exec -i galera_proxy.1.1bo8rf1s088zgmsl9ho16wtk7 add_cluster_nodes.sh 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   551  100   551    0     0  68328      0 --:--:-- --:--:-- --:--:-- 78714
10.20.1.9
Warning: Using a password on the command line interface can be insecure.
Warning: Using a password on the command line interface can be insecure.
10.20.1.7
Warning: Using a password on the command line interface can be insecure.
Warning: Using a password on the command line interface can be insecure.
10.20.1.8
Warning: Using a password on the command line interface can be insecure.
Warning: Using a password on the command line interface can be insecure.
Warning: Using a password on the command line interface can be insecure.
Warning: Using a password on the command line interface can be insecure.

– Three:

mysql -h$(docker-machine ip pxcm1) -uproxyuser -p

Output example:

mysql -h$(docker-machine ip pxcm1) -uproxyuser -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 5.1.30 (ProxySQL)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show variables like '%host%';
+-------------------------------+--------------+
| Variable_name                 | Value        |
+-------------------------------+--------------+
| host_cache_size               | 279          |
| hostname                      | 148ba588c919 |
| performance_schema_hosts_size | -1           |
| report_host                   |              |
+-------------------------------+--------------+
4 rows in set (0,00 sec)

Done!

You can enjoy a fresh PXC cluster on top of swarm managed by proxySQL :)

Have fun :)