fleetlock-consul

A FleetLock server backed by Consul KV.

FleetLock is a simple lock protocol used by Zincati to coordinate reboots across a cluster of Fedora CoreOS nodes. Only one node per group can hold the lock at a time, ensuring that rolling OS updates don't take down the entire cluster.

A node acquires the lock before rebooting and releases it after coming back online:

Node-1               fleetlock-consul              Consul KV
  |                        |                           |
  |-- POST /v1/pre-reboot ->                           |
  |                        |-- session create --------> |
  |                        |-- acquire mutex lock ----> |
  |                        |-- get data key ----------> |
  |                        |   (no lock held)           |
  |                        |-- put data key "node-1" -> |
  |                        |-- release mutex lock ----> |
  |<--------- 200 OK -----|                            |
  |                        |                           |
  |   ... node reboots and comes back ...              |
  |                        |                           |
  |-- POST /v1/steady-state ->                         |
  |                        |-- session create --------> |
  |                        |-- acquire mutex lock ----> |
  |                        |-- get data key ----------> |
  |                        |   (value = "node-1")       |
  |                        |-- delete data key -------> |
  |                        |-- release mutex lock ----> |
  |<--------- 200 OK -----|                            |

Meanwhile, if another node tries to lock while node-1 holds it:

Node-2               fleetlock-consul              Consul KV
  |                        |                           |
  |-- POST /v1/pre-reboot ->                           |
  |                        |-- get data key ----------> |
  |                        |   (value = "node-1")       |
  |<--------- 409 --------|                            |
  |   {"kind":"failed_lock"}                           |

The server uses two levels of keys in Consul KV:

• Mutex lock (group::lock) - a short-lived Consul session lock that protects read-modify-write operations on the data key. TTL 15s, auto-deleted on session expiry.
• Data lock (group) - a plain KV key that records who holds the lock. This persists until explicitly deleted on unlock.

This separation ensures that the data lock survives Consul session expiry while still providing safe concurrent access.

The data-lock value is a small JSON object so a manual consul kv get com.coreos.fleetlock/<group> is readable:

{"id":"b7834d91...","node":"nomad-1.example.internal"}

id is the FleetLock client id sent by Zincati and is authoritative for ownership. node is the client's hostname, resolved best-effort from its request source address via the Consul catalog and stored for display only (it may be empty, and never affects locking). Values written by older versions (a bare id string) are still read correctly.

The steady-state (unlock) endpoint deletes the data key synchronously and only returns 200 OK once the lock is actually released. On a transient failure (mutex contention, Consul briefly unreachable) it retries for a short window and, if still failing, returns a non-2xx.

This is safe because Zincati does not give up after a single steady-state call: while it fails, the update agent stays in its pre-steady state and re-reports on the next tick until it succeeds. Acking 200 before the delete actually happened would instead tell Zincati the lock is released while it is not - and any retry state kept only in the server's memory is lost if the process restarts, leaving the data key orphaned and blocking every other node.

If the caller is not the current holder (someone else's lock, or nothing held), there is nothing to release, so the endpoint returns 200 OK and does not retry.

The server exposes Prometheus metrics at GET /metrics:

Standard Go runtime (go_*) and process (process_*) metrics are also exported.

A lock that stays held far longer than a reboot takes indicates a stuck lock. Alert on it, e.g. max by (group) (fleetlock_lock_held) == 1 held for 30m. A rising fleetlock_requests_total{endpoint="steady-state",result="failed"} means unlocks are failing and locks may be about to strand.

This project is designed for clusters that already run HashiCorp Consul for service discovery or configuration. If you have Consul, you get a distributed lock backend for free - no additional infrastructure needed.

Good fit:

• Clusters already running Consul (Nomad + Consul, Kubernetes + Consul)
• Small to medium clusters (3-50 nodes)
• Environments where simplicity matters more than high-throughput locking

If you don't run Consul, consider FleetLock with etcd or airlock.

All settings are configured via environment variables with the FLEETLOCK_ prefix:

docker run -d --network host \
  -e FLEETLOCK_CONSUL_ADDRESS=127.0.0.1:8500 \
  axxapy/fleetlock-consul

variant: fcos
version: "1.5.0"
systemd:
  units:
    - name: fleetlock.service
      enabled: true
      contents: |
        [Unit]
        Description=FleetLock server
        After=network-online.target consul.service
        Requires=network-online.target

        [Service]
        ExecStart=/usr/local/bin/fleetlock-consul
        Environment=FLEETLOCK_CONSUL_ADDRESS=127.0.0.1:8500
        Restart=always

        [Install]
        WantedBy=multi-user.target

Configure Zincati to use the FleetLock strategy:

[identity]
rollout_wariness = 0.5

[updates]
strategy = "fleet_lock"

[updates.fleet_lock]
base_url = "http://127.0.0.1:9090"

See the Zincati fleet_lock documentation for details.

make help       # list all targets
make test       # unit tests
make e2e        # end-to-end tests (docker compose)
make coverage   # test coverage report
make docker     # build docker image

• FleetLock protocol spec
• Zincati update strategies
• Consul KV
• Consul sessions and locking