⢠⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠈⠙⠶⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⠓
⠀⠀⠀⠈⠙⠲⢤⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⡼⠉⠀
⠀⠀⠛⠤⣠⡀⠀⠀⠉⠑⠳⠤⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⡴⠋⣁⡴⠃
⠀⠀⠀⠀⠀⠈⠉⠀⠀⠀⠀⢀⠈⠙⢄⡀⠀⠀⢸⠀⠀⠀⠀⠀⠀⣠⠤⠤⣤⠤⠀⠀⠀⠀⢀⡤⡖⠉⠀⠖⠞⠁⢀⠀
⠀⠀⠀⠠⠤⣀⡀⠀⠀⠀⠀⢸⡀⠀⠀⠹⣆⠀⢹⡤⡀⠀⢀⣴⢫⣥⢀⠾⠁⠀⠀⠀⢀⣶⠋⠀⣧⢀⠀⣠⠤⠖⠉⠀
⠀⠀⠀⠀⠀⠉⠑⠒⠆⠀⠀⠸⡇⠀⠀⠀⠹⡦⠈⠓⡥⣄⠈⠉⠘⠋⢸⢀⠀⠀⠀⠀⣸⠉⠀⢠⡻⠈⠀⢋⣀⡤⠀⠀
⠀⠀⠀⠀⠀⠤⣀⣀⡀⠀⠀⠀⢳⠀⠀⠀⠀⠉⢧⣄⠓⢚⣪⡤⠀⠀⠀⢳⡄⠀⠀⣴⠃⠀⢀⣾⠁⠀⠘⠉⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⠀⣀⠈⣧⣄⠀⠀⠀⠀⠑⠶⣀⣀⡀⠀⠀⠀⠀⢯⠶⠉⠀⢀⢀⡾⠈⠘⠚⠒⡤⠄⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⢤⠒⠊⠉⠁⠀⠈⠱⣄⡄⠀⠀⠀⠀⠀⠈⠁⠀⠀⠀⠀⢸⡆⣠⣠⠶⢫⣀⠋⠒⣴⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⠤⠚⠀⡀⠀⠉⠓⠒⠤⠤⠤⠴⠀⠀⠀⠀⠀⢸⡄⠈⠸⢤⠈⠉⢦⡠⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠾⠁⠀⠀⡴⠋⠀⢠⡆⠀⢀⠀⠀⡗⠀⠀⠀⠀⢠⢯⡙⢧⡄⠀⢷⡦⠀⠁⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠾⠃⢀⡴⠋⠀⡰⠏⠀⡞⠎⢰⢄⢀⣰⡟⠈⢲⡄⠹⠓⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⠁⠀⠘⠁⢀⠞⠀⠀⢸⡯⡟⠹⡦⠀⠈⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⠛⠀⣠⠞⠋⢰⠳⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⡼⠉⠀⣼⠉⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡞⡀⠀⢰⡁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣸⡄⣿⠀⠌⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢻⠃⡏⠀⢹⠁⠀⢠⠶⠓⢧⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⡆⢹⠀⠀⢧⡀⡀⠀⢀⡼⠂⢀⠀⠀⠀⡴⠛⠉⠉⢦⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢹⣌⢧⣄⣰⠮⣩⠍⠉⠀⣠⠞⠂⠀⠀⢫⣤⣠⠀⣸⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠱⡎⠛⣷⣝⠶⢏⣩⣍⣁⣠⠀⠀⠀⠀⠀⠀⢠⡇⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢳⠀⠙⢻⣦⡈⠿⣍⣍⡉⠀⠀⠀⠀⣀⡠⠎⠁⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢧⠀⠀⠉⢫⣄⠈⠳⣈⠉⠉⠉⠉⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⠀⠀⠀⠹⡆⠀⠹⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⠂⠀⠀⠀⠀⣹⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀

FYI that work on Phoenix is paused. It may restart, but likely in a very different form. Phoenix proved an effective testing ground, but the results of those testing point development in a dramatically different direction.

Phoenix is a "Multi-Server Process Supervisor" — its job is to reliably run things on more than one machine.

Its peers are tools like:

• Kamal Deploy
• Kubernetes
• Docker Swarm
• HashiCorp's Nomad

Or PaaS offerings like Heroku

But you might also use it to replace system configuration or orchestration tools such as:

• Salt(Stack)
• Ansible
• Puppet
• Chef

Short version

Infrastructure software

• Should be blatantly open source.
• Should be simple & understandable.
• Should not hide the network.
• Should be capable of running other workloads than Docker/OCI containers.
• Should be self contained.
• Should be fast.
• Should be multi-homed (running on multiple clouds, and or bare metal from day zero).

Long version

It's a story. Though I'll try to make it a short one ;)

Hey there!

My name is Jordan, and I've been writing software and running it for some time. Long enough that it counts as a long time in our industry. I was there to see Ubuntu born, Thomas Hatch's "Hey I've got this thing I wrote" email (SaltStack's founder), I used Docker in production when that was considered crazy talk, ran a Kubernetes devops consultancy. I've also spent some time (and learned a thing or two) from systems that didn't capture as much mind share, things like Illumos, Triton, unikernels & OSv.

While I've used a lot of systems, and I really appreciate how the DevOps environment has matured over the last decade, I've still had an itche to scratch: nothing did quite what I wanted.

Kubernetes is way to complicated. It is getting better, but its a meme at this point. Docker Swarm is simple, but by attempting to make multiple machines appear as one machine, we lose sight of the network and have difficulty scheduling loads that need to be network/physically close (or far) from each other.

I needed a system that is simple, that I can trust is going to remain open, and that can handle running different kinds of loads — super dense, thousands of services per host using small self contained binaries, typical docker/OCI container loads, and loads that require virtual machine barriers (because they run untrusted code, or for compliance reasons).

I also want a system which is fast. Which means avoiding polling, simple network topology, avoiding nested load balancers, etc.

If these sound appealing, please give Phoenix a try!

I hope you find it useful!

A cluster starts with a master node. The master runs a proxy / load balancer to and handles SSL termination. The master acts as a registry — for docker images, binaries, or VM images. The master maintains the state for the cluster.

Minion nodes run the actual workloads (the same machine that is running the master daemon can also run a minion, though this is not typical).

The minion nodes run things — stand alone binaries, docker or OCI containers, or virtual machines.

Communication between the master and minion nodes is all routed over Wireguard links.

Each service in a cluster is given an dedicated IPv6 address. No DNS servers are run, but every service can be referenced by its dedicated address.

Regular binaries are run directly by the minion process supervisor. Docker and OCI containers are run via crun, crun is a fully featured OCI runtime, though it is smaller, more memory efficient, and smaller than the runc library which is typically used by Docker, Podman & Kubernetes. Virtual machine images are run under Firecrack.

The minion nodes forward information about running processes — the number of restarts, basic system load, etc back to the master node.

The registry syncs binaries, container images, and VM images to each minion node, using efficient, block level de-duplication.

Minion nodes can continue to operate and keep their processes running even if there is a network partition between them and the master node.

The system supports oneshot and longshot/daemonized jobs, automatically handles SSL certificates via LetsEncrypt, service health checks, and auto-restarting.

Cluster configuration explicitly sets where loads run. If you need a more dynamic scheduler, you can write one on top of Phoenix.

The entire system is easy to setup, using a single, small, cross platform (x86/AMD64/ARM) binary.

Expect breaking changes. Development is on hiatus, and will likely resume in an entirely different approach.

Phoenix is a single, self contained, relatively small (~5mb) executable that is all the things, on all architectures. The single executable is:

• The installer
• The master node daemon
• The minion node daemon
• The CLI interface

And the same binary will run on x86 & ARM.

(FWIW it will also run under Linux + Mac + Windows + FreeBSD + OpenBSD + NetBSD + BIOS on AMD64 and ARM64, but currently the only fully supported & validated OS for master and minion nodes is Alpine Linux)

You can get it from:

curl -O https://phoenix.jordanschatz.com/releases/current/phoenix

Then make it executable with:

chmod +x phoenix

and copy in into your path

mv phoenix /usr/local/bin/

1. Run phoenix master <public-ip> on a machine that will become your master node.

   This will:

   • Install required dependencies on the master node.
   • Setup an OpenRC system service to run the Phoenix master daemon, and start the master daemon.
   • Setup an OpenRC system service to run the Phoenix public daemon, and start the public daemon.
   • Setup cluster configuration store
   • Configure & Start the proxy
   • Generate any needed SSL certs via LetsEncrypt
   • Create a Wireguard endpoint
   • Create /srv/phoenix/registry for images

2. Run phoenix minion <name> <master-public-ip> <master-key> <minion-public-ip> <minion-private-ip> on one or more machines that will become your minion nodes. When you setup the master it printed it's public IP & key. You can also remind yourself of them with phoenix master-info. phoenix master-info also lists the next open minion node IP address.

   This will:

   • Install required dependencies on the minion node.
   • Setup a OpenRC system service to run the Phoenix minion daemon, and start the minion daemon
   • Setup cluster configuration store
   • Create a Wireguard endpoint
   • Send a request to the master node requesting to join the cluster

3. The minion nodes will each send a request to the master node to join the cluster.

   This is the only time that minion nodes communicate with the master nodes in the clear — no sensitive information is transferred, and the exchange is necessary to setup the secure Wireguard channel that all future communication is sent over.

   However it is the time when the cluster is at its most vulnerable. Each minion will only send its request to join the cluster once, and will send it immediately after being stood up. You should immediately access the master, and accept (and only accept) the node's you expect. Only accept a node into the cluster once. Only accept nodes that you expect.

4. On the master node accept each minion to the cluster.

   • phoenix waiting lists the machines that have requested to join the cluster.
   • phoenix accept <name> to accept a machine to the cluster.

5. The cluster is now setup and ready to go!

The "things" you want to supervise (binaries, docker images, OCI images, Firecrack VM images) need to be added to the /home/phoenix/registry folder on the master node. They will be automatically synced / pre-cached on the minion nodes.

Phoenix exposes a low level API, in JSON, for configuring what processes to run.

This is intentionally a low level API, which could be wrapped by higher "porcelain" layers. Example possibilities for such layers are a YAML based config like Kubernets, Kamal, etc, one that that directly consumes Heroku style Procfiles or one that auto-schedules loads based on cluster state.

Description of the config:

{
  "machine": "The name of the machine to run on. Required."
  "name": "The process name. This must be unique. Required."
  "image": "The binary image, Docker/OCI image, or VM image to run. Required."
  "type": "docker" or "binary" or "vm" Required.

  "duration": "oneshot" or "daemon". Daemon process are always restarted,  oneshots never are. Optional, defaults to "daemon"
  "env": An array of strings, like ["TERM=xterm"]. Optional, default empty.
  "proxyname": A domain name that will be exposed by the load balancer and proxy this service. "example.com". Optional.
  "proxyport": A port that the service will be available at for the proxy. All ports are available within the VPN, but this port will be where the proxy expects a HTTP service to be available at. Default 8000
  "healthcheck": "A URL path, such as /healthcheck to test if the process is health. Optional"

  "ip": An IPv6 address to be used by this process. This field is optional, and defaults to an automated assignment by Phoenix.Optional, and generally assigned by Phoenix. However you can specify it so long as it is unique.
  "args": An array of arguments to be passed to the process, if it is of type "binary". Otherwise ignored. Optional.
  "link": An array of process names that this process should be allowed to communicate with.
}

Addition Notes

proxyname

The load balancer will automatically provision an SSL cert via LetsEncrypt, and handle SSL unwrapping. Downstream of the proxy traffic is carried over the wireguard VPN, so can be sent "in the clear" inside the VPN tunnel. You can back a service with as many processes as you would like by specifying the same proxyname for all of them. Phoenix will automatically load balance between them

healthcheck

Healthchecks are really useful. I advise giving them a try and/or making them a required field for your specs. Phoenix does healthchecks a little differently than what you may be familiar with from the Dockerfile HEALTHCHECK instruction. For some discussion see this OCI spec issue. We (like Kubernetes) think that the health check should be handled at the orchestration/supervision level, not within the container. Additionally OCI images, VM images, and binaries don't have built in support for health checks, and reaching inside a VM to run a healthcheck would break the security boundary, and is nonsensical for binaries.

We also want to make healthchecks as easy as possible. Simply specify a URL, and if it returns a non-200 response, the process will be considered unhealthy and terminated/restarted as applicable.

What about processes that don't speak HTTP? For example a Postgres container? For application startup, we've found it to be a better pattern to build automated re-trying into the application level containers — they shouldn't require the orchestration system to insure all of their subsystems are up, and healthy before they are started. Ideally they are also engineered to tolerate their dependencies transient restarted.

ip

IP addresses are unique to each service, and are stable. Moving the service to another machine, potentially in another data center etc. does not change it's IP, the network "handle" where you can reach it.

By default Phoenix will auto-assign IP addresses, and you should refer to containers by their more human friendly, unique name. However you can manually assign IP addresses, so long as they are unique, and in the fd38:dde8:32dd::/48 subnet.

You have a cluster, you have image/thing staged on the master node, you have a spec file, not lets run it:

phoenix deploy spec.json

Updated the spec, the image, or just manually restart?

phoenix update spec.json

Ready to remove a service?

phoenix del service-name

Get a service's status?

phoenix status service-name

Get the spec for a service?

phoenix info service-name

Get a list of running services?

phoenix services

It is useful to have a name for things that you can execute: docker / OCI images, virtual machine images, and binaries. These are all very different formats, but their purpose is the same: a "blob" of something that you can get the computer to execute. I'd rather not call them "executable" — while accurate, it is a lot of syllables, and at least for me has a strong connotation of window .exe files. We can't call them "containers" without introducing a lot of confusion with the docker ecosystem, so I'll just call them images. They are after all already called docker/OCI images, virtual machine images and binary files can be considered a process "image".

So what image formats does Phoenix support?

For binaries: whatever will run on your target nodes, which generally means statically linked binaries or ones linked against musl (as the nodes all run Alpine Linux).

For virtual machines: any image that firecracker can run compressed with zstd and having a file extension of .zst

A planned feature is to support running docker and OCI images as VM images, handling the translation in phoenix.

For Docker/OCI images: The docker & OCI specs have just a bit of haze about them. Unless you are very in the know, you probably round them both (and container formats too) to "docker". The docker and OCI specs differ in a few minor (but significant) ways. Things are generally trending towards OCI, and away from Docker's legacy "v2" format, but not all tooling outputs/consumes OCI images by default. Phoenix only supports OCI images and creates OCI containers.

If you are using Podman or Buildah to create images, you will create OCI images by default. If you are using docker's buildx then you can specify exporting images as OCI tarballs

There is also the handy Skopeo which can download images from various repositories and convert them to OCI format if needed.

If you are currently using docker, and looking for the shortest path, just use:

docker image save IMAGE:TAGNAME -o image.tar

So long as you are using Docker v25.0 or latter (2024-02-06) this will output a valid OCI image, in a tar. Phoenix can use that just fine. Specify the name image (.tar will be assumed) in the process's spec file.

This is a monorepo phoenix proper is in the phoenix directory, testy, an unreliable service for testing phoenix's supervising is in the testy directory, and higher level tools live in the porcelain directory.