A Temporal activity worker written in Go for ZFS branch management. It listens on the fs-worker task queue and handles a single activity:

The worker expects a pre-formatted base zvol and snapshot to exist before any InitBranch activity with mode zvol is dispatched. This is created once during host setup by scripts/vm-setup.sh:

<pool>/_base/             ZFS filesystem (parent)
<pool>/_base/vol          thin zvol, 1 GiB, formatted ext4
<pool>/_base/vol@empty    snapshot of the freshly-formatted zvol

When InitBranch is called with mode zvol, the branch is created by cloning <pool>/_base/vol@empty — no formatting happens at activity time.

When InitBranch is called with mode zds, a plain ZFS filesystem dataset is created directly — the base volume is not involved.

• Go 1.23+
• A running Temporal server
• zfs CLI tool available in $PATH (part of zfsutils-linux on Ubuntu)
• The worker must run as a user with permission to execute ZFS commands (typically root or a user with appropriate ZFS delegation)
• The base volume (<pool>/_base/vol@empty) must exist before dispatching zvol branches (created by make setup)

All configuration is via environment variables:

go build -o fs-worker .

# With defaults (Temporal on localhost, testpool ZFS pool)
./fs-worker

# With custom configuration
TEMPORAL_HOST=temporal.example.com:7233 \
TEMPORAL_NAMESPACE=production \
ZFS_POOL=datapool \
./fs-worker

Input

{
  "id": "my-branch",
  "mode": "zvol"
}

ID validation rules

• Must not be empty.
• Must not start with _.
• Must not contain / or @.
• May only contain alphanumerics and the characters -, _, ., :.

Output (zvol mode)

{
  "dataset": "testpool/my-branch",
  "device": "/dev/zvol/testpool/my-branch",
  "fs_type": "ext4",
  "mode": "zvol"
}

Output (zds mode)

{
  "dataset": "testpool/my-branch",
  "mount_point": "/testpool/my-branch",
  "fs_type": "zfs",
  "mode": "zds"
}

<pool>/
├── _base/                           ← created by vm-setup.sh
│   └── vol                          ← thin zvol, 1 GiB, formatted ext4
│       └── @empty                   ← snapshot (base for all zvol clones)
├── my-branch                        ← zvol clone of _base/vol@empty
└── another-branch                   ← zds (plain ZFS filesystem dataset)

The base volume under _base/ is created once during host setup and is never modified at runtime.

fs-worker/
  main.go               Entrypoint: reads config, dials Temporal, registers activities, runs worker
  activities/
    types.go            Input/output structs and BranchMode constants
    zfs.go              ZFS helpers: naming, validation, CLI wrappers, device polling
    activities.go       FsWorkerActivities struct and InitBranch implementation
  go.mod
  go.sum
  scripts/              Deployment and VM management scripts
  Makefile              Make targets for all lifecycle operations
  README.md
  .gitignore

The scripts and Makefile support two target modes:

Mode is auto-selected: once REMOTE_HOST is present in .env (written automatically by make provision), all scripts and make targets default to remote mode with no extra flags required. --remote / REMOTE=1 exist as an explicit override for edge cases where you want to force remote mode without a REMOTE_HOST in .env.

Create a .env file in the project root (it is gitignored). All scripts load it automatically.

The only values you might want to override are the VM name and pool name:

# .env (all optional)
VM_NAME=zfs-dev       # Multipass VM name          (default: zfs-dev)
ZFS_POOL=testpool     # ZFS pool name               (default: testpool)

# .env
REMOTE_HOST=1.2.3.4            # IP or hostname of the bare-metal machine (required)
REMOTE_USER=ubuntu             # SSH login user                            (default: ubuntu)
REMOTE_PEM=/path/to/key.pem    # Path to the SSH private key on your Mac  (required)
REMOTE_POOL_DEVICE=/dev/nvme1n1 # Block device for the ZFS pool on remote  (required)

# Optional
REMOTE_PORT=22
REMOTE_WORK_DIR=/home/worker/fs-worker
ZFS_POOL=testpool
TEMPORAL_HOST=http://localhost:7233
TEMPORAL_NAMESPACE=default
TEMPORAL_TASK_QUEUE=fs-worker

When vm-provision.sh is used to create an EC2 instance, it writes REMOTE_HOST, REMOTE_PEM, REMOTE_USER, REMOTE_WORK_DIR, and REMOTE_POOL_DEVICE into .env automatically — no manual editing required after provisioning.

Install these tools on your Mac before running any scripts:

brew install go              # Go toolchain (local builds)
brew install temporal        # Temporal CLI (server + smoke tests)
brew install multipass       # Multipass (local VM mode only)
brew install awscli          # AWS CLI    (remote EC2 mode only)
brew install jq              # JSON tool  (remote EC2 mode only)

The local mode runs the worker inside a Multipass Ubuntu VM on your Mac. The project directory is bind-mounted into the VM so edits are reflected immediately without an explicit sync step.

temporal server start-dev

This starts Temporal on localhost:7233 and opens the Web UI at http://localhost:8233.

make setup
# or: ./scripts/vm-setup.sh

This creates a Multipass VM (zfs-dev, 2 CPU / 4 GB RAM / 20 GB disk), installs ZFS and Go, creates a loopback-backed ZFS pool (testpool), creates the pre-formatted base volume (_base/vol@empty), mounts the project directory into the VM, and installs a fs-worker systemd unit.

Pass --recreate / make setup-recreate to tear down and rebuild the VM from scratch.

make build           # debug build
make build-release   # optimised build

make run             # foreground — logs stream to your terminal
make run-detach      # background via systemd
make logs            # follow logs (works for both foreground and systemd)

make test

make stop            # stop worker + export ZFS pool + halt VM
make stop-service    # stop worker only, leave VM running
make suspend         # suspend the VM instead of shutting down

The remote mode targets a real Linux machine over SSH. The recommended instance type is a1.metal (arm64 bare-metal) because ZFS benefits from direct hardware access and avoids nested-virtualisation overhead.

aws sso login
# or: export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...

make provision
# or: ./scripts/vm-provision.sh

vm-provision.sh is idempotent — re-running it safely skips steps that are already done. It performs the following:

1. Creates (or reuses tagged) VPC, subnet, internet gateway, and route table.
2. Creates a security group with SSH (port 22) open only to your current public IP.
3. Resolves the latest Ubuntu 24.04 arm64 AMI from Canonical.
4. Launches an a1.metal instance with a 30 GB gp3 root volume and IMDSv2 enabled.
5. Allocates an Elastic IP and associates it with the instance so the public IP is stable across reboots.
6. Creates and attaches a separate EBS gp3 data volume (default: 20 GB) for the ZFS pool.
7. SSHes into the instance to detect the NVMe device name of the data volume and writes it as REMOTE_POOL_DEVICE in .env.
8. Writes REMOTE_HOST, REMOTE_USER, REMOTE_PEM, and REMOTE_WORK_DIR to .env.

After this step all other scripts and make targets work with no further configuration.

Configurable variables (set in .env or environment before running):

Run this on your Mac. The scripts set up a reverse SSH tunnel so the worker on the remote host can reach Temporal at localhost:7233.

temporal server start-dev

make setup
# or: ./scripts/vm-setup.sh

The script connects over SSH and:

• Installs zfsutils-linux, zfs-dkms, and other required packages.
• Installs Go 1.23.6 from the official tarball under /usr/local/go.
• Creates the ZFS pool directly on the EBS block device (REMOTE_POOL_DEVICE).
• Creates the pre-formatted base volume (_base/vol@empty) for zvol branches.
• Enables the standard ZFS import services so the pool is automatically re-imported after a reboot.
• Syncs the project source to REMOTE_WORK_DIR via rsync.
• Installs a fs-worker systemd unit that starts the worker on boot.

make build          # debug build
make build-release  # optimised build

The source is rsynced to the remote before every build.

make run            # foreground — logs stream to your terminal
make run-detach     # background via systemd
make logs           # follow logs

The worker exposes Prometheus metrics on port 9090. To scrape them locally (e.g. for the monitoring stack), forward the port through SSH:

make port-forward-bg   # start tunnel in background (localhost:9090)
make port-forward-stop # stop it

The reverse tunnel for Temporal (-R) is set up automatically by vm-run.sh when TEMPORAL_HOST in .env is localhost — no manual forwarding needed.

make test

make shell       # SSH as REMOTE_USER
make shell-root  # SSH then sudo -i bash

make stop-service   # stop worker service, leave host running
make stop           # stop worker + export ZFS pool (host stays up)

make suspend is not available in remote mode — there is no VM lifecycle to manage.

The security group's SSH rule is locked to the IP you had when vm-provision.sh ran. If your IP changes (different network, VPN, ISP reassignment) all SSH-based operations will time out. Fix it with:

make ssh-allow
# or: ./scripts/vm-ssh-allow.sh

make provision-destroy
# or: ./scripts/vm-provision.sh --destroy

This terminates the EC2 instance, deletes the EBS data volume, releases the Elastic IP, deletes the security group, and removes all REMOTE_* and AWS_* entries from .env. The VPC and subnet are left in place (they are cheap) but are tagged and will be reused on the next make provision run.

make help   # print all available targets

Once REMOTE_HOST is set in .env, all targets automatically target the remote host. REMOTE=1 is available as an explicit override if you need to force remote mode without REMOTE_HOST being set:

make build REMOTE=1

The worker exposes a Prometheus metrics endpoint on port 9090 (configurable via METRICS_ADDR). The port-forward script tunnels this alongside the Temporal port so you can scrape metrics locally.

cd monitoring
docker compose up -d

This starts:

1. Open Grafana at http://localhost:3000
2. Go to Connections → Data sources → Add data source
3. Select Prometheus, set URL to http://prometheus:9090, click Save & test
4. Create a dashboard or explore metrics like temporal_activity_execution_latency

cd monitoring
docker compose down

Every script except vm-provision.sh and vm-ssh-allow.sh accepts --remote to force remote mode regardless of .env. This is rarely needed — once REMOTE_HOST is set in .env remote mode is selected automatically.