frostyard lab

A GitOps-driven QA pipeline for snosi bootc images, running on the single-node selfie k3s cluster. Everything is declared in git, reconciled by Argo CD, and orchestrated by Argo Workflows.

Modelled on projectbluefin/lab, with the parts that assume Fedora replaced by ones that suit a Debian/mkosi image family.

snosi publishes bootc OCI images continuously. This repo is the machinery that answers "is the image that just got published actually good?" without anyone watching:

1. A CronWorkflow polls the registry digest for an image tag.
2. If the digest moved, the QA pipeline runs against that exact digest.
3. Suites run inside the image itself — booted as a nested systemd container.
4. The new digest is recorded only after QA passes, so a failure retries on the next poll instead of being silently marked as seen.

No hypervisor, no SSH, no persistent test machines. The container lanes are Kubernetes-native end to end.

image-poll CronWorkflow
        │
        ▼
  skopeo inspect → digest
        │
        ▼
  compare with `image-polling-digests` ConfigMap
        │
        ├─ unchanged ──────────────► exit cleanly
        │
        └─ changed ────────────────► snosi-qa-pipeline (pinned to that digest)
                                     │
                                     └─ run-container-tests, one lane per suite
                                        │
                                        ├─ podman pull <image>@<digest>
                                        ├─ podman run --systemd=always /sbin/init
                                        ├─ apt-get install python3-behave
                                        └─ behave tests/<suite>/features
                                     │
                                     ▼
                          persist digest ONLY on success

GitOps loop:

git push main
    │
    ▼
Argo CD reconciles
    │
    ├─ argo/workflow-templates/ ──► WorkflowTemplates  (App: frostyard-lab)
    └─ manifests/               ──► CronWorkflows, RBAC, config
                                                       (App: frostyard-lab-infra)

WorkflowTemplates are never applied by hand. selfHeal: true reverts a manual kubectl apply on the next reconcile, so git is the only way to change a pipeline. The Application split and its hand-applied bootstrap boundary are recorded in ADR-0001; the digest-gated trigger flow in ADR-0002.

The suites boot the bootc image as a nested systemd container (podman run --systemd=always … /sbin/init) and run behave against that live system. Tests then read exactly like assertions a user would make on a running machine, and a lane costs a pull plus ~20 seconds of boot rather than a VM.

The trade is real and bounded — a container cannot assert on:

• the kernel (it runs the host's), so snowfield's linux-surface kernel is not covered by its container lane
• a graphical seat, so "GDM actually starts a session" is out of reach; the desktop suite asserts installation and configuration only
• the disk layout — EROFS, dm-verity, Secure Boot, TPM/LUKS /var, and the A/B update path all need a real boot

Those belong to the incus VM lane below. The container lanes cover the large majority of what breaks, cheaply and fast.

run-incus-vm-tests boots a published ISO in an incus VM with UEFI Secure Boot and a vTPM, then asserts on the serial console. It covers the one class of failure no container lane can see: if the shim/UKI signing chain is broken, OVMF refuses to boot the image and only this lane notices.

How a pod drives incus without SSH. The host's /usr/bin/incus is a shell wrapper that sets PATH and LD_LIBRARY_PATH over /usr/incus. The workflow pod mounts that directory plus /var/lib/incus/unix.socket and runs the host's own client — version-matched by construction, with nothing to keep in sync and no shell on the host. swtpm comes from the same directory, so the vTPM needs no host package.

kubectl create -f argo/snosi-vm-boot-test.yaml

Guests are named after the workflow and deleted from an EXIT trap, so a failure mid-run cannot leak a VM holding host memory and vTPM state. Runs serialize on the snosi-vm-qa semaphore.

ISOs are cached on the host at /var/lib/snosi-lab/iso and validated against the origin's ETag on every run — snow-live-latest.iso is a stable name whose bytes change, so caching on filename alone would pin the lane to a stale artifact.

firn replaces both fisherman (bootc) and snosi-install (native A/B) as the one snosi installer. This lane is its analogue of the native-install lane, but a matrix: one run-firn-install-tests invocation is a single cell (family, image, encryption, secureboot), and firn-install-test.yaml fans out a representative 12-cell set with withItems. It generates a recipe TOML per cell in the guest, drives firn install <recipe> --confirm <disk> --json-progress, then boots the result — for encrypted cells, booting is the unlock proof. Cells serialize on the snosi-vm-qa semaphore (one VM at a time), so the full matrix runs back-to-back; trim the withItems list for a smoke run.

The matrix covers every bootc encryption mode (none, luks-passphrase, tpm2-luks, tpm2-luks-passphrase) and every ab mode (none, luks, tpm2-luks), Secure Boot on and off in both families (ab pre-seeds the snosi MOK into the guest varstore, exactly as the native lane does), cayo + snow throughout, snowfield once. The bootc × tpm2-luks* cells are the point: they exercise the encrypted-boot unlock firn ADR-0012 installed but left unproven.

The firn ISO is published by snosi's build-native-images.yml (build-iso → promote-iso), installing firn from the frostyard-firn apt package (snosi PR #699 switched the published installer from native-installer to firn-installer). iso-url points at the snosi-installer-latest alias; the A/B pubring-path is /usr/lib/snosi/os-update-pubring.gpg (shipped by the firn-installer mkosi).

The three installer lanes are the ones that matter most. Booting an image tests an artifact; only running an installer tests the thing that creates the on-disk layout — none of verity, LUKS, the A/B slots, or a bootc deployment exists in a shipped image at all.

snosi ships native A/B and bootc install paths, and bootc deliberately splits again into mechanics and secure tiers. Native A/B uses a signed sysupdate image with dm-verity; bootc owns its deployment layout; secure bootc assembly must go through the external recipe-driven installer. None of these lanes substitutes for another.

The native A/B install lane is verified green against cayo-ab:

installed and verified: cayo-ab (verity+luks+erofs, secureboot=false, skip-mok=true)
  verity=ok            dm-verity backing the root device
  luks=ok              /var is LUKS
  varsource=/dev/mapper/var
  rootfs=erofs
  osrelease=cayo-20260805002345
  bootc=absent         expected — native A/B does not use bootc

verity, luks, and rootfs=erofs are the gating checks: each is created by snosi-install at install time and exists in no published image, so a lane that passes them has genuinely exercised the installer.

Corrected 2026-08-05. This section previously argued that a *-ab.disk.raw is a pre-install artifact which cannot be expected to boot standalone. That was wrong on both counts, and it was wrong in the direction that let a broken harness look like an open question about snosi.

The image is a complete, self-contained bootable system. Inspecting the published cayo-ab disk directly:

1  esp                    1.0 GiB  vfat    shim + MokManager + systemd-boot + UKI
2  cayo_<ver>_v         256.0 MiB  verity  slot A hash
3  cayo_<ver>_r           5.0 GiB  erofs   slot A root — populated
4  _empty               256.0 MiB  verity  slot B — empty, awaiting first update
5  _empty                 5.0 GiB  root    slot B — empty
6  var                    4.0 GiB  ext4    plain, NOT LUKS

The UKI's embedded cmdline is roothash=5c356dcd…9ab76e69 lockdown=integrity console=ttyS0 rd.luks=1 rd.etc.overlay=1, and that roothash is exactly the slot-A root partition UUID concatenated with the verity partition UUID — the systemd convention, correctly formed. Booted with Secure Boot off it reaches multi-user.target and graphical.target in about eleven seconds, on a dm-verity /dev/mapper/root, with sshd up.

The real cause of the red was this line, in this repo:

if incus console "${VM}" --show-log 2>/dev/null | grep -qaF "${EXPECT_CONSOLE}"; then

grep -q exits on the first match and closes the pipe; incus console then dies of SIGPIPE (141); set -o pipefail promotes that to a failed pipeline. The lane therefore reported failure precisely when it found the marker. The other lanes escape this only because they route through a console_log() helper that ends in || true. Fixed by capturing to a file and grepping the file, which is what the ISO lane always did.

Two lessons worth keeping: a lane that has never once been green is not evidence about the thing under test, it is evidence about the lane; and one plausible-sounding narrative ("pre-install artifact") is exactly how a harness bug acquires the appearance of a product question.

Note that /var ships as plain ext4 here, while snosi-install --encrypt-var produces a LUKS /var. The two paths genuinely produce different systems, so this lane and the install lane are not redundant.

Corrected 2026-08-10 and re-run as snosi-bootc-install-fkplf. The mechanics lane installed the current snow:mechanics image and booted it successfully:

bootc installed and verified: ghcr.io/frostyard/snow:mechanics (secureboot=false)
  backend=composefs
  booted=ok
  failedunits=0
  osrelease=snow-20260810050914
  rootfs=overlay

The old red result was a tier mismatch, not evidence that current bootc images could not install. The lane had aimed direct bootc install to-disk at ghcr.io/frostyard/snow:latest, a secureboot-capable=true assembly that must be installed by the external secure installer. The mechanics template now checks that label and rejects the combination as not applicable instead of building a misleading broken deployment. Snosi issues #504 and #505 were both closed as not planned after that diagnosis; the explicit --filesystem argument in the mechanics harness is deliberate.

There was also a real /dev/gpt-auto-root-luks failure later in the secure path, but it had a different root cause. Forky systemd 261 moved the GPT-auto udev links from 99-systemd.rules into 90-image-dissect.rules, which dracut did not include in the UKI initrd. snosi#520 ships that rule explicitly, and Snosi's secure artifact validation now rejects an initrd that cannot create the GPT-auto root link.

The secure lane proved that repair repeatedly before it was retired. Its last successful committed run, snosi-secure-install-auto-fwzbj, verified 18/18 assertions on 2026-08-10 with Secure Boot enforced, MOK enrolled, recovery available, and TPM unlock working. The lane — run-secure-install-tests, the external Dakota/bootc-installer/Fisherman path — was removed on 2026-08-12: firn is the single installer (core ADR-0027/0028), and the firn install matrix now owns secure-boot + encrypted bootc coverage. See the secure installer status and blocker history for the complete evidence and limits of the retired lane.

snosi images ship no incus guest agent, and a live ISO has no provisioned SSH key — so there is no obvious way to run a command inside a guest. systemd solves it: it reads credentials from SMBIOS type 11, and the well-known systemd.extra-unit.<name> credential defines an entire unit from thin air.

The lane passes qemu two credentials via raw.qemu — the unit to run, and a multi-user.target drop-in that pulls it in — and the guest executes it at boot with no cooperation from the image:

systemd[1]: Received regular credentials: systemd.extra-unit.snosi-qa-install.service, ...
systemd[1]: Acquired 2 regular credentials, 0 untrusted credentials.

Results come back on the serial console, which is the only channel that exists before a system is installed. The same mechanism carries the post-install assertions. The protocol is recorded in ADR-0005; how the lanes reach the host's incus daemon at all is ADR-0006.

Booting a published *-ab disk image with secureboot=true fails:

Verification failed: (0x1A) Security Violation

OVMF is right to refuse it. snosi signs its UKI with its own MOK, and nothing has enrolled that MOK into a fresh firmware's db. Enrollment is a step the installer performs — and snosi-install stages it as a one-time MokManager prompt at first boot, which no unattended run can answer.

So the native A/B install lane currently runs --skip-mok with secureboot=false. That covers everything the native installer builds, but leaves its signed boot chain uncovered end to end. Successful external bootc secure runs have enforced Secure Boot, but they do not prove the native A/B path. Two ways to close that remaining native gap:

1. Lab-side. Pre-seed the VM's OVMF variable store with the snosi MOK before first boot (e.g. virt-fw-vars against the per-instance qemu.nvram). No installer change; the lab simulates an operator who already enrolled the key.
2. Installer-side. An unattended enrollment path in snosi-install — for example a flag that enrolls directly into db when the firmware permits it, rather than staging a MokManager prompt.

(1) is less invasive and testable today; (2) is closer to what a real user does. This is a decision for the snosi maintainer, not the lab.

cayo skips the desktop scenarios by design — it is the headless server image, and the suite gates them on variant so one set of features runs unmodified across the whole family.

Enable or disable a lane by setting spec.suspend in its CronWorkflow and pushing. Doing it through git keeps the set of active lanes reviewable in history, and selfHeal would revert a kubectl patch anyway.

Defined in frostyard/testsuite.

smoke is implemented. system and sysext are declared in the pipeline's validation list but not yet populated — adding features to those directories in the testsuite repo is all that is needed to light them up.

lab/
├── argocd/
│   ├── application.yaml          # App: frostyard-lab       → argo/workflow-templates
│   ├── infra-application.yaml    # App: frostyard-lab-infra → manifests
│   └── argo-workflows-app.yaml   # App: argo-workflows      → upstream Helm chart
│
├── argo/
│   ├── workflow-templates/       # ← Argo CD auto-syncs these
│   │   ├── image-poller.yaml         digest compare → QA → persist
│   │   ├── snosi-qa-pipeline.yaml    validate suites → fan out lanes
│   │   └── run-container-tests.yaml  nested systemd boot + behave
│   └── snosi-smoke-test.yaml     # submit file: one-off manual run
│
├── manifests/                    # ← Argo CD auto-syncs these
│   ├── argo-rbac.yaml                argo ServiceAccount + Roles
│   ├── workflow-controller-configmap.yaml
│   ├── workflow-semaphores.yaml      cross-workflow concurrency caps
│   ├── image-polling-digests.yaml    digest state (values owned by cluster)
│   ├── image-poll-*.yaml             one CronWorkflow per lane
│   ├── namespaces.yaml
│   └── orphan-pod-gc.yaml
│
├── docs/                         # see docs/README.md for the full index
│   ├── adr/                      # repo-local decision records
│   ├── ops/bootstrap.md          # from-zero cluster setup
│   └── quality.md                # quality signals, evidence, and known gaps
├── policies/                     # executable agent-governance policy and checker
└── Justfile                      # operator wrappers

Run the recipes from the repository root. They use the current kubeconfig context; there is no separate Argo server endpoint because this lab disables the Argo Workflows server. Before operating the lab, verify that kubectl config current-context names the intended cluster and that kubectl get namespaces argo argocd succeeds with your current credentials.

Install these workstation clients:

Both kubectl and argo use the current kubeconfig context and need access to the argo namespace; the status, refresh, and bootstrap recipes also need access to argocd. The recipes do not select a context for you. Confirm the clients before relying on a wrapper:

just --version
kubectl version
argo version --client

Common operator commands and the client each wrapper invokes:

just status     # kubectl: Application sync/health + enabled lanes
just smoke      # kubectl: one-off smoke run against snow:latest
just qa IMAGE TAG SUITES VARIANT  # argo: submit an arbitrary QA run
just watch      # argo: watch the most recently submitted workflow
just runs       # kubectl: recent run history
just logs       # argo: follow the most recent workflow
just validate   # kubectl: server-side dry-run every YAML before pushing
just refresh    # kubectl: force Argo CD to re-read git now

Setting up a cluster from scratch: docs/ops/bootstrap.md.

Repository quality signals and their current limits: docs/quality.md.

The deny-by-default automated-contributor policy and its local validator: policies/.

Public aggregate dashboards and pull request metrics: docs/metrics/.

Copilot review feedback automation and its repository setup: docs/automated-review.md.

site/ is an Astro page published to GitHub Pages, showing per-lane status and recent run history: https://frostyard.github.io/lab/

The pipeline-results data flow deliberately has no link between GitHub and the cluster in either direction:

publish-results CronWorkflow (in cluster)
    │  reads the Argo API, regenerates site/src/data/runs.json
    ▼
git push main
    │
    ▼
.github/workflows/pages.yml → builds site/ → GitHub Pages

The collector reads the Kubernetes API rather than being wired into each lane, so a lane added tomorrow appears with no reporting change (ADR-0004; the unproven lane state it carries is ADR-0003). It skips the commit when only the generation timestamp moved, so an idle cluster does not push a commit every 30 minutes.

e2e/ holds the Playwright end-to-end suite. It builds the site and drives the same static output GitHub Pages serves, asserting that what runs.json contains is what the dashboard renders. Run it with just site-e2e; CI runs it via .github/workflows/e2e.yml.

The page is styled with the frostyard design system, following its Pilothouse dashboard language: ink surfaces, hairline separation, square corners, cold ice/sky accents, mono kickers. site/src/styles/tokens/ is copied verbatim from that repo — change tokens there and re-copy rather than patching them here, or the next copy silently reverts the edit. Because those tokens define no light palette, the page is dark-only by design.

publish-results ships suspended and needs a token that can push to this repo:

kubectl create secret generic github-token -n argo \
  --from-literal=token=<PAT with contents:write on frostyard/lab>

Then set spec.suspend: false in manifests/publish-results.yaml and push. Until that exists the page renders from whatever runs.json is committed.

Locally: just collect regenerates the data from the cluster, just site-dev serves the page.

• display-manager.service is not linked in the snow image. gdm.service ships with no [Install] section, so the systemctl enable gdm.service in snosi's postinst is a no-op and /etc/systemd/system/display-manager.service is absent from the built image. Whether GDM still starts on a real boot is unresolved — a container cannot answer it. Tracked for the VM lane.
• No result publication. Bluefin's lab publishes per-suite results back into the repo and renders them with Astro. Results here live in the workflow's output parameters and the pod logs only.
• No artifact storage. Argo needs a configured artifact repository to save output artifacts, and the lab has no object store. The VM lane's full serial console therefore goes to the workflow log rather than an artifact — 400 lines on failure, 40 on success. Standing up an object store would let the whole console and the behave results.json be retained per run.
• The native A/B signed boot chain is not covered end to end. Its install lane runs secureboot=false because of the MOK gap above. Successful bootc secure runs enforce Secure Boot, but that is a separate path; closing native A/B coverage is still a decision — see "Native A/B Secure Boot remains a separate gap".
• No A/B update or rollback coverage. The install lane proves a system gets built correctly; it does not yet stage a systemd-sysupdate run, switch slots, and boot the other side. That is the natural next lane and the machinery (SMBIOS credentials, console assertions) already exists.
• The install lane needs 8 GiB of guest RAM. snosi-install stages a UKI copy in /var/tmp, a tmpfs sized from guest memory; at 4 GiB the install fails partway with objcopy: ...[.initrd]: No space left on device. Worth knowing outside the lab — a real user on a low-memory machine hits the same wall, with the same unhelpful error.
• No registry pull-through cache. Every lane pulls from ghcr.io directly. Fine at three lanes on a 3-hour poll; revisit if lane count grows.

MIT