feat(ci): Linux verification job for the declared dev services by inkatze · Pull Request #103 · inkatze/dotfiles

inkatze · 2026-08-07T02:24:09Z

Task 6 of specs/dev-services, executed from
specs/dev-services/kickoff-brief.md.
REQs: REQ-E1.1, REQ-E1.2, REQ-E1.3, REQ-E1.4, REQ-C1.1 · D-7.

Summary

The declared dev-services layer had no Linux coverage. The services entry in
the macOS matrix proves the negative (a Mac provisions none of it) and
structurally cannot prove the positive, having neither apt nor systemd. This
adds a separate job on a pinned ubuntu-26.04 runner that provisions by tag,
asserts the result against systemd and a socket, re-runs, and gates the re-run
on convergence.

It found two real bugs on its first run. Neither role nor harness had ever
executed on a Linux host before — the macOS matrix skips the whole lifecycle
and Task 7's host convergence is still pending — and both used psql --command= with :'var' interpolation, which psql does not perform on a -c
string. See What the job caught below.

A separate job, not a matrix entry. The matrix sets runs-on once at job
level, so it is macOS-only by construction; taking a per-entry runner would
mean editing the step definitions every pre-existing entry shares, which is
exactly what REQ-E1.4 forbids. The workflow change is a pure append —
diff against origin/main reports 120a121,235 with no c or d hunks and
zero removed lines, and scripts/services-declaration-test.sh now asserts that
additions-only rule on every run.

Verification (all observed, none assumed)

`Done when:` clause	Evidence
Every declared service running, enabled and reachable at its declared address and port	`Assert every declared service is running, enabled and reachable` — green on run 31143015346
A second run reporting any changed task fails the job, verified by deliberate mutation	Commit `1e1b5fc` added an always-changed task; the job failed at `Assert the second run converges` with `changed=1 task(s)` and skipped the steps after it. Reverted in `75d6e13`; green again on the next run
The job declares no secrets	No `secrets` context anywhere in the job, and it blanks the workflow-level `GITHUB_TOKEN` — the CI log shows `GITHUB_TOKEN:` empty in every step's env
The existing macOS matrix jobs are unmodified and still pass	Unmodified structurally (pure append, asserted by test). Matrix results on this PR

Implementation notes

scripts/ansible-idempotency-gate.sh (new) reads a PLAY RECAP and
treats ok=0 as a failure. Task 2's convergence found that changed=0
reads identically for a run that converged and for one that selected no tasks
at all, so the matrix's inline grep changed= cannot tell a working gate from
a broken one. A missing recap refuses outright rather than passing.
scripts/dev-services-runtime-test.sh (new) asks systemd and a socket,
from outside the run, whether every declared service is active, enabled
and accepting connections at its declared address. Driven entirely from
roles/services/defaults/main.yml, so it names no service and a newly
declared one is covered without editing it. enabled is asserted separately
from active because the two come apart precisely in the case REQ-C1.2 is
about: started by hand, gone after a reboot.
The two are a pair. The gate proves the run stopped changing things; the
runtime harness proves there is something there to have stopped changing.
Neither separates "converged" from "silently stopped provisioning" alone.
REQ-E1.3 is executable, not just reviewed. The job references no secrets
context and blanks GITHUB_TOKEN for itself, so a later edit that adds an
authenticated fetch fails outright instead of passing on an ambient token. It
calls scripts/playbook.sh directly rather than through mise for the same
reason (mise would resolve pinned tools through GitHub's API for tools the
provisioning never calls); a test asserts the two commands stay identical.
The host alias is set explicitly (DOTFILES_HOST: server), so the log
reads Running on host: server rather than falling back to the macOS alias
and warning about it.
The pin is checked at runtime. The job reads /etc/os-release back and
fails if the label no longer means 26.04.
Task 3's postgresql-access-test.sh is wired in (its own header names this
job as a caller). Task 4's valkey-client-compat-test.sh deliberately is
not: it needs an Elixir toolchain and network, and its header assigns the
in-CI form of the loopback assertions to this job — which is what the runtime
harness does.

What the job caught

Both are the same bug from the same cause, in code this PR does not otherwise
touch. psql hands a --command string to the server as a single request
without parsing it, so :'var' and :"var" are never interpolated there
(Postgres 18 psql docs; interpolation happens on stdin and -f input).

roles/services/tasks/postgresql.yml (Task 3) — both statements failed
with syntax error at or near ":", so provisioning died on its fourth task.
scripts/postgresql-access-test.sh (Task 3) — the drop-confirmation
query failed the same way, and reported it as '…' survived the drop when
the drop had actually succeeded.

Both moved to stdin, keeping --set as the only route the name takes — pasting
it into the SQL would also have worked and is exactly the shape those files'
own notes refuse. Fixing them is outside Task 6's deliverable and was applied
here because Task 6's Done when: is unreachable without it: the job cannot
assert a converged second run against a play that fails partway.

Tests

test-spec.md calls REQ-E1.1 "the job's own successful run", so the job is the
verification. What was written first, and confirmed failing for the right
reason before implementation, is the job-definition and gate coverage in
scripts/services-declaration-test.sh:

Job definition: the release pin (not -latest, and D-7's release), the
runtime release check agreeing with runs-on, the explicit host alias, no
secrets reference and a blanked token, two provisioning runs, parity with
the mise run services task, and all three harnesses wired.
Gate fixtures: converged, changed, never-ran (ok=0), failed, coloured,
and no-recap. The coloured case was confirmed to refuse (exit 2) with the
stripping removed and pass with it, so it bites rather than decorates.
Runtime harness: asserted to report visibly rather than pass vacuously
when the units are absent, the same posture the access harness is held to.

Wider suite green locally: mise run lint (yamllint 0 failures; ansible-lint
profile production passed; ansible-playbook --syntax-check), shellcheck --severity=warning over all scripts, gitleaks clean,
scripts/services-declaration-test.sh all assertions passed.

Audit record

Lens coverage

Lens	Findings	Notes
Correctness, logic, edge cases	4	`needs: lint` coupled the Linux job to the scarce macOS tier; GNU-only escape stripping was a silent no-op on BSD sed; two `psql --command` interpolation failures found by the job itself
Security	none	The job adds no credential path and removes one (blanked token); no untrusted input — the declaration and recap are repo/CI-produced. The TCP probe passes address and port as arguments rather than concatenating them into a `bash -c` string
Error handling and failure modes	2	A refusal printed an empty reason (python's message went to stderr); a malformed `mise.toml` would end the test in a traceback
Performance	none	Two apt-backed playbook runs inside a 30-minute timeout; the whole job runs in ~65s
Concurrency / state	none	Steps are strictly sequential; the only shared state is `second-run.log` in the job's own workspace
Naming, readability, structure	none	New files; no pre-existing structure worsened
Documentation	1	The gate's pass line did not say how much work it was calling converged
Tests / verification	1	Recap fixtures omitted `rescued`/`ignored`, and no fixture exercised the escape stripping
Cross-file consistency	none	Verified against the headers of `postgresql-access-test.sh` and `valkey-client-compat-test.sh`, both of which assign work to this job; both now consistent

Auto-applicable

Finding	Rule	Commit
none	—	No finding was both tool-grounded and mechanical; shellcheck, yamllint and ansible-lint were clean throughout

Agent-resolvable

Finding	Evidence	Commit
none	—	The findings with genuine failing-then-passing evidence (the coloured recap, both psql bugs) each also carry a second change or sit outside this task's deliverable, so they routed to sign-off rather than claiming this bucket

Needs sign-off

#	Finding	Fix applied	Commit
1	`needs: lint` put `macos-latest` back on the Linux job's critical path, contradicting the job's own resilience rationale	Dropped `needs:`; trade written down in the comment	`200daf1`
2	Escape stripping used GNU sed's `\x1B`, a literal `x` to BSD sed, so the strip silently no-opped on macOS — and the gate's pass line did not report the `ok=` tally, hiding a facts-only run	Real ESC byte via `$'...'`; pass line reports per-host `ok=`; fixtures gained the full field set and a coloured case. Also carries the `tomllib.TOMLDecodeError` catch (bundled by an editing slip; two findings, one commit)	`410c02e`
3	The declaration-parse refusal printed an empty reason, python's message having gone to stderr	Folded stderr into the capture	`4fe36ca`
4	`roles/services/tasks/postgresql.yml` used `psql --command` with `:'rolname'`, which psql never interpolates — provisioning failed on Linux, always	Both statements moved to stdin	`b9fa1b9`
5	`scripts/postgresql-access-test.sh` had the same bug, and reported a successful drop as a failed one	Query moved to stdin	`137d711`

Items 4 and 5 touch Task 3's work rather than Task 6's. They are the reason
this bucket is worth reading closely.

Needs human judgment

Finding	Options
none	No fork survived the resolution ladder

Declined log

Finding	Rationale
`test-spec.md`'s REQ-E1.3 entry calls the no-secrets pin "weak — a review confirms what the definition says today, and nothing prevents a later edit from adding either", and names a workflow-linting rule as the stronger form that is "out of proportion for this bundle"	Now conservative rather than wrong: `services-declaration-test.sh` asserts no `secrets` reference and a blanked token, so an edit adding one does fail. Amending is expression-only and permitted, but it would churn the spec anchor mid-bundle while a sibling worker is in flight, for a caveat that under-credits coverage rather than misleading anyone into unsafe action. Flagged for the human to fold
A GitHub Actions schema linter (`actionlint`) — yamllint validates the workflow as YAML and nothing validates it as Actions	Adding a pinned tool is outside Task 6's scope. Recorded as an observation (`specs/_observations/entries/2026-08-06-workflow-lint-gap-8abcb94a.md`)
`ubuntu-26.04` is a public-preview runner label, not GA	D-7's reasoning is unaffected and the pin stands; preview status only explains flakiness the provisioning did not cause. Recorded as an observation (`…2026-08-06-ci-runner-preview-75ba0b2e.md`)

Pending sign-off

1. ci(services): decouple the Linux job from the macOS lint gate — 200daf1
2. fix(scripts): read a coloured recap on both platforms, and say how much converged — 410c02e
3. fix(scripts): name the parse failure the refusal reports — 4fe36ca
4. fix(services): interpolate the role name psql never expanded — b9fa1b9
5. fix(scripts): interpolate the scratch database name psql never expanded — 137d711

Approve an item by leaving its commit in place; reject with git revert <sha>.

Note: commit 410c02e's body has a few mangled apostrophes, from quoting in
the tooling that wrote it. Left as-is rather than corrected, since history is
append-only here.

A separate job on a pinned ubuntu-26.04 runner, not an entry in the macOS matrix: that matrix sets `runs-on` once at job level, so taking a per-entry runner would mean editing the step definitions every pre-existing entry shares, which REQ-E1.4 forbids. The workflow change is a pure append (`120a121,227`, no `c` or `d` hunks, zero removed lines). The job provisions by tag, asserts the result, re-runs, and gates the re-run on convergence. Two new harnesses carry the assertions: - `scripts/dev-services-runtime-test.sh` asks systemd and a socket, from outside the run, whether every declared service is active, enabled and accepting connections at its declared address. Driven from the declaration, so it names no service. - `scripts/ansible-idempotency-gate.sh` reads a PLAY RECAP and treats `ok=0` as a failure, not a pass. Task 2 found that `changed=0` reads identically for a converged run and for one that selected no tasks at all, so the matrix inline grep cannot tell a converged gate from a broken one. Read with the runtime harness, the pair separates the two. REQ-E1.3 is made executable rather than reviewed: the job blanks the workflow-level GITHUB_TOKEN for itself, so it runs with no credential in its environment and calls the playbook wrapper directly instead of through mise, whose tool resolution would reach the GitHub API for tools the provisioning never calls. `scripts/services-declaration-test.sh` grows assertions over the job definition (the pin, its runtime release check, the explicit host alias, the absence of secrets, the two runs, and that all three harnesses are wired), plus recap fixtures for the gate covering converged, changed, never-ran, failed and no-recap. Planwright-Task: dev-services/6

Surfaced while wiring Task 6s CI job. D-7s pin is unaffected; the note records that the image status can explain flakiness the provisioning did not cause. Planwright-Task: dev-services/6

…g-sign-off] The job carried `needs: lint`, copied from the matrix, while its own comment argued it runs on Ubuntu so a macOS runner shortage cannot take the Linux gate down. `lint` is `runs-on: macos-latest`, so the dependency put the scarce tier straight back on the critical path and spent the argument. Dropped, with the trade written down: a lint failure now burns an Ubuntu runner before anyone notices it, and still surfaces -- from the playbook instead of from the linter. Pending sign-off because it is CI configuration and because "no `needs:`" is the kind of asymmetry with the job above that should be a decision rather than an omission. Planwright-Task: dev-services/6

…ch converged [pending-sign-off] Two things the gate got away with because every fixture was plain text. The escape-stripping expression used `\x1B`, which is a GNU sed extension; BSD sed reads it as a literal `x`, so on macOS it matched nothing and the strip silently became a no-op. A coloured `PLAY RECAP` does not start its line, so the gate would refuse a run that converged fine. Replaced with a real ESC byte from bash `$q...q`, which both seds accept. The new `gate-coloured` fixture colours the header specifically -- confirmed to refuse (exit 2) with the strip removed and to pass with it, so the case bites rather than decorating. The pass line now reports each host`s `ok=` tally. `ok=1` -- a play that gathered facts and skipped everything else -- satisfies a changed=0 gate and is not what anyone means by provisioned. Ruling that out is the runtime harness`s job, not arithmetic over a recap; printing the number is what makes it visible in the log either way. Fixtures also grew ansible`s full field set (`rescued`, `ignored`), since the gate reads fields by name and a half-populated fixture would not notice if that stopped. Planwright-Task: dev-services/6

…n-off] The declaration is read through a python heredoc whose error path is sys.exit("..."), which writes to stderr. The capture took stdout only, so a malformed declaration produced "could not read the declaration:" followed by nothing, with the real explanation stranded on an earlier line. Folded stderr into the capture; python writes nothing there on success. Planwright-Task: dev-services/6

Surfaced by Task 6s review pass. yamllint validates the workflow as YAML and nothing validates it as Actions. Planwright-Task: dev-services/6

…-sign-off] Task 6s CI job ran roles/services/tasks/postgresql.yml for the first time on a Linux host -- nothing before it ever had, since the macOS matrix skips the whole lifecycle and Task 7s host convergence is still pending -- and both tasks failed: ERROR: syntax error at or near ":" LINE 1: SELECT 1 FROM pg_roles WHERE rolname = :'"rolname"' psql sends a --command string to the server as a single request without parsing it, so its :'"var"' and :"var" interpolations never fire there; the server receives the literal text. Interpolation is performed on stdin and -f input (Postgres 18 psql docs, confirmed by the failure above). Both queries move to the command modules stdin, keeping --set as the only route the account name takes. Pasting it into the SQL would have worked too and is exactly the shape the files own note refuses. Outside Task 6s deliverable, and applied here because Task 6s Done-when is unreachable without it: the job cannot assert a converged second run against a play that fails on its fourth task. This is the job doing precisely what it was added to do, on its first run. Planwright-Task: dev-services/6

…ed [pending-sign-off] The same --command interpolation bug as the previous commit, in the harness rather than the role. The drop-confirmation query was the only call here carrying a psql variable, so it was the only one that failed: FAIL[scratch-drop]: survived the drop (pg_database reports ERROR: syntax error at or near ":") Misleading twice over -- the drop had in fact succeeded, and the assertion reported the opposite. Moved to stdin, where psql does interpolate. Both instances surfaced on Task 6s CI job, the first thing ever to run either file on Linux. Planwright-Task: dev-services/6

Task 6s Done-when requires the idempotency gate be seen to fail rather than assumed to work, because a gate that has never gone red is indistinguishable from one that cannot. This adds a task that reports changed on every run; the next commit reverts it. The CI run on this commit is the evidence. Planwright-Task: dev-services/6

…te mutation" This reverts commit 1e1b5fc. The evidence is recorded. On 1e1b5fc the Linux job failed at "Assert the second run converges" with: server : ok=9 changed=1 unreachable=0 failed=0 skipped=10 FAIL: the run did not converge: server: changed 1 task(s) The two steps after it were skipped, so the gate is load-bearing rather than advisory. Task 6s Done-when asked for the failure to be observed rather than assumed; it has been. Planwright-Task: dev-services/6

Three of Task 6s four Done-when clauses are verified by observation, including the deliberate-mutation proof the gate demanded. The fourth -- that the pre-existing macOS matrix still passes -- is stalled on runner capacity rather than on anything in the branch: ten test jobs queued over an hour with none allocated, while lint and the new Linux job both passed. Recorded rather than left in the PR body, since it is a bundle-level fact: the same shortage stalled the gate twice today, which is the concrete argument for putting the new job on Ubuntu. Planwright-Task: dev-services/6

Task 6 (#103) merged after this branch was cut. Both branches appended a reference bullet to the same position in tasks.md's ## Awaiting input, which git cannot order for itself; the merge base has neither, so both sides are purely additive. Resolved by keeping both, Task 5 then Task 6. They name different tasks, so both are legal under spec-format's one-bullet-per-task rule, and dropping either would discard a recorded operator action. No other change taken from either side: every remaining file auto-merged, and the content anchor is unchanged at fecce5e, since reference bullets are outside the canonical task-definition extraction. Planwright-Task: dev-services/5

* fix(services): converge the database role's attributes, not just its existence The existing pair guards on existence alone. A role that already exists without CREATEDB or LOGIN was left exactly as found: the existence query returns 1, the creation is skipped, the run reports zero changed tasks, and REQ-A1.4 is unmet. A green run with an unmet requirement is the failure shape worth two extra tasks. It was also invisible to the verification this bundle planned. test-spec.md pins REQ-C1.2 to Task 6's idempotency re-run, and that second pass runs against a role this file created one task earlier, which by construction already has both attributes. On a fresh host nothing is wrong; the case is a host where someone made a same-named role by hand, which is precisely REQ-C1.2's "already provisioned" condition. This is what community.postgresql's postgresql_user does internally and the reason it is idempotent in the convergent sense rather than the create-once one. D-6 keeps that collection out, so the behaviour is spelled out here instead. Placed after the creation so it reads post-create state; ordered the other way it would predict an ALTER on every fresh host. Grants nothing beyond what the creation already asks for: same two attributes, no superuser, no role membership. Verified against the running server rather than by reading. The query returns `1` for an account holding both attributes and empty for pg_read_all_data, a predefined role with no LOGIN, so the guard is shown to discriminate in both directions rather than merely to pass. Both exits are 0, which is why the `when:` compares stdout and not rc. Claude-Session: https://claude.ai/code/session_01F8oKrXASMQSXMVjSvuJpn9 * feat(ci): run the stale-declaration scan in the lint job specs/dev-services REQ-B1.5 is pinned `[test]` in test-spec.md, which spec-format defines as "runs in the repo's CI". Task 5 wrote the script and mutation-tested every assertion, but wiring it edits this file, which is a hard-disqualifier zone and something that task's own REQs did not call for, so it recorded the step instead of applying it. The pin has been inaccurate since: the script existed and nothing ran it. The lint job rather than services (linux): the scan reads tracked files and needs no provisioned service, and lint already gates the whole matrix, so a stale declaration now fails before anything spends a runner. Additions only, so REQ-E1.4 is untouched. Its three dependencies are already present in that job and were checked rather than assumed: git from checkout, python3 from setup-python, and PyYAML transitively from the `pip3 install ansible` step (the script exits early on `import yaml`). The job runs on macos-latest, so the script was also read for GNU-isms — the sed calls are portable and the scanning is done in python3. Claude-Session: https://claude.ai/code/session_01F8oKrXASMQSXMVjSvuJpn9 * docs(spec): clear the resolved dev-services awaiting-input entries Four of the five entries no longer describe anything open, and a queue that lists resolved work is the same defect this bundle's Task 5 existed to remove: a tracked statement that stopped being true. Task 2 Entirely about CI creating no workflow run, on the exhausted- macOS-minutes hypothesis. That was withdrawn — the outage was runner scarcity and cleared on its own. Task 3's entry already recorded the correction. Task 3 Both halves resolved. The runtime clause is now verified in CI rather than pending on the host: the services (linux) job runs scripts/postgresql-access-test.sh and passes on main. The ALTER ROLE finding it recorded is applied in this branch. Task 5 The REQ-B1.5 [test] wiring it recorded is applied in this branch, so the operator action it asks for is done. Task 6 Asks for a re-read of `gh pr checks 103` once macOS capacity frees. #103 is merged. The osx and upgrade jobs are still red, but for the standing 30-minute Brewfile timeout that reproduces on main and has its own observation entry, not for anything in that task. Task 1 stays: it needs ~/.config/dotfiles/private-identifiers, which is machine-local by design and cannot be supplied from here. spec-validate passes (0 errors, 0 warnings) on the trimmed bundle. Claude-Session: https://claude.ai/code/session_01F8oKrXASMQSXMVjSvuJpn9

inkatze added 10 commits August 6, 2026 19:00

chore(observations): ubuntu-26.04 runner label is preview, not GA

547a600

Surfaced while wiring Task 6s CI job. D-7s pin is unaffected; the note records that the image status can explain flakiness the provisioning did not cause. Planwright-Task: dev-services/6

chore(observations): no Actions schema linter for the workflow

8add8a8

Surfaced by Task 6s review pass. yamllint validates the workflow as YAML and nothing validates it as Actions. Planwright-Task: dev-services/6

inkatze mentioned this pull request Aug 7, 2026

docs(services): correct the stale out-of-scope service declarations #104

Merged

1 task

inkatze marked this pull request as ready for review August 7, 2026 16:08

inkatze merged commit 2c763a6 into main Aug 7, 2026
10 of 12 checks passed

inkatze deleted the planwright/dev-services/task-6 branch August 7, 2026 16:09

Navigation Menu

feat(ci): Linux verification job for the declared dev services - #103

feat(ci): Linux verification job for the declared dev services#103
inkatze merged 11 commits into
mainfrom
planwright/dev-services/task-6

inkatze commented Aug 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

inkatze commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Verification (all observed, none assumed)

Implementation notes

What the job caught

Tests

Audit record

Lens coverage

Auto-applicable

Agent-resolvable

Needs sign-off

Needs human judgment

Declined log

Pending sign-off

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

inkatze commented Aug 7, 2026 •

edited

Loading