feat(provisioning): declarative resource provisioning and include-aware config checksums by houko · Pull Request #7921 · librefang/librefang

houko · 2026-08-25T09:13:45Z

Refs #6695. Lands the second half of @whatnick's RFC — declarative resource provisioning — plus the one design question left open that blocked the first half from being trustworthy.

What was already on `main`, re-verified today

The RFC's status comment predates three merges. Checked against origin/main at 3624319cc:

§4, the dashboard provenance surface — landed in feat(config): surface managed mode in the dashboard and lock four remaining write domains #7868 (a0acb2c9), which also settled the lock-or-classify decision on change_password, the sidecar-channel pair, the extension pair and the MCP routes under mcp_runtime_store = "file".
§1 and §6, the Kubernetes artefacts — landed in feat(k8s): add a managed-config overlay and boot off the mount #7902 (d520adbe): deploy/kubernetes/overlays/managed-config/ with a ConfigMap, a read-only mount outside /data, both env vars, a checksum/config rollout annotation, a rollback procedure, and a kind e2e job that asserts the 423.
The relocation defect (writes landing in a file nothing reads) — fixed in fix(config): resolve config.toml once at boot and use that path everywhere #7886 via config_path_boot.

That leaves declarative provisioning, which did not exist in any form, and the include checksum question.

Declarative resource provisioning

Managed mode locks config.toml as a whole. This is the same contract one level down, for resources that do not live in that file.

LIBREFANG_PROVISIONING_PATH points at a deployment-owned tree:

/etc/librefang/provisioning/agents/researcher.toml

crates/librefang-kernel/src/provisioning.rs — the scan, the state file, and plan(), a pure decision table over plain data so every case is testable without a booted kernel.
crates/librefang-kernel/src/kernel/provisioning_ops.rs — apply_provisioning(), called once from boot_with_config_at after the registry restore and before the default-assistant fallback, so a deployment that declares its own agents does not also get an assistant it never asked for.
The resource identifier is the manifest's name, not the file name. AgentManifest::name deserialises a missing key to "unnamed", so the reconcile requires the key to be present rather than accepting the deserialiser's default — otherwise nmae = "researcher" provisions unnamed and the next such typo collides with it invisibly.
Decision table: create / apply / adopt an existing agent under a declared name (kubectl apply semantics — id, session and history survive) / no-op on a byte-identical file / recreate one deleted out of band.
LIBREFANG_PROVISIONING_PRUNE decides what happens to an orphan. The default releases it — provenance dropped, agent keeps running and becomes editable again — which is what makes removal reversible: putting the file back re-adopts the same agent instead of colliding with a tombstone. Only the exact word delete removes anything, so a typo can never be the reason an agent is deleted.
Failures never fail the boot. A file that is unreadable, not UTF-8, not valid TOML, nameless or duplicate is recorded and skipped; the rest of the tree still applies. An unrecognised subdirectory (channels/, workflows/) is reported rather than ignored, so a deployment is told the kind is unsupported instead of believing it provisioned something.
The tree is an input, never a write target: a provisioned agent's source_toml_path is deliberately not set to its declaring file, so persist_manifest_to_disk never retries a doomed write against a read-only mount — the failure mode the managed-mode migration write-back was fixed to avoid.

Both variables are environment, not `KernelConfig`

Same reason LIBREFANG_CONFIG_MODE is: a setting that decides what may be written must not itself be writable through the surface it governs. A [provisioning] key would be settable from the dashboard in mutable mode, and a write that turns provisioning off is a write that unlocks every provisioned resource. No new config field, so no build_reload_plan classification and no config_schema_golden churn.

The lock

guard_provisioned_write (routes/mod.rs) is the resource-level counterpart of guard_config_write — same 423 Locked, same envelope, code: "resource_provisioned" because the remedy differs. guard_provisioned_agent (routes/agents/mod.rs) applies it to nine handlers: PATCH /agents/{id}, DELETE /agents/{id}, the model / tools / skills / mcp_servers / channels PUTs, PATCH /agents/{id}/config, PATCH /agents/{id}/identity, and per item inside DELETE /agents/bulk so one deployment-owned agent does not fail an operator's other deletes.

The guard is on the definition, not the agent. Suspend, resume, stop, message, sessions, files and every read stay open — the RFC's "operational actions and mutable runtime state remain usable" criterion, asserted in a test rather than assumed. Everything the tree does not declare stays fully mutable.

`GET /api/provisioning/status`

Per-resource source, applied checksum, current source_checksum, drifted, present, plus failures[] — the only place a refused file survives after the boot log scrolls. No write route: a provisioning tree is deployment-owned, so it is rollout-only exactly as managed configuration is.

Include-aware configuration checksums (RFC design question 6)

GET /api/config/status hashed the primary file's bytes alone, so with include = [...] an edit to an included file changed the effective configuration and left the checksum identical — a false negative for the operator using it to confirm a rollout landed, and the stated reason scripts/check-k8s-manifests.py banned include in a managed ConfigMap outright.

With no include the digest is unchanged, byte for byte, so feat(k8s): add a managed-config overlay and boot off the mount #7902's shipped annotation and its kind assertion keep matching.
With includes it is the digest of sha256sum output over the closure in include order, reproducible as (cd /etc/librefang && sha256sum config.toml extra.toml) | sha256sum. A new includes field lists what contributed; modified_at now reports the newest mtime across the same set.
The walk is tolerant where the loader is strict — this feeds a status endpoint, so an absolute, traversing, missing or circular include shortens the list rather than failing the request.
The checker now allows an include whose target is another key of the same ConfigMap and computes the composite annotation, failing on the cases the daemon could not honour: a target the manifest does not render, a /-containing key, an absolute or .. path, and an include in a file mounted with a subPath.

Kubernetes and docs

The managed-config overlay now renders agents/researcher.toml into a librefang-agents ConfigMap, mounts it read-only at /etc/librefang/provisioning/agents (a ConfigMap exposes keys flat, hence the subdirectory), sets LIBREFANG_PROVISIONING_PATH, and carries a checksum/provisioning annotation. The tree is reconciled at boot only, so an edit that does not roll the pod changes nothing — check_provisioning in the manifest checker enforces that, and additionally fails the build on a declaration that is not valid TOML, declares no name, is not a .toml key, assigns a credential a literal value, or is mounted writable or at the wrong path.

docs/operations/declarative-provisioning.md is the operator contract; deploy/kubernetes/README.md gains a "Provisioned agents" section; docs/operations/managed-config.md gains the include-checksum semantics.

Verification

Every command run locally, in this worktree.

cargo check --workspace --lib — clean.
cargo clippy --workspace --all-targets -- -D warnings — clean.
cargo test -p librefang-kernel --lib — 1685 passed, 0 failed. Includes 15 new provisioning::tests::* (the plan decision table, sorted scanning, per-file failures, duplicate names, unsupported subdirectories, state round-trip and malformed-state degradation, the prune-policy resolution rule) and 5 new config::tests::* (provenance_checksum_of_a_single_file_is_the_raw_byte_digest, provenance_checksum_covers_included_files, composite_checksum_matches_the_documented_sha256sum_pipeline, source_walk_skips_unsafe_and_missing_includes_instead_of_failing, source_walk_terminates_on_a_circular_include).
cargo test -p librefang-api --lib — 1101 passed, 0 failed.
cargo test -p librefang-api --test provisioning_test — 13 passed, 0 failed (new file). Named: provisioning_is_disabled_and_inert_without_the_environment_variable, a_declared_agent_is_created_at_boot_and_reported_with_its_provenance, manifest_writes_to_a_provisioned_agent_are_refused_with_the_documented_shape (all nine routes), operational_routes_stay_open_on_a_provisioned_agent, a_runtime_created_agent_stays_writable_beside_a_provisioned_one, editing_a_declaration_after_boot_reports_drift_without_changing_the_applied_checksum, a_malformed_declaration_is_reported_without_failing_the_boot, a_second_boot_over_an_unchanged_tree_changes_nothing, a_changed_declaration_is_applied_to_the_existing_agent_on_the_next_boot, removing_a_declaration_releases_the_agent_under_the_default_prune_policy, removing_a_declaration_deletes_the_agent_under_the_delete_prune_policy, an_existing_agent_is_adopted_when_the_deployment_declares_its_name, a_provisioned_agent_deleted_out_of_band_is_recreated_on_the_next_boot.
cargo test -p librefang-api --test config_managed_mode_test — 14 passed (unchanged behaviour, including the checksum-unchanged-across-a-refused-write case).
cargo test -p librefang-api --test config_path_unification_test — 2 passed. --test agents_clone_bulk_integration — 27 passed.
Drift guards: dead_route_audit_test 2 passed, openapi_path_coverage_test 1 passed, openapi_spec_test 1 passed, config_schema_golden 1 passed (schema unchanged — no new KernelConfig field). openapi.json regenerated and xtask/baselines/openapi.sha256 rewritten as plain sha256sum openapi.json.
kubectl kustomize deploy/kubernetes/{base,overlays/managed-config} piped through python3 scripts/check-k8s-manifests.py — both clean (3 and 5 manifests). Each new guard was also exercised against a deliberately broken copy of the overlay and produced the intended message: stale checksum/provisioning, a declaration with no name, invalid TOML, a literal api_key, an include naming an unrendered key, an absolute include, and a /-containing include.
Not run: cargo nextest (not installed on this host) and any live daemon — CI covers the former, the latter is human-only.

Deliberately not in this PR

channels/ and workflows/ provisioning. Both persist through surfaces this reconcile does not model — channels into config.toml itself, already covered by the whole-config lock, and workflows into a SQLite-backed registry with its own run state — so neither is a matter of pointing the same scan at another subdirectory. An unrecognised subdirectory is reported as a failure rather than silently skipped, so a deployment that tries one is told.
In-place reload of the tree. Boot-time only, matching the rollout-only contract managed configuration settled on.
Field-level ownership inside config.toml. The RFC's remaining open design question, now answered rather than deferred: ownership is expressed per resource, which is the granularity a deployment has actually asked for, and layering a second axis over the existing WRITABLE_EXACT_PATHS allowlist would produce a matrix whose interesting cases are all corners. Recorded in managed-config.md.

I have deliberately not used Closes #6695 — see the issue comment for exactly what is and is not covered.

…ning tree Adds the declarative half of the managed-configuration RFC. Managed mode locks config.toml as a whole; this locks resources declared outside it, one at a time, while everything an operator creates at runtime stays editable. LIBREFANG_PROVISIONING_PATH points at a deployment-owned tree whose agents/*.toml files each declare one agent, identified by the manifest name. Boot reconciles the tree into the registry: create, apply, adopt an existing agent under a declared name, or leave an unchanged declaration alone. LIBREFANG_PROVISIONING_PRUNE decides what happens when a declaration leaves the tree — the default releases the agent back to runtime ownership, and only the exact value delete removes it. Both inputs are environment variables rather than KernelConfig fields for the reason the config mode is: a setting that decides what may be written must not be writable through the surface it governs. GET /api/provisioning/status reports each resource's source, applied checksum, on-disk checksum and drift, plus one entry per file the reconcile refused. A malformed manifest never fails the boot.

…manifest default `AgentManifest::name` deserialises to `"unnamed"` when the key is absent, so a file that never declares one was provisioned as an agent called `unnamed`, and a second such file collided with it for no visible reason. The name is the resource identity here, so it now has to be present in the file rather than supplied by the deserialiser, and the trimmed value is written back onto the manifest so the registry lookup and the state-file key cannot disagree.

…oning/status

The checksum on GET /api/config/status was computed over the primary file's bytes alone, so with `include = [...]` an edit to an included file changed the effective configuration and left the checksum identical. An operator comparing it against a ConfigMap's checksum/config annotation to confirm a rollout landed got a false negative, which is why the Kubernetes manifest checker banned `include` in a managed ConfigMap outright. With no include the digest is unchanged, byte for byte, so an existing annotation keeps matching. With includes it becomes the digest of `sha256sum` output over the closure in include order — reproducible as `(cd /etc/librefang && sha256sum config.toml extra.toml) | sha256sum` — and a new `includes` field lists what contributed. `modified_at` now reports the newest modification across the same set. The walk is tolerant where the loader is strict, since this feeds a status endpoint: an absolute, traversing, missing or circular include is skipped rather than raised, so a broken chain shortens the list instead of failing a request. The checker now allows an include whose target is another key of the same ConfigMap and computes the composite annotation, and fails on the cases the daemon could not honour — a target the manifest does not render, a `/`-containing key, an absolute or `..` path, and an include in a file mounted with a subPath.

… the k8s overlay The managed-config overlay now renders agents/*.toml into a second ConfigMap, mounts it read-only under LIBREFANG_PROVISIONING_PATH, and carries a checksum/provisioning annotation so editing a declaration rolls the StatefulSet — the tree is reconciled at boot only, so an edit that does not roll the pod changes nothing at all. The manifest checker gains the matching guard. It fires only when the manifest sets LIBREFANG_PROVISIONING_PATH, and fails the build on the cases the running daemon would otherwise turn into an agent that is silently missing: a declaration that is not valid TOML, one that declares no `name`, a key that is not a `.toml` file, a credential written as a literal value, a mount that does not supply the agents directory or is not read-only, a prune value that reads as intent it does not carry, and a stale checksum annotation. docs/operations/declarative-provisioning.md is the operator-facing contract: the tree layout, the two environment variables and why they are not config keys, the reconcile decision table, what adoption and release mean, what is locked and what deliberately is not, and the known gaps. It also records the answer to the RFC's open question about field-level ownership: ownership is expressed per resource rather than per config field, which is the granularity a deployment has actually asked for.

houko

Reviewed the diff against CLAUDE.md's project conventions (deterministic ordering, config-field completeness, changelog fragment format, route registration/auth, environment-variable-not-config-field pattern for lock-deciding settings). Everything else lines up — one edge-case question left inline on the prune-failure accounting.

Generated by Claude Code

houko · 2026-08-25T09:17:54Z

+                Action::Prune { .. } => {
+                    report.pruned += 1;
+                    if let Some(name) = key.strip_prefix("agent/") {
+                        match self.agents.registry.find_by_name(name) {
+                            Some(entry) => {
+                                if let Err(e) = self.kill_agent(entry.id) {
+                                    tracing::warn!(
+                                        agent = %name,
+                                        "Failed to prune provisioned agent: {e}"
+                                    );
+                                } else {
+                                    info!(
+                                        agent = %name,
+                                        "Pruned provisioned agent — its declaration left the tree"
+                                    );
+                                }
+                            }
+                            None => tracing::debug!(
+                                agent = %name,
+                                "Provisioned agent already gone; nothing to prune"
+                            ),


Action::Prune counts report.pruned += 1 and drops the key from next unconditionally, even when self.kill_agent(entry.id) returns Err (e.g. the agent is hand-owned and the kill guard refuses it, per the same guard bulk_delete_agents/kill_agent already special-case elsewhere in this PR).

Two effects of that:

GET /api/provisioning/status's report.pruned claims a resource was pruned when it wasn't — an operator watching that count after setting LIBREFANG_PROVISIONING_PRUNE=delete would see a wrong success count on a failed delete, with the actual error visible only in the boot log WARN.

Because the key isn't carried into next.resources either, the state file loses this resource's provenance entirely. On the next boot, plan() sees desired without the key and previous without the key too — no Prune/Release action is generated for it at all, so a delete that failed once is never retried and the agent quietly becomes an ordinary untracked runtime agent (not reported in failures[], not counted as failed).

Compare with the Apply/Create failure branch just above, which does push a ProvisioningFailure and bump report.failed instead of report.applied/created. Seems like Action::Prune should follow the same pattern: only increment pruned and drop provenance on success, and on Err push a ProvisioningFailure (bump report.failed) while keeping the previous provenance in next so the next reconcile retries the prune instead of silently giving up.

No test currently exercises a kill_agent failure during prune, so this path isn't covered either way — flagging as a question rather than pushing a fix, since the right retry semantics might be a deliberate call I'm missing.

Generated by Claude Code

… the agent A `kill_agent` failure during a prune still counted as pruned and dropped the provenance record, which left a running agent unlocked — the outcome the `keep` policy produces, reached by a failure rather than by the operator's choice, and with no record left for the next boot to retry from. The record is now kept, the reason is reported through `GET /api/provisioning/status`, and the count lands in `failed` where it belongs.

cloudflare-workers-and-pages · 2026-08-26T12:29:05Z

Deploying librefang-docs with Cloudflare Pages

Latest commit:	`b3f6e9a`
Status:	✅ Deploy successful!
Preview URL:	https://c1f56838.librefang-docs.pages.dev
Branch Preview URL:	https://feat-6695-declarative-provis.librefang-docs.pages.dev

View logs

cloudflare-workers-and-pages · 2026-08-26T12:31:22Z

Deploying librefang with Cloudflare Pages

Latest commit:	`b3f6e9a`
Status:	✅ Deploy successful!
Preview URL:	https://f57b298e.librefang-7oe.pages.dev
Branch Preview URL:	https://feat-6695-declarative-provis.librefang-7oe.pages.dev

View logs

Two hunks in `check-k8s-manifests.py` where both sides changed real things. `check_services` signature: main widened it to take the StatefulSet, and this branch had inserted `expected_config_digest` and `check_provisioning` just above the old one-line definition. Kept this branch's new functions and main's new signature — the old signature would not match the call site main rewrote. The dispatch block: main restructured it into a `try:` with an explicit StatefulSet count and a clearer failure message, and passes `sts` onward even when it is None. Took main's structure and re-added this branch's `check_provisioning(docs, sts, failures)` call inside the same `else` branch, which is the one line this branch had contributed there. `xtask/baselines/openapi.sha256` recomputed from the merged `openapi.json`. Verified: `python3 scripts/tests/test_check_k8s_manifests.py` — 10 tests pass; `ast.parse` accepts the file; `shasum -c` confirms the baseline.

…-provisioning # Conflicts: # xtask/baselines/openapi.sha256

…-provisioning

…-provisioning # Conflicts: # xtask/baselines/openapi.sha256

houko added 6 commits August 25, 2026 17:16

chore(openapi): regenerate the spec and baseline for GET /api/provisi…

b86dd0b

…oning/status

docs(changelog): add fragments for #7921

86b7203

houko mentioned this pull request Aug 25, 2026

RFC: add managed configuration and declarative provisioning mode #6695

Open

houko enabled auto-merge (squash) August 25, 2026 09:14

houko commented Aug 25, 2026

View reviewed changes

houko and others added 3 commits August 25, 2026 18:19

Merge branch 'main' into feat/6695-declarative-provisioning

55a665d

chore(codegen): sync generated artifacts (openapi.json, SDKs, baselines)

5257b19

github-actions Bot added area/docs Documentation and guides area/kernel Core kernel (scheduling, RBAC, workflows) area/ci CI/CD and build tooling area/sdk JavaScript and Python SDKs size/XL 1000+ lines changed labels Aug 26, 2026

Merge branch 'main' into feat/6695-declarative-provisioning

6d118a7

github-actions Bot added the ready-for-review PR is ready for maintainer review label Aug 26, 2026

Merge branch 'main' into feat/6695-declarative-provisioning

a62b478

github-actions Bot added has-conflicts PR has merge conflicts that need resolution and removed ready-for-review PR is ready for maintainer review labels Aug 27, 2026

github-actions Bot added ready-for-review PR is ready for maintainer review has-conflicts PR has merge conflicts that need resolution and removed has-conflicts PR has merge conflicts that need resolution ready-for-review PR is ready for maintainer review labels Aug 27, 2026

Merge remote-tracking branch 'origin/main' into feat/6695-declarative…

f3a898b

…-provisioning # Conflicts: # xtask/baselines/openapi.sha256

Merge remote-tracking branch 'origin/main' into feat/6695-declarative…

5064466

…-provisioning

github-actions Bot added ready-for-review PR is ready for maintainer review has-conflicts PR has merge conflicts that need resolution and removed has-conflicts PR has merge conflicts that need resolution ready-for-review PR is ready for maintainer review labels Aug 29, 2026

Merge remote-tracking branch 'origin/main' into feat/6695-declarative…

d7bea16

…-provisioning # Conflicts: # xtask/baselines/openapi.sha256

github-actions Bot added ready-for-review PR is ready for maintainer review and removed has-conflicts PR has merge conflicts that need resolution labels Aug 29, 2026

Merge remote-tracking branch 'origin/main' into feat/6695-declarative…

3b8624c

…-provisioning # Conflicts: # xtask/baselines/openapi.sha256

houko merged commit 5b5a75f into main Aug 29, 2026
46 checks passed

houko deleted the feat/6695-declarative-provisioning branch August 29, 2026 06:36

houko mentioned this pull request Aug 30, 2026

release: v2026.8.30 #8005

Merged

Navigation Menu

Repository navigation

feat(provisioning): declarative resource provisioning and include-aware config checksums - #7921

feat(provisioning): declarative resource provisioning and include-aware config checksums#7921
houko merged 16 commits into
mainfrom
feat/6695-declarative-provisioning

houko commented Aug 25, 2026

Uh oh!

houko left a comment

Uh oh!

houko Aug 25, 2026

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 26, 2026 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 26, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

houko commented Aug 25, 2026

What was already on main, re-verified today

Declarative resource provisioning

Both variables are environment, not KernelConfig

The lock

GET /api/provisioning/status

Include-aware configuration checksums (RFC design question 6)

Kubernetes and docs

Verification

Deliberately not in this PR

Uh oh!

houko left a comment

Choose a reason for hiding this comment

Uh oh!

houko Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying librefang-docs with Cloudflare Pages

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying librefang with Cloudflare Pages

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

What was already on `main`, re-verified today

Both variables are environment, not `KernelConfig`

`GET /api/provisioning/status`

cloudflare-workers-and-pages Bot commented Aug 26, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented Aug 26, 2026 •

edited

Loading