docs(operations): document the sanctioned binary-downgrade recovery - #8148

The procedure asserts unconditionally that unzipping the archive "returns user_version to the pre-upgrade value", but that only holds when the SQLite databases actually live under <home_dir>/data/.

Both locations are operator-settable:

• KernelConfig::data_dir (crates/librefang-types/src/config/types.rs:3470, default home_dir.join("data") at types.rs:6835) is a documented config.toml field (docs/src/app/configuration/page.mdx:144), and boot.rs:379-383 resolves the memory DB as config.data_dir.join("librefang.db").
• [memory] sqlite_path (types.rs:7401) overrides the file path outright.

The backup side follows neither: BACKUP_LAYOUT's ArchiveScope::Tree("data") is resolved by backup_source as home_dir.join(archive) (backup.rs:152), a hard-coded <home>/data.

Concrete failure: an operator with data_dir = "/mnt/fast/librefang-data" takes a POST /api/backup before the upgrade, hits the guard after the downgrade, follows this page step by step, and the archive turns out to contain no database at all — user_version is untouched, the older binary prints the same refusal, and the page gives them nothing to explain why. The same assumption sits in the "Start fresh" bullet (line 73, move <home_dir>/data/ aside), which is a no-op on such a deployment.

Worth stating the precondition explicitly and naming data_dir / [memory] sqlite_path as the thing to check first.

Confirmed, and fixed in fefd5d28a.

You are right on both halves. boot.rs:378-383 resolves the memory database as config.memory.sqlite_path when set and config.data_dir.join("librefang.db") otherwise, while backup_source (backup.rs:152) resolves every non-agents scope as home_dir.join(archive) — a hard-coded <home_dir>/data that consults neither key. So the archive of a deployment with data_dir = "/mnt/fast/librefang-data" genuinely contains no database, and the page walked the operator through four steps that cannot move user_version.

The page now opens the backup section with an explicit precondition rather than burying it in step 3 — new ### Check first: the databases may not be under <home_dir>/data/ at all (docs/operations/downgrade-recovery.md:40-47), naming data_dir (types.rs:3470, default at types.rs:6835, documented at page.mdx:523) and [memory] sqlite_path (types.rs:7401) as the first two things to read out of config.toml, and stating that the archive can only restore the rest.

Step 3's claim is now conditioned on it rather than asserted flat (:57), and the "Start fresh" bullet you flagged as the same assumption now says "move the data directory aside — <home_dir>/data/, or wherever data_dir points" (:91).

One thing I want to flag rather than quietly document around: the divergence itself is a product bug, not only a doc gap. An operator who relocates data_dir gets a POST /api/backup that reports success and silently archives no database — the backup is worthless and nothing says so. That fix belongs in librefang-api, a different crate from this docs-only PR, so I have not folded it in here. Happy to open it as its own issue if you agree it should be one.

"Configuration, agent workspaces, skills and workflows survive" is not true of three components that BACKUP_LAYOUT names separately but that physically live inside data/ (backup.rs:69-79):

• data/cron_jobs.json (cron_jobs)
• data/hand_state.json (hand_state)
• data/custom_models.json (custom_models)

Citing the layout table as the evidence that configuration survives reads backwards here: the table is exactly what shows those three are under the tree being moved aside. They are covered only by the catch-all "every other data/ artefact are lost", which an operator will not read as "your cron schedules, hand state and custom model definitions".

Concrete failure: operator with no backup takes this option, moves <home_dir>/data/ aside, boots the older binary, and every cron job, hand state record and custom model definition is gone — having just read a sentence saying configuration survives.

Since all three are plain JSON rather than schema-versioned SQLite, the fix is cheap: tell the operator to copy those three files back out of the moved-aside directory.

Confirmed, and fixed in fefd5d28a.

BACKUP_LAYOUT (backup.rs:68-80) does name all three as their own components while giving each an ArchiveScope::File under data/ — cron_jobs → data/cron_jobs.json, hand_state → data/hand_state.json, custom_models → data/custom_models.json. Your reading of the citation is right and mine was backwards: the table is the evidence those three go with the tree being moved aside, not evidence that configuration survives it.

The bullet now separates the two groups instead of leaning on a catch-all (docs/operations/downgrade-recovery.md:91-95):

config.toml, agent workspaces, skills and workflows survive, because the backup layout holds them as separate trees outside data/ (backup.rs:68-80).
Three components that same layout names separately do not: data/cron_jobs.json, data/hand_state.json and data/custom_models.json live inside the tree being moved aside (backup.rs:69-79).
All three are plain JSON rather than schema-versioned SQLite, so copy them back out of the moved-aside directory before starting the old binary — otherwise every cron schedule, hand state record and custom model definition goes with it.
Memory, sessions, the audit trail and every other data/ artefact are lost either way.

Took your suggested remedy directly: because none of the three is schema-versioned, copying them back is safe across the version boundary, so the bullet prescribes it rather than just warning.

This is the guard's format string, not the line the daemon prints. The message is wrapped twice on the way out:

• substrate.rs:217 — run_migrations(...).map_err(LibreFangError::memory)?, and LibreFangError::Memory renders as #[error("Memory error: {message}")] (crates/librefang-types/src/error.rs:61).
• boot.rs:397 — .map_err(|e| LibreFangError::BootFailed(format!("Memory init failed: {e}")))?.

So the operator's log actually reads:

Memory init failed: Memory error: Database schema version 58 is newer than this binary supports (55). Downgrade is not supported. Use the correct binary version or restore from backup.

Concrete failure: an ops page is used by grepping. Someone searching their journal for the fenced line as written gets no hit, and the page's own promise of a byte-identical quote makes them doubt they hit this guard at all rather than suspect a prefix.

(The 58 / 54 pair is fine as an illustration of an older binary — SCHEMA_VERSION is currently 55 — it is only the missing prefix that breaks a copy-paste search.)

Confirmed, and fixed in fefd5d28a.

Traced the wrapping exactly as you describe. substrate.rs:217 is run_migrations(&migration_conn).map_err(LibreFangError::memory)?; LibreFangError::memory (error.rs:261-267) stores source.to_string(), and rusqlite's Display for SqliteFailure(_, Some(msg)) renders just the message, which the #[error("Memory error: {message}")] variant at error.rs:61 then prefixes. boot.rs:397 wraps that a second time as BootFailed(format!("Memory init failed: {e}")).

The page now quotes the line the operator actually has in their journal (:11-19):

Memory init failed: Memory error: Database schema version 58 is newer than this binary supports (54). Downgrade is not supported. Use the correct binary version or restore from backup.

and follows it with the search advice, since the grep is the whole point of quoting it:

Search for Downgrade is not supported rather than for the start of the line — the two prefixes are exactly what a grep built from the guard's format string in migration.rs will miss.

I also dropped the PR body's "The quoted daemon error is byte-identical to the runtime output of the guard" claim, which was the assertion that made this worth catching rather than a typo — it stated verification I had not actually done. Left the 58 / 54 illustration alone per your note.

The procedure removes the stray -shm but says nothing about a stale -wal, and the -wal is the one that can silently undo the rollback.

unzip -o only overwrites entries the archive actually carries. If the pre-upgrade archive has no data/librefang.db-wal entry — a backup taken while the old daemon was stopped, or after a checkpoint that removed the file — the extraction restores the older librefang.db and leaves the newer binary's -wal sitting next to it. SQLite validates WAL frames by their own checksums and salts; there is no cross-check binding a WAL to a particular copy of the database file, so on the next open those frames are replayed. They are the newer binary's pages, page 1 (which carries user_version) included.

Concrete failure: the operator completes every step, starts the older binary, and either sees the identical version refusal — because the replayed page 1 puts user_version back at 58 — or boots onto the migrated schema the guard exists to keep it away from. Nothing in the procedure tells them a leftover file caused it.

Suggest widening the bullet: for each restored SQLite database, delete both the -wal and the -shm before extracting, then let the archive's own pair (if any) land.

Confirmed, and fixed in fefd5d28a — this was the most dangerous of the five, because it is the failure that looks like the procedure not working rather than like a leftover file.

Verified the asymmetry you name: is_sqlite_shared_memory_index (backup.rs:191) is name.ends_with("-shm") and nothing else, so the -wal is archived (backup.rs:1591/:1608 cover exactly that round-trip) and skipped only on the -shm side, at backup.rs:314 on the way out and backup.rs:949 on the way back. Which means the archive carries a -wal only if one existed when the backup ran — and a backup taken against a stopped or freshly-checkpointed daemon has none, so unzip -o has no entry to overwrite the newer binary's WAL with, and it survives.

Took your suggested remedy — delete both sidecars first, then let the archive's own pair land (docs/operations/downgrade-recovery.md:61-66):

Delete both the -wal and the -shm next to every SQLite database being restored before extracting — data/librefang.db and data/a2a_tasks.db under a default layout — and let the archive's own pair land if it carries one.
The -wal is the one that can silently undo the whole rollback.
unzip -o overwrites only the entries the archive actually holds, and a backup taken while the old daemon was stopped, or after a checkpoint, carries no -wal entry at all; the newer binary's -wal then survives next to the restored older database.
SQLite validates WAL frames by their own checksums and salts, and nothing binds a WAL to a particular copy of the database file, so those frames are replayed on the next open — including page 1, which is where user_version lives.

with the observed symptom spelled out, since that is what an operator will search for, and a closing line on why copying the endpoint's -shm-only behaviour is wrong here: it restores into the same daemon rather than across a version boundary.

Naming both databases explicitly also fixed a smaller thing in the same bullet — it previously said "any -shm file", leaving the operator to work out which files that meant.

One knock-on: the bullets' lead-in used to read "Two corrections the restore endpoint applies and unzip does not", which your finding makes false for the second one, since the endpoint does not delete the -wal either. Now "Two things a plain unzip gets wrong, the first of which the restore endpoint corrects for you and the second of which it does not" (:58).

This fragment lands in the GitHub release body verbatim, and it points readers at the opposite of what the page it advertises concludes: "the way back is the backup/restore feature the daemon already ships" reads as use POST /api/restore, while downgrade-recovery.md:56-65 ("Why not POST /api/restore for this") says the endpoint is the wrong tool and prescribes an offline unzip.

Same drift in the PR body's third summary bullet ("POST /api/restore with the pre-upgrade archive from the newer binary, then boot the older one"), which is the procedure the doc explicitly argues against.

Concrete failure: someone reads only the release notes, calls POST /api/restore from the live newer daemon, and walks into exactly the hazard the doc documents — the pool's open connections on data/librefang.db and the daemon checkpointing its stale WAL over what was just restored.

Suggest naming the sanctioned path in the fragment: the way back is an offline restore of a pre-upgrade backup, not the restore endpoint.

Confirmed, and fixed in fefd5d28a.

The drift was real and in three places, not two. The fragment now names the sanctioned path and the reason, so a reader who sees only the release notes is not sent to the endpoint:

Document the supported recovery for a binary downgrade over a migrated database: the older daemon's refusal to boot is the fail-loud guard working as designed, not corruption, and the way back is an offline restore of a backup taken before the upgrade.
Deliberately not POST /api/restore, which runs inside a live daemon still holding the database open and can checkpoint its own stale write-ahead log over what was just restored.
The page also names the two things that quietly break the procedure: a data_dir or [memory] sqlite_path that puts the databases outside the tree the archive carries, and a leftover -wal whose frames replay the newer schema version back over the restored file (#8148) (@DaBlitzStein)

The PR body's third bullet had the same drift and now describes the offline restore, with POST /api/restore marked as explicitly not sanctioned.

The third place was the page's own opening line — "the way back is restoring a backup taken before the upgrade" — which was the sentence the fragment was paraphrasing, and was just as easy to read as "use the restore endpoint". It now says "an offline restore … not the POST /api/restore endpoint" and links down to the section (:5), so the summary and the section it summarises no longer disagree.

python3 scripts/check-changelog-attribution.py passes on the reworded fragment.