@buger — I took the review's "minimum" option and landed it on top of your commit rather than asking you to redo the work. Your commit is untouched; there is one commit on top of it.

Where it is: maintainerCanModify is off for this PR, so I could not push to probelabs:fix/autoindex-incremental-sync-1254. The result is pushed to this repo as pr156-review-followup — 76d8bb0 → your 2a55358 → 66e91d7. If you want it in this PR, git pull that branch into yours (or flip "allow edits by maintainers" and I will push it here).

What changed

The gate now fires only on removals. A rerun is refused when a canonical content group the plan published no longer exists in the folder — nothing in the pipeline deletes the documents, aliases, graph edges or catalog entries that file wrote, so ignoring the deletion leaves them live with no source behind them. That is the case your PR was right to fail closed on. --fresh is refused for it too, because discarding the plan does not delete those documents either.

Additions are reported, not fatal. They are still not absorbed by a frozen plan, but the run now names them on stderr, records them as skipped (exit 3), and points at --fresh. --fresh is a plain in-place plan rebuild again, so the add-then-rerun-then---fresh workflow converges as it did on main. xerj brain follows from that: a note added after the first run no longer errors.

The refusal message names what is unsupported and what to do. It lists every removed file with its content key, states that no remote mutation was made, and gives three routes ordered by cost: restore the files and rerun; rebuild in place by deleting the named indices and the state directory (with the honest note that edges the removed file taught stay live in the named edges index); rebuild isolated under a new --state-dir/--prefix/--brain. --json keeps your typed xerj.autoindex.unsupported_sync_delta.v1 payload with stdout/stderr routing — that part I kept as you wrote it, it is good.

In-place recovery wording is back in the journal-corruption, key-collision and journal-mismatch errors, in --fresh help, the recipes, llms-full.txt and the second-brain docs, and the --fresh idempotency leg is back in demo/usecases/autoindex/run-eval.sh (keeping the per-run exit-code checks you added).

xerj brain: the live-node probe keeps your absent / authoritative-zero / malformed / transport-failure distinction, which is the part of this PR I most wanted to keep. One fix: a brain meta doc with no nodes_index is treated as absent evidence (Ok(None)) instead of an error, so brains written before that field existed still work. The disagreement message now also names the --fresh rerun for a genuinely wiped data directory, next to the isolated rebuild.

What I did not do

• Full add/change/delete reconciliation is still follow-up work, as your PR said. This only narrows the refusal.
• deleting_an_entire_duplicate_group_strands_no_pending_replacement stays flipped to assert refusal. That test deletes an entire content group, which is exactly the unsafe case, so the refusal is correct there; I kept its original invariants (no stranded replacement intent, no journal growth) asserted alongside. The other flipped test, a_planned_key_never_gains_two_owners_when_old_content_moves_paths, is an addition and is restored to its pre-PR assertions.
• PR body / CHANGELOG honesty: I rewrote the CHANGELOG entry to state plainly what fails and what does not. The PR body still describes the wider contract — worth updating if you pull this in.

Verification

Measured on 66e91d7:

• cargo test -p xerj-autoindex: 221 passed, 0 failed.
• cargo test -p xerj-server --bins brain::: 5 passed, 0 failed.
• cargo clippy -p xerj-autoindex --all-targets -- -D warnings and the same for xerj-server: clean. cargo fmt --all clean.
• Mutation check on the gate: with UnsupportedInventoryDelta::refuses() forced to false, completed_plan_rejects_vanished_content_group_before_remote_mutation fails with called Result::unwrap_err()on anOk value: 0 — i.e. the deletion rerun silently succeeds, which is the bug — and only_a_removed_content_group_refuses_a_rerun fails on assertion failed: UnsupportedInventoryDelta::between(&[], &[], &plan).refuses(). Both pass with the gate restored.
• Branch merges cleanly into main as of 2e6fccd; no rebase needed.

Thanks again for finding this one — the silent-skip on rerun was a real hole and the fail-closed machinery you wrote is what made narrowing it a small change.

@buger — thanks for pulling the follow-up in and for bace26c. Naming the new-note bypass as a predicate and asserting it directly is a better guard than the implicit records_total == 0 check it replaces; that is the branch most likely to be broken by future probe hardening, and now it cannot be broken silently.

I re-reviewed the branch at bace26c against the original review. The four blocking items are addressed and I have no further objections to the code. One commit on top, for the last gap.

Where it is: maintainerCanModify is still off for this PR, so a push to probelabs:fix/autoindex-incremental-sync-1254 is rejected. The commit is on this repo as pr156-rerun-gate-ci — your three commits untouched, 87ee42c on top. git pull it into your branch, or flip "allow edits by maintainers" and I will push it here.

What the commit adds

Nothing in CI reruns either tool over a folder or a server that changed — which is exactly where the silent-skip bug lived. run-eval.sh reruns autoindex over an unchanged corpus, and the brain harness never wipes a data directory out from under a resume journal. The refusals have good unit and in-process HTTP tests; this is the missing end-to-end gate, two phases on the harnesses that already exist:

• autoindex: remove logs/app.log from the corpus run-eval.sh just indexed, rerun, require a failure that names the removed file. Then put the file back and rerun — recovery (1) from your own refusal message — and require it through again, so the gate is provably a stop and not a brick.
• brain: kill the demo brain's server, delete its data directory (the resume journal lives outside it, under ~/.xerj/autoindex/<hash>, and survives), rerun. Require the journal/server disagreement, then require the --fresh that message names to actually bring the brain back.

Both assert on the refusal text, not on a nonzero exit. A run that fails for an unrelated reason also exits nonzero, and the pre-fix brain path exits 1 with "nothing indexable" — exit code alone cannot tell those apart from a refusal.

Verification, measured on 87ee42c

• cargo test -p xerj-autoindex: 221 passed, 0 failed.
• cargo test -p xerj-server --bins brain::: 5 passed, 0 failed.
• cargo fmt --all -- --check: clean.
• Full .github/scripts/usecase-smoke.sh against the release binaries: every phase PASS, exit 0 — including the new phase 4 (a rerun after a removal refuses (exit 1) and names the removed file) and phase 5 (the --fresh rebuild the refusal names recovers the brain (exit 0)).
• Mutation check, so the new phases are not decoration. With UnsupportedInventoryDelta::refuses() forced to false and the release binaries rebuilt, phase 4 fails with ::error::expected a nonzero refusal naming logs/app.log, got exit 0 and the script exits 1 — that exit 0 is the original silent skip. Phase 5 still passes under that mutation, correctly, since it gates the brain live-node probe rather than the inventory delta. The same mutation fails completed_plan_rejects_vanished_content_group_before_remote_mutation with called Result::unwrap_err() on an Ok value: 0, and only_a_removed_content_group_refuses_a_rerun on assertion failed: UnsupportedInventoryDelta::between(&[], &[], &plan).refuses(). Restored, everything is green again.
• The branch merges into main with no conflicts.

One thing left, and it is not code

The PR title and body still describe the original shape of this change: --fresh "accepted only when the selected state directory has no durable plan", "adding or removing a canonical content group is detected and rejected", the single-preflight-lock section. None of that is what the branch does now — additions are reported and skipped rather than rejected, --fresh is a plain in-place plan rebuild, and only removals are refused. The Validation section has the same problem: the 219-test count and the ES-YAML run were measured on the first version of the tree.

I have not touched your description. It does need updating before this merges — "honest claims only" is the repo's first rule, and a squash merge would carry that text into main's history as the record of what shipped. The CHANGELOG entry on the branch already says the accurate thing, so it is mostly a matter of bringing the body in line with it.

Thanks again for finding this one and for staying with it through two rescopes.

Status at head bace26c: the code is in good shape and I have no further objections to it. Three things stand between it and a merge, and all three need you — I have no push access here (maintainerCanModify is off, and a push to probelabs:fix/autoindex-incremental-sync-1254 returns 403).

1. It no longer merges cleanly. This is my doing, not yours — I merged #160 a few minutes ago (96f6d1a), and this branch now conflicts with main in two files:

• CHANGELOG.md — both sides added entries in the same region; a normal adjacent-entry conflict.
• engine/crates/xerj-autoindex/src/state.rs — overlaps with the embedding-identity work that came in with Feat/embedding execution identity #160.

Everything else in the diff auto-merges. A rebase onto current main should be short. I deliberately did not force anything.

2. The CI rerun-gate commit is still off the branch. The earlier review's point that no job reruns either tool over a mutated folder is unaddressed on the branch itself — both refusals are still gated only by unit and in-process HTTP tests. The commit that fixes it is 87ee42c on xerj-org/xerj branch pr156-rerun-gate-ci (0bea6bb → b0ed1c8 → bace26c → 87ee42c). It adds two phases to .github/scripts/usecase-smoke.sh: phase 4 removes logs/app.log from the corpus autoindex just indexed, reruns, requires a failure naming the removed file, then restores it and requires the rerun to pass — so the gate is provably a stop and not a brick; phase 5 does the equivalent for the brain and then requires the --fresh that the refusal names to actually rebuild. Both assert on the refusal text rather than a nonzero exit, because the pre-fix brain path also exits 1.

Please git pull it, or enable "allow edits by maintainers" and I will push it. The PR is mergeable without it — this is a coverage gap, not a correctness one.

3. The PR title and body still describe the first version of this change. They say --fresh is accepted only with no durable plan, that additions are rejected, and they include the single-preflight-lock section; the Validation section quotes counts measured on that tree. The branch no longer behaves that way — additions are reported and skipped, --fresh is a plain in-place plan rebuild, and only removals are refused. Since this will be squash-merged, that body becomes main's record of what shipped, and under "honest claims only" it should not. The CHANGELOG on the branch is already accurate, so this is only a matter of bringing the description in line. I have deliberately not rewritten your prose.

On the substance, for the record — the refusal does fail closed with an actionable message and is genuinely covered. The autoindex gate runs before the journal is opened and before any remote call other than the readiness ping; it names each removed file with its content key, states that no remote mutation was made, and gives three recoveries in cost order, including the honest note that edges taught by a removed file stay live. --json emits the typed xerj.autoindex.unsupported_sync_delta.v1 with exit 1. The brain refusal names the journal path, URL, prefix and brain, and names both the in-place --fresh rerun and the isolated rebuild.

The review state is still CHANGES_REQUESTED from the original round; I have left it as-is rather than clearing it for you, but treat the code objections as withdrawn.

@buger — first, the good part: everything I raised as a code objection in the earlier rounds is closed. The gate fails closed before any remote mutation, the refusal is typed and exits 1, the new-note probe bypass is pinned as a predicate at bace26c, and the four original blockers are gone. Thanks for staying with this through two rescopes 🙏

Then the part I owe you an apology for. I re-measured --fresh on this branch with clean, separate CARGO_TARGET_DIRs, and my earlier all-clear was wrong — an earlier run shared a target dir and gave me a false reading, so I rebuilt from scratch to confirm. --fresh has two regressions on this branch that I signed off on last round. That is my miss, not a new demand. Verdict is back to hold, and here is everything, with what would make each one pass.

1. --fresh is refused after any same-path content edit (regression, introduced by b0ed1c8)

engine/crates/xerj-autoindex/src/lib.rs:739-741 — under cfg.fresh the comparison uses inventory.keys.clone(), i.e. the raw post-edit content keys. Those are compared at lib.rs:760 against a plan that permanently retains the pre-edit key: lib.rs:865 writes inventory.keys[index] = planned_key, so the planned key never follows the edit. The pre-edit key is therefore "vanished", refuses() at lib.rs:624 returns true, and the run dies.

Measured both ways, clean target dirs: edit a file in place, rerun with --fresh → ERROR at bace26c3; the same sequence on main at 96f6d1a → Ok(0). So this is a regression, not a pre-existing limitation.

What would make it pass: when cfg.fresh is set the delta has to be computed over paths that genuinely left the folder, not over content keys the plan is guaranteed to hold stale. An edited file is still present; it should never appear in vanished_content_groups.

There is also no test for edit-then---fresh. a_rerun_after_an_added_file_succeeds_and_fresh_absorbs_it only adds a file, and fresh_cannot_erase_the_plan_and_bypass_the_removal_gate (failure_resume_http_tests.rs:399) only removes and adds. Both assert the new behaviour without ever exercising the path that broke — a test that edits a file's bytes at the same path and then runs --fresh is the one that fails today.

2. --fresh no longer clears a journal root/url/prefix mismatch (regression, introduced by 0bea6bb, not repaired by b0ed1c8)

engine/crates/xerj-autoindex/src/state.rs:296 calls read_plan_for_preflight, which hits anyhow::ensure!(recorded_root == root && ...) at state.rs:238. That fires inside preflight — before the fresh deletion at state.rs:347 ever runs. So --fresh can no longer rescue a journal whose identity moved.

Measured: same --state-dir with a changed --prefix plus --fresh → ERROR at head, Ok(0) on main.

Two things compound it: the preflight message that now fires offers no recovery at all, and the message that does name --fresh (state.rs:383-386, "or --fresh to rebuild the plan in place") is now unreachable dead code.

What would make it pass: perform the fresh journal deletion before the identity check, or skip the identity ensure! when fresh is set — the recorded identity of a journal you are about to delete cannot be a precondition for deleting it.

3. The refusal message contradicts itself (repo rule: honest claims only)

lib.rs:519-560. In the measured edit-plus-add case, one path appears on both sides of the same message: Removed content groups [a.csv (axf2-49c7...)] and Also present but not in the frozen resume plan ... [a.csv (axf2-443b...)]. The surrounding prose then asserts the file no longer exist[s] in the folder and that their documents are still live in the destination — while a.csv is sitting right there and its documents were just republished.

What would make it pass: the message needs to separate "this content key is no longer present" from "this file is no longer present", and must not claim a file is gone when the same relative path is on both lists. Fixing item 1 removes most of these cases; the wording still needs to be safe for the ones that remain.

4. CHANGELOG claim is false for the changed-file case

CHANGELOG.md around line 29 states that "Adding or changing files and rerunning keeps working; only removals are refused", and that the run "points at --fresh, which rebuilds the plan in place and picks them up". The second half is false for a changed file, per the measurement in item 1. Flagging this specifically because the CHANGELOG was the one artifact I certified as accurate last round, so it should not be assumed still-good after item 1 is fixed.

5. --fresh advice is wrong for a legacy journal (inference — read, not executed)

lib.rs:395 now advises Deleting the journal at {} (or rerunning with --fresh) also clears the collision. For a legacy journal the --fresh half looks wrong to me: legacy plan keys are {16hex}-{sizehex} (ids.rs:25) while inventory keys are axf2-{32hex}-{sizehex} (content.rs:134), and the plan permanently retains the legacy keys — so --fresh would report every file in the corpus as removed. The same reasoning applies to use --fresh after verifying the existing index at lib.rs:345 and lib.rs:355. I derived this by reading rather than running it, so treat it as inference; worth a legacy-journal + --fresh test either way.

6. Still no CI job that reruns either tool over a mutated folder

Unchanged from the last round: .github/scripts/usecase-smoke.sh at head is phases 0-3 only, and 87ee42c (phases 4 and 5, on xerj-org/xerj branch pr156-rerun-gate-ci) is still not on this branch. Both refusals remain gated only by unit and in-process HTTP tests — which is exactly the coverage shape that let items 1 and 2 through.

7. The branch no longer merges

git merge-tree origin/main bace26c3 conflicts in CHANGELOG.md and engine/crates/xerj-autoindex/src/state.rs. Merge base is 2e6fccd; main is now 96f6d1a (#160). Still my doing, still a short rebase.

8. PR title and body still describe the first version

They still say --fresh is accepted only with no durable plan, that additions are rejected, they carry the single-preflight-lock section, and the Validation counts were measured on the 219-test tree. A squash merge writes that text into main's history as the record of what shipped. Note that item 4 means the CHANGELOG can no longer just be copied across — it needs the same correction.

Two notes, not objections

xerj brain has no working in-place route left. For a removed note the gate refuses (lib.rs:760) and also refuses --fresh for that same case (lib.rs:755-762), and xerj brain exposes no way to delete the ax-* indices the message tells you to delete. On main this exited 0. The prior review accepted that trade explicitly and I am not re-opening it as a code objection. I am only pointing out the combination: with the --fresh edit regression in item 1, xerj brain over a live notes folder currently has no working in-place route for either add-plus-edit or removal. Fixing item 1 restores one of the two.

A pre-existing invariant hole this PR makes routine. On main already: an added-but-skipped file gets a live catalog document written for it (lib.rs:2052, all_junk.extend(new_unplanned.iter())) yet never enters the durable plan. Delete that file later and the delta sees nothing vanished, so the stale catalog entry stays live forever — precisely the staleness the new gate refuses for planned junk (failure_resume_http_tests.rs:468, deleted_planned_junk_fails_closed_before_catalog_can_stay_stale). Not introduced by you; the newly-blessed add-then-rerun path just makes it the common case. Worth its own issue rather than more scope here.

On access: thank you for offering write access to probelabs/xerj — the invitation is still unaccepted on our side (our PAT gets a 403 accepting it), so we genuinely cannot push to your branch yet and I am not going to pretend otherwise. Please keep pushing as you have been, and the moment that access goes live we will happily pick up the rebase, the CI phases and the body rewrite ourselves so you do not have to carry all of it. 🚀

Items 1 and 2 are the blockers; the rest travel with them.

Reviewed. The bug is real and the fail-closed gate is the right interim shape, but three things need fixing before this lands.

--fresh is no longer an escape hatch. Journal::preflight calls read_plan_for_preflight unconditionally, and that hard-errors on a malformed record, a root/url/prefix mismatch, or an undecodable plan — all before open_after_preflight reaches the remove_file. Before this PR, --fresh deleted the journal without parsing it. Two consequences: your own message at state.rs:224 advertises --fresh as the recovery from exactly the error that now blocks it, and Journal::open decodes the plan tolerantly while the preflight turns the same condition into a hard ?. Make the preflight no more fatal than open.

Unbounded error rendering. The render closure joins every vanished entry into one String, no cap — even though this same PR caps its "appeared after the plan was frozen" block at .take(10). On a real journal here (82,551 files) that's ~7 MB in one String plus the intermediate Vec. An unmounted bind mount makes every group vanish. Apply the same cap.

xerj brain takes the wrong branch on the case its new message is written for. live_node_docs propagates via ? on a non-2xx, so if the meta doc survives but the nodes index was deleted, the probe 404s and never reaches journal_server_disagreement(). User gets a raw HTTP 404 instead of the recovery text.

Also the PR description doesn't match the code: additions are not rejected (lib.rs:624), and your own test a_rerun_after_an_added_file_succeeds_and_fresh_absorbs_it asserts that. And --fresh after an in-place edit is refused because the old digest key vanishes, which contradicts the CHANGELOG line.

Branch is 22 commits behind and conflicting on CHANGELOG.md and state.rs. Rebase and re-run CI, the green run is from 2026-08-04 against an old base.

Quickwit is the precedent worth reading for the follow-up you mention: mark-then-sweep with grace periods and bounded batches rather than refuse. Fail-closed is defensible until then.