fix(engine): prevent older binaries from rewriting newer cluster state - #313

Why this matters

Consider an older or stale XERJ binary started against a data directory already written by a newer binary.

Before this change, the format-1 loader did not reject unfamiliar cluster_state.json bytes before storage discovery. The older binary could scan existing indices, retain only metadata fields it understood, and later replace the live document with its own reduced format-1 representation. That silently discarded newer control-plane and publication metadata while WAL replay, Console bootstrap, background storage work, and request-driven mutations remained reachable. This is metadata loss, even though the file itself is rewritten rather than deleted.

A source-isolated causal A/B reproduced the failure with a representative synthetic future-format document; this does not claim that format 2 currently ships. The exact parent rewrote the 653-byte document as a 364-byte format-1 document, discarded the future control-plane/publication metadata, reconciled state from index directories, and accepted index-template management PUT and DELETE. The candidate preserved all 653 bytes exactly across two blocked boots and rejected the same mutations with a typed error.

Compatibility effect

This does not prevent rollback while cluster_state.json remains in the exact shipped format-1 schema. That path is covered by byte-stable restart, supported mutation, and parent/candidate equivalence tests.

It intentionally refuses rollback after a newer binary has written persistent state the older binary cannot fully understand, including a higher or alternate discriminator or unknown format-1 envelope and typed-record fields. The previous behavior was not safe compatibility: the older binary could silently discard that metadata and overwrite the newer state. This change chooses explicit storage unavailability requiring operator action over allowing an incompatible binary to overwrite control-plane metadata it cannot represent.

Rolling back across a future schema change therefore requires restoring supported format-1 bytes or running an explicit downgrade/migration using software that understands both schemas; otherwise the operator must continue with a compatible binary. Any downgrade path must define how every persisted field and any affected index/WAL invariant is handled, and may need to refuse conversion when the older schema cannot represent the state. This PR does not implement such a conversion.

Result

XERJ now classifies persisted cluster state before opening any user or system index or replaying a WAL. If the document is newer than this binary, malformed, duplicate-keyed, schema-incompatible, or unreadable:

• the process remains live for diagnosis while readiness reports unavailable;
• authenticated and authorized HTTP storage requests receive typed 503 cluster_state_unavailable, while existing body-size, authentication, and authorization precedence remains unchanged;
• authenticated gRPC calls receive UNAVAILABLE;
• no user/system index, WAL, durable auxiliary registry, Console bootstrap, cluster transport, storage-mutating background task, or shutdown index flush is activated;
• the rejected live document and cluster_state.tmp remain untouched, with no salvage rewrite.

Supported format-1 cluster-state and managed-storage behavior remains equivalent under the audited comparison boundary.

Implementation

• Classify cluster_state.json once, before directory discovery and any Index::open.
• Accept only the exact seven-field format-1 persistence envelope emitted by the shipped writer, with recursive duplicate-key detection and strict typed shapes for index templates and data streams.
• Construct a diagnostic in-memory Engine shell for unsupported, malformed, incompatible, or unreadable state.
• Skip durable audit/API-key/alias/ISM/ML registries, Console bootstrap/routes, cluster transport, lifecycle work, periodic flushing, resource sampling, ingest tracing, context sweepers, and final index flush while blocked.
• Apply the HTTP fence after the existing decompression, body-size, authentication, and authorization gates.
• Apply the gRPC fence centrally after authentication across all six current service methods.
• Guard direct durable Engine, lifecycle, snapshot, alias, API-key, mapping, ISM/ILM, data-stream, create/delete, retry, and CLI entry points before side effects.

Recovery

Restore a cluster_state.json supported by this binary or run a binary that understands the stored format, then restart the node. XERJ does not automatically repair, downgrade, quarantine, or migrate rejected state.

Data-directory creation, node.lock acquisition/PID diagnostics, and credential/TLS preparation that precedes Engine construction remain outside this storage fence.

Verified

Verified on exact parent/head
18f85755b2f25d2a4eefb6b7539a129f017ef6b9 -> d8b243023f3c325c3c433cd384ad74ca4e12af51:

• the final branch is one commit, its worktree is clean, and both committed-range and worktree diff checks pass;
• nine focused fence commands executed 24 passing tests and 0 failures;
• complete xerj-common, xerj-engine, xerj-api, xerj-console-api, xerj-server, and xerj-autoindex package gates executed 1,917 passing tests, 0 failures, and 26 declared ignored tests;
• cargo fmt --all --check, scoped server check, strict six-package all-target Clippy with -D warnings, and scoped thin-LTO server/runner builds passed;
• the current ES-YAML suite passed 1,366 cases, failed 0, and skipped 3 (1,369 total);
• the fresh causal A/B established the parent/candidate failure difference described above, stable candidate storage inventories across both blocked boots, supported format-1 equivalence under the audited boundary-C comparison, and typed refusal without target-index creation through the direct non-HTTP CLI path.

The A/B outputs are machine-local validation records, not checked-in review artifacts. The tracked regression tests and reproducible fixture under engine/crates/xerj-engine/tests/fixtures/cluster_state/ carry the public review contract. No previous-candidate runtime result is reused for this tree.

Scope boundaries and validation policy

• This is a format-1 compatibility fence, not a format-2 implementation. The tests exercise detection, typed refusal, exact byte preservation, two blocked boots, and operator-directed recovery for a representative synthetic future-format document. No format-2 writer, reader that interprets format 2 as supported state, downgrade, or migration path exists in this PR; adding one requires a separate versioned data-format design.
• Strict power-loss durability was not tested because this change adds no persistence algorithm. Supported-state writes retain the existing temporary-file write, file sync, atomic rename, and best-effort parent-directory sync behavior. The tested rejected-state contract is no storage activation or publication while live, after mutation attempts and shutdown, and across two restarts; hardware/filesystem power-loss guarantees require a separate fault-injection environment.
• No performance claim is made. The A/B harness was designed for causal correctness and storage-integrity comparison, not controlled performance measurement, so its build and wrapper times are provenance only. Production binary size was not compared because the qualification binaries use profiling/thin LTO rather than the release fat-LTO profile.
• Repository policy requires scoped local builds, so no workspace-wide or local fat-LTO build was run. Complete tests for the six relevant packages, strict all-target Clippy for those packages, scoped thin-LTO builds, and the full ES-YAML conformance gate were run instead; release fat LTO remains a CI/release concern.
• GitHub-hosted checks remain external merge gates and are not counted as local validation.

Reproduction coverage

The tracked tests cover exact format-1 acceptance and rewrite stability; future, malformed, duplicate-key, unknown-field, and unreadable classifications; two blocked boots without WAL/index/audit mutation; staging-file preservation; direct durable mutators; lifecycle delete/detach ordering; HTTP 413/401/403/503 precedence; native and ES-compatible storage routes; AppState ML registry isolation; Console bootstrap side effects; all six gRPC methods; readiness/liveness; snapshot restore; and recovery after installing supported bytes and restarting.