fix(vector): Fix HNSW entry metadata and stale hits when an external ID is reused - #64

Summary

This fixes a deterministic HNSW consistency bug triggered by inserting a new vector under an external ID that is already owned by the graph entry point.

The flat slab intentionally retains the superseded slot so existing edges can continue traversing it, while slot_of makes only the replacement slot current. The entry metadata did not follow that ownership change. Reusing a layer-3 entry ID with a layer-0 vector could leave the old orphan slot as entry_slot with entry_layer = 3.

That stale state had two observable consequences:

• Search could return the superseded vector under the current external ID.
• Persistence wrote the current slot once, but paired its external ID with the orphan slot's old layer. The resulting file was CRC-valid while its entry metadata described a layer the serialized entry node did not have.

No authoritative document or latest vector is discarded. The failure affects graph result correctness and persisted graph consistency; the engine can reconstruct derived HNSW state from its authoritative data.

Reproduction

The regression uses deterministic levels:

ID 7, vector [1, 0], layer 3
ID 7, vector [0, 1], layer 0

Before this change, the runtime and reopened graph retain entry layer 3, even though the only current serialized node for ID 7 has maximum layer 0. A query for [1, 0] can also return the orphaned vector as ID 7.

The focused reproduction can be run with:

cd engine
cargo test -p xerj-vector reused_entry_external_id_with_lower_layer_saves_a_reloadable_graph -- --nocapture

Implementation

The change makes live-slot ownership explicit: a slot is current only when the reverse map resolves its external ID back to that exact slot.

When insertion reuses the entry identity, publication selects the highest-layer current slot after the replacement becomes live. This may be the replacement itself or a different remaining live node; it is not automatically the new lower-layer slot.

Search still traverses superseded slots so existing graph connectivity is preserved, but only current slots are admitted to the result set.

The persistence writer independently derives both entry ID and entry layer from live topology. This is defense in depth for an already inconsistent runtime graph and prevents an orphan slot's layer from being paired with a replacement slot's identity.

The on-disk format version and byte layout are unchanged.

Tests

Three deterministic regressions cover:

• Reusing a layer-3 entry ID at layer 0, excluding the superseded vector from results, saving, and reopening with the latest vector and entry layer 0.
• Selecting a different live layer-2 node as the entry instead of blindly choosing the layer-0 replacement, with search checks before and after reopen.
• Recreating the old stale (orphan slot, layer 3) state directly and proving the writer emits a live, reloadable layer-0 entry header.

Full scoped verification:

cargo test -p xerj-vector
38 passed, 0 failed

cargo test --release -p xerj-vector
38 passed, 0 failed

cargo clippy -p xerj-vector --all-targets -- -D warnings
passed

cargo fmt --all --check
passed

git diff --check
passed

Scope and trade-offs

The contribution changes only engine/crates/xerj-vector/src/hnsw.rs.

Normal insertion retains its existing entry-selection path. Reusing the current entry identity performs one scan of the slab to find the highest live entry. Persistence already scans the complete graph, so deriving the live entry does not change its asymptotic cost. Result admission performs a reverse-map ownership check so orphan slots remain traversal-only. No end-to-end throughput improvement is claimed by this correctness fix.