Skip to content

fix(engine) Publish semantic companion vectors to HNSW without weakening passage correctness - #68

Merged
xerj-org merged 7 commits into
xerj-org:mainfrom
probelabs:fix/current-main-semantic-companion-hnsw
Jul 29, 2026
Merged

fix(engine) Publish semantic companion vectors to HNSW without weakening passage correctness#68
xerj-org merged 7 commits into
xerj-org:mainfrom
probelabs:fix/current-main-semantic-companion-hnsw

Conversation

@buger

@buger buger commented Jul 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

DEPENDS ON #67 being merged first

Summary

This dependent stack makes the vector generated by a semantic_text field a real internal native vector field, publishes eligible companions to HNSW, routes eligible plain semantic queries through HNSW with exact rescoring, and deliberately keeps passage-bearing documents on the authoritative exact scorer.

For AI-agent users, this fixes a major mismatch between the zero-configuration promise and runtime behavior: semantic embeddings were already generated and stored, but the implicit companion field was not registered as a native vector, so the graph skipped every semantic document. After this change, pooled-only semantic corpora can build, persist, restore, and query a fully covered graph without exposing an extra field in the public mapping.

This is a correctness and path-selection change, not a semantic throughput claim.

Dependency

This is PR 2 and must target the head of PR 1 (bc811c35b2e7bc2669f4758fa0057a340da9269d) or be opened after PR 1 merges. It relies on PR 1’s generic publication barrier so semantic raw/turbo writes cannot expose source state while their graph mutation is missing, and so persistence cannot mark an overlapping graph generation as current.

Root cause

The mapping converter attached embedding configuration to the public semantic_text source field but did not register its implicit or custom target in the engine schema. Dynamic inference then treated values such as body_vector as ordinary numeric arrays, while HNSW correctly accepts only explicitly native Vector fields. Semantic embeddings existed in _source, but all documents were skipped by graph publication and semantic queries remained on the O(N) exact scan.

The query executor also called the brute-force semantic scorer directly, so repairing graph construction alone would not have selected the ANN path.

A second correctness problem appears for long documents. Document-level HNSW stores one pooled companion vector per source document, but XERJ ranks chunked documents by their best passage. Candidate pooling on one document vector cannot prove that it retained the document containing the globally best passage. Candidate-local inspection is insufficient because the omitted best-passage document is, by definition, unavailable for inspection.

Design and invariants

  • Register implicit and custom semantic companions as internal native Vector fields with the producer’s dimensions and similarity.
  • Preserve the public mapping round trip: implicit companions remain internal engine state and are not presented as user-declared fields.
  • Never overwrite an explicit target. Reject incompatible dimensions, similarity, non-vector targets, shared-target ambiguity, unsupported nested semantic sources, and ambiguous literal-versus-nested dotted targets before mutation.
  • Publish semantic source and companion schema additions atomically within an index.
  • Prevalidate the merged raw mapping and candidate schema delta for every index selected by wildcard or comma-list PUT _mapping before publishing the first target.
  • Route only eligible plain semantic queries through the existing HNSW admission gate and exact-rescore tail. Filters, aggregations, non-pinned fields, incomplete or stale graphs, small indices, non-cosine mappings, and other unsupported shapes remain exact.
  • Thread request deadlines through graph admission and candidate hydration. Expired ANN work never publishes partial candidates or caches a timed-out answer.
  • Maintain a monotonic per-field passage-exact guard before every source-publication path, persist it beside HNSW identity state, restore it from authoritative data, and fail closed for missing, malformed, or unknown-version markers.
  • Sample the passage guard before ANN work and again before returning. Once a field has observed <target>_chunks, queries for that field remain exact even after replacement or deletion until the index is rebuilt.
  • Preserve the winning passage ordinal and response provenance on the exact path.

The central invariants are: arbitrary numeric arrays never become vector graphs; a semantic graph is used only when its native field identity and coverage are proven; and document-level ANN never replaces passage-local scoring where doing so can change the winner.

Manual reproduction

Before

  1. Create an index with a semantic_text field such as body and its implicit target body_vector.
  2. Bulk enough documents to exceed the HNSW minimum.
  3. Refresh and inspect _cat/ann?format=json.
  4. Observe that the companion is inferred as an ordinary numeric array rather than a native vector, graph coverage remains absent or zero, and a plain semantic query executes the exact scan.
  5. Flush and restart; there is no usable semantic companion graph to restore.

The audited FinanceBench A-v4 corpus demonstrated this shape on 6,622 records: embeddings were present, but the implicit companion schema was not native and graph publication was unavailable.

After

Repeat the pooled-only reproduction. The internal companion is registered as a native vector, every eligible document is published, _cat/ann reports full coverage, a plain eligible semantic query enters HNSW and exact-rescores candidates, and flush/restart restores the graph.

For an adversarial passage reproduction, index 1,024 pooled-only documents plus one chunked document whose pooled vector lies outside the ANN winners while its second passage is the exact winner. The passage guard forces the exact scorer, returns the correct document, passage ordinal, and passage text, survives restart, and remains conservative after replacement or deletion.

For multi-index mapping validation, place a clean index before a later index whose hidden body_vector schema conflicts with the proposed semantic companion, then apply one comma-list mapping update. The request is rejected before either index’s engine schema or public mapping changes.

FinanceBench result and limitation

The audited 20-PDF FinanceBench A-v4 corpus contains body_vector_chunks on 5,693 of 6,622 documents, or 85.97%, and every one of the 19 unique PDF titles contains passage-bearing documents. The stack therefore remains exact for FinanceBench semantic retrieval.

No FinanceBench semantic speedup is claimed. This PR repairs companion graph construction and enables ANN for pooled-only corpora, but the current document-level representation cannot safely accelerate realistic long PDFs. Passage-node ANN with bounded document reconstruction and authoritative exact rescoring is the next architecture step.

Files changed

  • engine/crates/xerj-api/src/es_compat.rs: semantic companion schema planning, mapping-contract validation, all-target prevalidation, ANN routing, deadline propagation, and API regressions.
  • engine/crates/xerj-engine/src/index.rs: internal companion registration support, passage-exact guard, persistence/reload behavior, graph admission, exact fallback, publication-path integration, and engine regressions.
  • engine/reports/2026-07-28_semantic-companion-hnsw.md: implementation rationale, source-chain evidence, current-main composition audit, FinanceBench boundary, and remaining transactionality limitation.

Verification

Verification on exact source HEAD 9193ae16e59e87516f46f2d3f69dff8113762293 includes:

  • cargo test -p xerj-engine --lib --no-default-features: 241 passed, 0 failed.
  • cargo test -p xerj-engine raw_vector_publication_tests --release --no-default-features: 7 passed, 0 failed.
  • Focused release passage tests: 7 passed, 0 failed.
  • Isolated deadline regressions: 5 passed, 0 failed.
  • cargo test -p xerj-api --lib --no-default-features: 75 passed, 0 failed; 66 seconds wall time.
  • cargo test -p xerj-api semantic_companion_schema_tests:: --lib --no-default-features: 4 passed, 0 failed; under 1 second wall time.
  • cargo test -p xerj-api semantic_companion_hnsw_api_tests:: --lib --no-default-features: 6 passed, 0 failed; 2 seconds wall time. Coverage includes multi-index prevalidation, collision atomicity, literal dotted targets, graph construction, and restart.
  • cargo clippy -p xerj-api --no-default-features -- -D warnings: passed with zero warnings; 29 seconds wall time.
  • No-default-features engine Clippy with -D warnings: passed.
  • cargo fmt --all --check: passed.
  • git diff --check: passed.

The exact source-bound ES-YAML gate passed 1360, failed 0, skipped 3, total 1363 across 199 files in 139 seconds. The tested server binary is 44,063,720 bytes with SHA-256 8b0c0ca6c97d8b5ff9d883e6c9f6ff23a85d8977c5c973721523f53207b63a71; the runner SHA-256 is 576429d4d1749ba9328f73fc59300f629b2150e0519357f76ea16e71c105f68f. The source-build manifest SHA-256 is 394be20a0834bc69c0f3b5bce611061199d1a70963555b160a4942a8264aa935, and the final gate-result manifest SHA-256 is 8fb79a88ebcdaff71f06fe3bb703af928d3591c7f5898d85aa127f1dd657d3e4.

Performance and memory tradeoffs

Pooled-only semantic fields gain access to the existing HNSW candidate path and exact-rescore tail, but this PR does not publish a speedup number. The only honest claim is that the previously unreachable path is now constructed and selected when its eligibility invariants hold.

Passage-bearing fields intentionally retain O(N) exact semantic scoring. Persisting the passage guard adds a small sorted field set to HNSW identity metadata, and publication paths add monotonic guard checks. Internal companion registration adds graph storage only for a field that is actually selected and eligible. No claim is made that the full FinanceBench corpus fits a particular latency or memory target after this change.

Compatibility and rollback

Public ES-compatible mapping responses remain stable because implicit companions are internal. Existing explicit vector targets continue to work when their dimensions and similarity match the semantic producer. Literal top-level dotted custom targets remain supported and tested.

Reverting this stack returns semantic retrieval to the exact path and stops constructing implicit companion graphs; it does not require a stored-source migration. Reverting PR 1 underneath this stack is not supported because it would remove the publication ordering contract on which raw/turbo semantic graph consistency depends.

Risks and honest limitations

  • FinanceBench remains exact because passage arrays are prevalent; this PR provides no 368-PDF completion-time or semantic requests-per-second claim.
  • The passage marker is monotonic. If the final chunked document is replaced or deleted, the field conservatively stays exact until reindexing.
  • Deterministic multi-index mapping conflicts are prevalidated before mutation, but this is not universal cross-index transactionality. Concurrent dynamic schema evolution and filesystem persistence failures remain separate engine limitations.
  • Only one pinned eligible vector field is graph-served in the current design; non-pinned semantic fields remain exact.
  • ANN recall is not exact by construction. Eligible HNSW results are exact-rescored, while corpus-wide ANN recall remains governed by the engine’s measured recall contract.
  • Default embeddings remain lexical feature hashing. Neural semantic embeddings require explicit --embed-mode neural; the schema and passage correctness rules apply independently of that choice.
  • Passage-node ANN, bounded candidate materialization, and execution diagnostics are future work and are not hidden inside this contribution.

buger added 7 commits July 29, 2026 04:46
@buger
fix(engine): make vector publication complete before ANN eligibility
bc811c3 
Dense-vector writes could make their WAL/version-map and FTS source visible before the matching HNSW mutation. During that interval the old coverage equality still looked complete: a new raw document was absent from both the graph map and its denominator, while a reused ID retained its old graph node. An ANN query could therefore serve omitted or stale candidates instead of falling back to the authoritative source scan.

The same inverse boundary affected persistence. A save that started after source visibility but before HNSW completion could serialize the old graph and id map with the new WAL sequence and stale=false. On restart the sequence stamp then appeared current even though the vector state was not.

Introduce one per-index RAII publication contract shared by single/prepared/update writes, Value turbo, async raw, sync raw, realtime turbo, and live delete. Writers increment an in-flight counter and monotonic generation before their first visibility change and release both only after HNSW publication. ANN samples both before graph search and after hydration; overlap at either edge returns None so the caller performs an exact scan. Snapshot saves sample generation around graph serialization, capture the WAL sequence before their final generation/in-flight sample, and persist stale=true whenever publication overlaps.

Complete the previously missing ingest maintenance too. Value turbo now scans the full mixed batch instead of gating all vector work on processed[0]. Async raw publishes the parsed authoritative sources to HNSW. Sync raw conditionally parses only indices with an explicit dense-vector mapping, crosses the cached engine runtime once per vector batch, and finishes graph publication before success. Ordinary non-vector sync ingest retains its sealed raw-byte, no-parse, no-runtime-crossing path.

A public sync vector call made directly from a Tokio runtime thread would panic in Handle::block_on. It now returns an actionable error before validation, WAL, version-map, memtable, or graph mutation; CLI/Rayon and other dedicated worker threads keep the supported synchronous path.

Regression evidence uses >= HNSW_MIN_DOCS graphs and deterministic true winners. It covers a mixed Value batch whose first document is non-vector, async raw reused-ID publication with flush/restart persistence, sync raw reused-ID publication with flush/restart persistence, direct-Tokio rejection with zero state change, new and reused writers paused after source visibility but before HNSW while queries exact-fallback and saves persist stale=true, and a new winner published after ANN graph search that is found only through the generation-triggered exact retry.

Verified on the exact afcb3a7 base:

* cargo test -p xerj-engine raw_vector_publication_tests --no-default-features: 7 passed, 0 failed

* cargo test -p xerj-engine raw_ --no-default-features: 15 passed, 0 failed

* cargo test -p xerj-engine --lib --no-default-features: 234 passed, 0 failed

* cargo fmt --all --check and git diff --check: clean

Performance tradeoff: non-vector ingest is unchanged except the existing Value-batch candidate scan now stops at the first numeric array anywhere in the batch. Vector-mapped sync raw batches pay one JSON materialization plus one runtime crossing per batch, followed by the graph inserts already required for correct ANN. Vector-mapped write paths add two begin/end atomics; ANN adds two generation/in-flight sample pairs. No model work or extra parsing is added to async raw because those sources were already parsed.

Implementation and focused regression coverage are in engine/crates/xerj-engine/src/index.rs.
@buger
fix(semantic): publish companion vectors to HNSW
983bd4c 
semantic_text ingestion writes derived embeddings into _source, but current
main does not register the implicit target in the native schema. Dynamic
mapping consequently infers body_vector as double, while HNSW deliberately
accepts only mapped Vector fields. Every semantic document is skipped and
semantic queries remain on the O(N) exact scan.

Register implicit and custom companions as internal native Vector fields
without exposing them in the public mapping or overwriting explicit targets.
Publish multi-field schema additions atomically, reject ambiguous or
incompatible semantic contracts before mutation, and preserve the exclusion
of arbitrary numeric arrays. Route eligible plain semantic queries through
the fully covered HNSW path with exact rescoring and deadline-aware hydration;
filters, aggregations, non-pinned fields, stale graphs, multi-passage
documents, and other ineligible shapes retain exact fallback.

Regression coverage uses real API create and bulk operations over 1,024
documents. It asserts complete graph coverage, ANN path selection, exact
winner and score, fixed-fixture recall, timeout/cache/filter/aggregation
behavior, mapping collisions, literal dotted custom targets, and
flush/restart restoration.

This commit replays production change 9d677073 onto current main afcb3a7.
Static composition inspection preserves merged passage-winner provenance,
the optional passage ordinal carried by scored rows, multi-passage exact
fallback, and the newer HNSW storage-view and reused-ID repairs. cargo fmt
--all -- --check and git diff --check pass. Current-main tests, builds,
conformance, and benchmarks are intentionally deferred until independent
source audit; historical source-branch verification is labeled as such in
the accompanying report.
@buger
fix(api): prevalidate multi-index semantic mappings
d6c75f7 
A wildcard or comma-list PUT _mapping previously validated and published one
index at a time. If an early target accepted a semantic_text source and
companion but a later target had a conflicting engine-schema field, the
request returned 400 after the earlier schema and raw mapping had already
changed.

Build every target's merged raw mapping and candidate schema delta first,
including derived semantic companions, and reject every deterministic
collision before publishing the first target. The regression deliberately
orders a clean index before a later hidden body_vector:double schema collision
and proves both internal schemas and both GET mapping responses remain
unchanged.

This is intentionally not universal cross-index transactionality. Concurrent
dynamic schema evolution and filesystem persistence failures remain a
separate engine limitation because Index::add_fields is per-index and
Engine::put_index_mapping cannot roll back a previously published target.

This commit replays cf32c955 onto current main afcb3a7 after the companion
publication change. Static composition inspection preserves current passage
provenance and HNSW storage/ID-reuse behavior. cargo fmt --all -- --check and
git diff --check pass. Current-main tests, builds, conformance, and benchmarks
are intentionally deferred until independent source audit; the report labels
the earlier source-branch verification separately.
@buger
fix(semantic): keep passage retrieval exact
309f0b6 
Document-level semantic HNSW pools candidates by one companion vector per source document. That representation cannot prove passage-local correctness when a source also carries <field>_chunks: the passage containing the true answer can lose during document-level pooling even though exact passage scoring would select it.

Track a monotonic per-field passage-exact guard before every source publication path, persist the guard beside HNSW identity metadata, and fail closed for legacy, malformed, or unknown markers. Semantic queries sample the guard around ANN work and route guarded fields to the authoritative exact scan. Rebuilds reconstruct the guard from authoritative sources, and mapping activation over existing documents marks newly semantic vector fields before schema visibility.

Expose guarded fields and blocked semantic ANN fields through HNSW diagnostics. Cover dynamic, explicit nested, raw async/sync, mapping activation, replacement/delete monotonicity, marker compatibility, save/write races, restart persistence, and adversarial passage provenance where document-level ANN omits the exact winner.

FinanceBench remains intentionally exact for the current document-level representation: 5,693 of 6,622 documents (85.97%) and all 19 titles carry passage-local evidence. Accelerating that workload without losing provenance requires passage-node ANN rather than weakening this guard.

Focused passage tests passed before composition onto the generic vector-publication barrier; the integrated stack is reverified after replay.
@buger
fix(semantic): compose raw publication contracts
f7dc4b1 
The semantic companion path and the generic vector-publication barrier both consume validated raw batches. After replaying them together, sync vector ingest could attempt to parse a batch whose semantic preflight had already retained its JSON DOM, tripping the sealed-batch debug assertion.

Reuse the retained parsed values when present and parse only sealed byte-only batches. This preserves the ordinary non-vector no-parse path, keeps vector HNSW publication on the authoritative source, and avoids a second JSON materialization.

Adapt the prerequisite ANN tests to the shared request-deadline signature and run the synchronous passage assertion from a dedicated thread, matching the supported sync-ingest contract instead of invoking it from a Tokio runtime worker.

Verified: cargo test -p xerj-engine raw_vector_publication_tests --no-default-features (7 passed); seven focused semantic companion/passage guard tests (7 passed); cargo fmt --all --check; git diff --check.
@buger
fix(semantic): scope passage reload guard before await
9193ae1 
The HNSW reload path explicitly dropped its standard-library passage-guard write lock before awaiting the graph write lock. Runtime ownership was correct, but Clippy could not prove the guard lifetime ended and rejected the branch under -D warnings for await_holding_lock.

Place passage marker restoration in a lexical scope so the compiler and reviewer can see that the synchronous guard is destroyed before the async graph lock. This is ownership-only clarification; marker union and reload behavior are unchanged.

Verified: xerj-engine library suite 241 passed; no-default-features Clippy with -D warnings passed; release raw-publication tests 7 passed; release passage-focused tests 7 passed; isolated deadline regression 5 passed out of 5; formatting and diff checks clean.
@buger
Delete engine/reports/2026-07-28_semantic-companion-hnsw.md
926fd07
@xerj-org
xerj-org merged commit 7b99e07 into xerj-org:main Jul 29, 2026
4 checks passed
xerj-org added a commit that referenced this pull request Jul 29, 2026
@xerj-org
merge: origin/main into second-brain
1a54d70 
Brings the raw/turbo vector publication ordering fix (PR #67) and the
semantic companion HNSW work (PR #68) under the graph layer. index.rs
auto-merged despite a 1,858-line overlap; graph_expand still 5/5.

Note for the release: es_compat::reindex_pages_past_10k_via_keyset fails
on this box BEFORE and AFTER this merge, and fails identically on
pristine origin/main where CI is green — the documented core-count
sensitivity (ingest shards by worker thread, so a 2-core runner and a
32+-core box lay out segments differently). Pre-existing, not a graph
regression, tracked separately.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@buger @xerj-org