# feat(search): return the winning semantic passage - #63

Why

XERJ already scores a long semantic_text document by its best matching generated chunk, but the search response does not tell the caller which chunk produced the score. An agent must therefore receive or re-read the complete source document to recover the evidence behind a hit. This is especially costly for page-oriented PDF indices, where the desired response is usually the relevant passage plus the document and page identity.

What changes

This change records compact provenance for generated semantic chunks and exposes the winning passage only when the caller requests the _passage pseudo-field.

• Ingest persists the original semantic field name and UTF-8 byte-offset pairs beside its derived chunk vectors.
• Exact chunk scoring retains the winning zero-based chunk ordinal and uses a deterministic lowest-ordinal tie-break.
• Response assembly slices the winning text from the authoritative source only for returned hits, so passage text is not stored twice.
• Elasticsearch-compatible search returns the result as fields._passage[0].
• Native search accepts fields: ["_passage"] and returns the result as hit-level _passage.
• A numeric source page is copied into the passage result, preserving page-local PDF provenance.
• Scalar, array, and Elasticsearch object forms of fields are normalized consistently.
• Scroll continuations preserve opted-in passage provenance.
• Engine-owned metadata is rejected in caller writes and hidden from _source, GET, field discovery, full-text indexing, aggregations, and Painless scripts.
• Update, delete, flush, merge, restart, and source-filtering paths retain correct provenance.

The response object contains:

{
  "field": "body",
  "ordinal": 2,
  "start_offset": 896,
  "end_offset": 1408,
  "text": "The covenant remains in effect through the fourth quarter...",
  "page": 17
}

Offsets are UTF-8 byte offsets into the original semantic field. page is omitted when the source does not contain a numeric page.

Usage

No mapping change is required beyond the existing semantic_text field:

curl -sS -X PUT "$XERJ_URL/reports" -H 'content-type: application/json' -d '{
  "mappings": {
    "properties": {
      "body": {"type": "semantic_text"},
      "file": {"type": "keyword"},
      "page": {"type": "integer"}
    }
  }
}'

Index normally, then opt in through fields:

BODY="$(printf 'Routine quarterly operating context. %.0s' {1..80})The covenant remains in effect through the fourth quarter."

curl -sS -X PUT "$XERJ_URL/reports/_doc/q4?refresh=true" -H 'content-type: application/json' -d "$(jq -n --arg body "$BODY" '{
  body: $body,
  file: "acme-2025-q4.pdf",
  page: 17
}')"

curl -sS -X POST "$XERJ_URL/reports/_search" -H 'content-type: application/json' -d '{
  "fields": ["_passage"],
  "_source": {"includes": ["file", "page"]},
  "query": {
    "semantic": {
      "field": "body",
      "query": "What covenant applies in the fourth quarter?",
      "k": 10
    }
  }
}'

The same request may use "fields": "_passage" or "fields": [{"field": "_passage"}]. Ordinary searches that omit _passage preserve their previous response shape and do not materialize passage text.

For native search, add "fields": ["_passage"] to the request. Native hits contain _passage directly instead of the Elasticsearch-compatible fields._passage array.

Honest scope and limitations

• This PR exposes provenance for the existing exact per-chunk max-sim scoring path; it does not introduce passage-node HNSW or make passage scoring approximate.
• _passage is valid when one semantic clause owns the score, or when one kNN clause targets a generated semantic chunk-vector field. An arbitrary dense-vector kNN field has no generated passage metadata, so _passage may be absent. Multi-kNN and hybrid fusion combine independent contributions and do not have one honest winning passage, so those requests return an actionable HTTP 400 telling the caller to use one eligible semantic/kNN clause or omit _passage.
• The built-in default embedder remains lexical feature hashing. Neural semantics still require explicitly enabling the neural embedding mode or configuring an embedding endpoint.
• This adds provenance metadata, not another copy of the passage text. The original source and existing generated vectors remain the authoritative data.
• Short single-passage values continue to use their existing single-vector behavior.

Storage measurement

The committed deterministic measurement uses 256 varied Unicode documents with variable passage counts. It compares identical stored-source batches with and without the field-name/offset metadata and also verifies that each authoritative text occurs exactly once.

Command:

cd engine
cargo test -p xerj-engine index::chunk_embed_tests::varied_passage_metadata_reports_raw_and_stored_overhead -- --exact --nocapture

Observed output:

varied passage metadata delta: raw=20547 bytes (80.26/doc), stored=1019 bytes (3.98/doc)

This is a controlled metadata-overhead measurement, not an end-to-end index-size claim.

Manual reproduction

1. Start XERJ against a throwaway data directory and create an index with a long semantic_text field plus numeric page.
2. Index a value longer than one chunk with a distinctive phrase in a later passage and use refresh=true.
3. Search for that phrase with "fields": ["_passage"].
4. Confirm that fields._passage[0].text is the exact source slice bounded by start_offset and end_offset, ordinal identifies the winning generated passage, field names the original semantic field, and page matches the source.
5. Repeat the search without fields and confirm _passage is absent.
6. Flush, restart XERJ on the same data directory, repeat the opted-in search, and confirm the same passage is returned.
7. Update the document with a new distinctive phrase and page number, search again, and confirm the old passage is no longer returned.
8. Request _passage from a hybrid or multi-kNN query and confirm HTTP 400 includes the concrete alternatives: use one semantic/kNN clause or omit _passage for fusion.

The repository integration test performs the update, flush/merge, restart, source-filter, Unicode-offset, and response-hiding sequence deterministically:

cd engine
cargo test -p xerj-engine semantic_passage_provenance_survives_update_merge_restart_and_source_filter -- --exact --nocapture

The router tests exercise all accepted fields forms, scroll continuation, both multi-kNN clause orders, and both hybrid child orders:

cd engine
cargo test -p xerj-api passage_

Validation

Current branch base and fetched upstream main are both 761b47dfcb5f8fcc57bc929385c422bcffbe130b; no rebase changed production code.

• cargo test -p xerj-query: 125 passed, 0 failed; 1 doc-test ignored
• cargo test -p xerj-engine passage_ -- --test-threads=1: 10 passage tests passed across unit and integration suites, 0 failed
• cargo test -p xerj-api passage_: 6 passed, 0 failed
• cargo clippy -p xerj-query -p xerj-engine -p xerj-api --all-targets -- -D warnings: passed
• cargo fmt --all --check: passed
• git diff --check: passed
• Release ES-YAML hard gate on this exact branch/base: 1360 passed, 0 failed, 3 skipped

The earlier full package runs on this exact branch/base also passed: 227 engine unit tests, 10 chaos tests, 65 ES integration tests, 101 engine integration tests with 1 ignored, and 61 API tests.