Skip to content

fix(search): tied scores — constant_score pages a mixed corpus like match_all, and every re-sort uses the one page-order key - #300

Merged
xerj-org merged 1 commit into
mainfrom
fix/issue-270-tied-score-resorts
Aug 11, 2026
Merged

fix(search): tied scores — constant_score pages a mixed corpus like match_all, and every re-sort uses the one page-order key#300
xerj-org merged 1 commit into
mainfrom
fix/issue-270-tied-score-resorts

Conversation

@xerj-org

Copy link
Copy Markdown
Owner

Fixes #270. Refs #191 (closes the constant_score exclusion recorded there), refs #267, refs #188.

Both defects reproduced at origin/main (c21d27a) with the new tests before the fix.

1. constant_score on a mixed corpus (bounded page had no segment documents)

40 flushed + 300 unflushed identical docs, {"constant_score":{"filter":{"match_all":{}}}}:

size:1  → ["mem000"]     (want ["seg0000"]; sizes 1/5/41/60/100/150/199 all non-prefix)
size:1000 and hits.total → correct

Root cause is upstream of the two hydrate_prefiltered_unsorted breaks the issue pointed at. The wrapper defeats is_match_all, and try_shortcut_count has no arm for a bare MatchAll (MatchAll totals are normally handled by the flag), so count_authoritative was false and the stored scan ran in exact-counting mode: it tallies past a full collector but never materialises into it. With the memtable walked first, the bounded page was all memtable documents. The flagged breaks are gated on count_authoritative, get per-segment headroom (scan_limit), and feed the #267 cross-segment re-merge — they were unreachable in the failing configuration and are untouched (zero new work on the B7 hot loop).

Fix: is_match_all_effective peels top-level constant_score/boosted chains for the count/bounds decisions only (count_authoritative, the final live_doc_count() total overwrite). Scoring keeps the raw flag; a top-level Constant's scores are overwritten with the wrapper boost before the page sort regardless.

2. Five post-sort re-sorts tied by _id alone

The bool-text IDF rescore, the near-zero-BM25 TF-IDF fallback, and the three request.rescore sorts. Any of them firing on a tied hit set reordered the page into _id ASC. Measured for the issue's own repro (40 flushed + 600 unflushed, {"match":{"body":"listpack"}}): every score exactly 1.0 (bits 1065353216) via the TF-IDF fallback — at ~640 identical docs BM25 IDF collapses under the 0.001 threshold and the fallback's tf.sqrt()*(1+ln(N/df)) is exactly 1.0 — page mem0000… before seg0000….

Fix: the main sort's decorate-sort-undecorate is hoisted into Index::sort_hits_page_order (score DESC, seq_no ASC, _id ASC) and the main default sort plus all five re-sorts route through it. Prior art (retrieved via the xerj-search reference corpus; approach only, no code copied): Lucene HitQueue.java:76-82; tantivy collector/sort_key/sort_by_score.rs:105-109 (heap) vs :187 (final sort) — same key both places; quickwit quickwit-search/src/collector.rs:1131-1156 ties by GlobalDocAddress in both comparators.

Tests (all fail at HEAD, pass here)

  • tests/tied_score_resorts.rs (new): TF-IDF fallback, bool-text IDF rescore, chained-rescore — full-materialisation arrival order + bounded-prefix where the design supports it.
  • constant_score added to the mixed-corpus case in tests/tied_score_page_order.rs (HEAD: size:1["mem000"]).
  • Known residual, documented in the bool-text test rather than asserted around: bounded pages for multi-clause bools on a mixed corpus are selected under first-pass scores, where the memtable's uniform tf-sum differs from segment BM25 (BM25 length normalisation is per-arm: overwriting a document moves it from last to first #188's remit) — admission happens before any re-sort, so no sort change can fix it.

Gate

  • Full xerj-engine suite: green.
  • ES-YAML conformance on a private port (9475) with a throwaway data dir: 1366 passed / 0 failed / 3 skipped.
  • cargo fmt --check clean; cargo clippy -p xerj-engine --all-targets clean.

Perf (uncached — query_cache cleared per iteration; 25k identical docs, size:10, N=200, release, interleaved base/fixed binaries)

shape base fixed
constant_score(match_all), 20k+5k mixed, cold-cache order 22.6 ms 10.4 ms
same, warm-cache order 22.6 ms 3.9 ms
match_all mixed 8.6 ms 8.7 ms
term mixed, identical cache state 4.63 ms 4.60 ms
fully-flushed fixture (B7 hot path), all shapes unchanged unchanged
rescore (three extra ~page-sized decorated sorts) 2.41 ms 2.50 ms

Forward-order runs showed apparent term/match_all swings; reversing the measurement order showed those were cache-warming side effects of the preceding constant_score no longer doing an O(N) scan — not per-shape regressions.

🤖 Generated with Claude Code

…atch_all, and every re-sort uses the one page-order key (fixes #270)

Two defects from #270, both reproduced at origin/main (c21d27a) with the
new tests before the fix.

1) constant_score on a mixed corpus returned a bounded page with no
   segment documents: 40 flushed + 300 unflushed identical docs,
   {"constant_score":{"filter":{"match_all":{}}}} at size:1 returned
   ["mem000"] where the full page starts ["seg0000"] (sizes 1/5/41/60/
   100/150/199 all violated the prefix property; totals were correct).

   Root cause is upstream of the early-outs the issue pointed at: the
   wrapper defeated `is_match_all`, and `try_shortcut_count` has no arm
   for a bare MatchAll (MatchAll totals are normally handled by the
   flag), so `count_authoritative` was FALSE and the stored scan ran in
   exact-counting mode — it tallies past a full collector but never
   materialises into it. With the memtable walked first, the bounded
   page was all memtable documents. The two breaks in
   `hydrate_prefiltered_unsorted` the issue flagged are gated on
   `count_authoritative` and get per-segment headroom (`scan_limit`)
   plus the #267 cross-segment re-merge, so they were never reachable
   in the failing configuration and are untouched.

   Fix: `is_match_all_effective` peels top-level `constant_score`/
   `boosted` chains for the count/bounds decisions only
   (`count_authoritative`, the final `live_doc_count()` total
   overwrite). Scoring keeps the raw flag; a top-level Constant's
   scores are overwritten with the wrapper boost before the page sort
   regardless.

2) Five post-sort re-sorts tied by `_id` alone — the bool-text IDF
   rescore, the near-zero-BM25 TF-IDF fallback, and the three
   `request.rescore` sorts — so any of them firing on a tied hit set
   reordered the page into `_id` ASC (measured: 40 flushed + 600
   unflushed, {"match":{"body":"listpack"}} → every score exactly 1.0
   via the TF-IDF fallback, page mem0000…mem0599 before seg0000…;
   the fallback fires at ~640 identical docs because BM25 IDF collapses
   to ln(1+0.5/(N+0.5)) < 0.001, and its recomputed score
   tf.sqrt()*(1+ln(N/df)) is exactly 1.0 for identical docs).

   Fix: hoist the main sort's decorate-sort-undecorate into
   `Index::sort_hits_page_order` (score DESC, seq_no ASC, _id ASC) and
   route the main default sort and all five re-sorts through it. Peer
   engines keep one comparator through collection, merge and final
   sort: Lucene HitQueue.java:76-82; tantivy
   collector/sort_key/sort_by_score.rs:105-109 (heap) vs :187 (final
   sort), same key; quickwit quickwit-search/src/collector.rs:1131-1156
   ties by GlobalDocAddress in both comparators (all read via the
   xerj-search reference corpus; no code copied).

Tests: tests/tied_score_resorts.rs (new; TF-IDF fallback, bool-text
IDF rescore, chained-rescore — all three failed at HEAD) and
constant_score added to the mixed-corpus case in
tests/tied_score_page_order.rs (failed at HEAD: size:1 → ["mem000"]).
The bool-text case asserts the full materialisation only: bounded
pages for multi-clause bools on a mixed corpus are selected under
first-pass scores where the memtable's uniform tf-sum differs from
segment BM25 — a scoring divergence (#188's remit) that admission hits
before any re-sort, documented in the test.

Gate: full xerj-engine suite green; ES-YAML conformance
1366 passed / 0 failed / 3 skipped on a private port (9475) with a
throwaway data dir.

Perf (uncached, query_cache cleared per iteration, 25k identical docs,
size:10, 200 iters, release; interleaved base/fixed runs):
- constant_score(match_all), 20k flushed + 5k unflushed:
  22.6 → 10.4 ms/query cold-cache order, 22.6 → 3.9 ms warm-cache
  order (the bounded scan replaces the full counting scan).
- match_all mixed: 8.6 → 8.7 ms (unchanged); term mixed measured in
  identical cache state: 4.63 → 4.60 ms (unchanged); fully-flushed
  fixture (the B7 hot path): all shapes unchanged (match_all 0.04,
  term 1.1, rescore 2.0-2.1 ms both builds). Apparent forward-order
  term/match_all deltas were cache-warming side effects of the
  preceding shape's scan no longer being O(N) — verified by reversed
  measurement order.
- rescore adds three ~page-sized decorated sorts: +0.1 ms at 256 hits.

Refs #191 (this closes the constant_score exclusion recorded there),
refs #267, refs #188 (bool-text bounded-page residual).
@cla-bot cla-bot Bot added the cla-signed label Aug 11, 2026
@xerj-org
xerj-org merged commit 95d9820 into main Aug 11, 2026
12 checks passed
xerj-org added a commit that referenced this pull request Aug 11, 2026
Cut from main at 83971fd after the pre-cut PR sweep. Nineteen pull
requests had merged since v1.0.0-rc.15 when the sweep started; five more
were open and every one got an explicit decision:

- merged after review: #305 (autoindex journal-lock release, fork-inherited
  flock), #307 (dynamic-mapping #292 close-out), #309 (default _search
  omits embedding companions), #308 (CLA co-author gate, CI-only),
  #306 (landing hero, already deployed)
- excluded: #274 (author-marked WIP, failing checks), #258 (in-progress
  #204 sweep, failing check)

Review residuals from #305/#307/#309 are filed as #310, #311, #312 and
disclosed in their CHANGELOG entries rather than left in review text.

CHANGELOG: the [Unreleased] section moves under 1.0.0-rc.16 (2026-08-11).
Eight merged PRs had filed no entry — #296, #300, #287, #284, #285/#304,
#286, #305 — written here, alongside the #309 entry expanded with its
copy-path guarantees and disclosed residuals, and the #292/#307 entry
extended with the field-budget tightening (multi-fields now count, matching
ES total_fields semantics). Both rc.15 known issues (progress-stream
forgery, outer-.gitignore reach into nested checkouts) are fixed in this
release and the section header says so. Structural repairs: duplicate
### Changed heading merged, two missing blank lines restored (one eaten
next to the rc.10 heading by #309's diff).

Version: engine workspace 1.0.0-rc.15 -> 1.0.0-rc.16; Cargo.lock refreshed
via scoped cargo check -p xerj-common.

Tag follows only after CI and the 8-target Release matrix both report
success on this exact commit.
xerj-org added a commit that referenced this pull request Aug 12, 2026
…build on release staleness (#298)

Issue #298 pins ROADMAP.md as the authoritative "where the project is
going" document and advertises that parts of it are machine-checked. At
HEAD (31d1fd5, the `release: cut 1.0.0-rc.16` commit) the file had
drifted from every promise the issue makes:

- "Last reviewed: … against v1.0.0-rc.15" while CHANGELOG.md already
  records 1.0.0-rc.16 as cut.
- "## Next release — v1.0.0-rc.16" described the release that had just
  shipped: five PRs listed "in flight" (#290 #281 #261 #296 #297) had
  merged; #258 alone remains open.
- Closed issues listed as open roadmap items: #293 (fixed by #304),
  and GA gate entries #270 (PR #300), #269 (PR #308), #207 (PR #261),
  plus the closed #204 umbrella.
- The ES-YAML count read 1,365/1,368 against a measured 1,366/1,369.

All 13 existing docs_capability_lists tests were green the whole time:
they pin capability counts, but nothing tied the roadmap to the release
history, so a cut that forgets to roll the roadmap forward passed CI.

Fix, in the same shape as the #211 capability-list guard:

- Two new tests in docs_capability_lists.rs pin ROADMAP.md to
  CHANGELOG.md's latest `## [<version>]` heading:
  `the_roadmap_next_release_has_not_already_been_cut` (the next-release
  heading must name a version strictly newer than the latest cut) and
  `the_roadmap_review_line_names_the_latest_cut_release` (the review
  line must name the latest cut release). Both FAIL before this
  ROADMAP.md refresh (verified: 13 passed / 2 failed at HEAD) and pass
  after it. Cutting rc.18 without re-reviewing ROADMAP.md now fails the
  build.
- ROADMAP.md re-reviewed against v1.0.0-rc.16 and main: rc.16 recorded
  as cut with the roadmap items it retired; next-release section now
  points at the v1.0.0-rc.17 milestone with the real open set (#294
  carried over; residuals #310 #311 #312; PRs #258 #313 #274 in
  flight); GA gate list drops the shipped items (#270 #269 #207) and
  reflects the closed #204 umbrella; conformance count updated.
- The same stale 1,365/1,368 count fixed in README.md (badge + prose),
  landing/llms.txt and landing/llms-full.txt. The fresh number traces
  to a verified run: CI job 93861837729 on 31d1fd5 — "1366 passed ·
  0 failed · 3 skipped · 1369 total".

GitHub-side, per the issue's "every open issue is triaged onto a
milestone": milestone v1.0.0-rc.17 created, #294/#310/#311/#312 moved
onto it, milestone v1.0.0-rc.16 closed.

Gate: cargo fmt --check clean, clippy clean, docs_capability_lists
15 passed / 0 failed. No engine runtime code is touched (the new file
is test-only), so the ES-YAML suite is unchanged from the green run
above; PR CI re-runs it.

Files:
- engine/crates/xerj-engine/tests/docs_capability_lists.rs (release
  freshness section, end of file)
- ROADMAP.md, README.md, landing/llms.txt, landing/llms-full.txt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tied scores: constant_score on a mixed corpus, and five post-sort re-sorts that tie by _id alone

1 participant