fix(search): tied scores — constant_score pages a mixed corpus like match_all, and every re-sort uses the one page-order key - #300
Merged
Conversation
…atch_all, and every re-sort uses the one page-order key (fixes #270) Two defects from #270, both reproduced at origin/main (c21d27a) with the new tests before the fix. 1) constant_score on a mixed corpus returned a bounded page with no segment documents: 40 flushed + 300 unflushed identical docs, {"constant_score":{"filter":{"match_all":{}}}} at size:1 returned ["mem000"] where the full page starts ["seg0000"] (sizes 1/5/41/60/ 100/150/199 all violated the prefix property; totals were correct). Root cause is upstream of the early-outs the issue pointed at: the wrapper defeated `is_match_all`, and `try_shortcut_count` has no arm for a bare MatchAll (MatchAll totals are normally handled by the flag), so `count_authoritative` was FALSE and the stored scan ran in exact-counting mode — it tallies past a full collector but never materialises into it. With the memtable walked first, the bounded page was all memtable documents. The two breaks in `hydrate_prefiltered_unsorted` the issue flagged are gated on `count_authoritative` and get per-segment headroom (`scan_limit`) plus the #267 cross-segment re-merge, so they were never reachable in the failing configuration and are untouched. Fix: `is_match_all_effective` peels top-level `constant_score`/ `boosted` chains for the count/bounds decisions only (`count_authoritative`, the final `live_doc_count()` total overwrite). Scoring keeps the raw flag; a top-level Constant's scores are overwritten with the wrapper boost before the page sort regardless. 2) Five post-sort re-sorts tied by `_id` alone — the bool-text IDF rescore, the near-zero-BM25 TF-IDF fallback, and the three `request.rescore` sorts — so any of them firing on a tied hit set reordered the page into `_id` ASC (measured: 40 flushed + 600 unflushed, {"match":{"body":"listpack"}} → every score exactly 1.0 via the TF-IDF fallback, page mem0000…mem0599 before seg0000…; the fallback fires at ~640 identical docs because BM25 IDF collapses to ln(1+0.5/(N+0.5)) < 0.001, and its recomputed score tf.sqrt()*(1+ln(N/df)) is exactly 1.0 for identical docs). Fix: hoist the main sort's decorate-sort-undecorate into `Index::sort_hits_page_order` (score DESC, seq_no ASC, _id ASC) and route the main default sort and all five re-sorts through it. Peer engines keep one comparator through collection, merge and final sort: Lucene HitQueue.java:76-82; tantivy collector/sort_key/sort_by_score.rs:105-109 (heap) vs :187 (final sort), same key; quickwit quickwit-search/src/collector.rs:1131-1156 ties by GlobalDocAddress in both comparators (all read via the xerj-search reference corpus; no code copied). Tests: tests/tied_score_resorts.rs (new; TF-IDF fallback, bool-text IDF rescore, chained-rescore — all three failed at HEAD) and constant_score added to the mixed-corpus case in tests/tied_score_page_order.rs (failed at HEAD: size:1 → ["mem000"]). The bool-text case asserts the full materialisation only: bounded pages for multi-clause bools on a mixed corpus are selected under first-pass scores where the memtable's uniform tf-sum differs from segment BM25 — a scoring divergence (#188's remit) that admission hits before any re-sort, documented in the test. Gate: full xerj-engine suite green; ES-YAML conformance 1366 passed / 0 failed / 3 skipped on a private port (9475) with a throwaway data dir. Perf (uncached, query_cache cleared per iteration, 25k identical docs, size:10, 200 iters, release; interleaved base/fixed runs): - constant_score(match_all), 20k flushed + 5k unflushed: 22.6 → 10.4 ms/query cold-cache order, 22.6 → 3.9 ms warm-cache order (the bounded scan replaces the full counting scan). - match_all mixed: 8.6 → 8.7 ms (unchanged); term mixed measured in identical cache state: 4.63 → 4.60 ms (unchanged); fully-flushed fixture (the B7 hot path): all shapes unchanged (match_all 0.04, term 1.1, rescore 2.0-2.1 ms both builds). Apparent forward-order term/match_all deltas were cache-warming side effects of the preceding shape's scan no longer being O(N) — verified by reversed measurement order. - rescore adds three ~page-sized decorated sorts: +0.1 ms at 256 hits. Refs #191 (this closes the constant_score exclusion recorded there), refs #267, refs #188 (bool-text bounded-page residual).
xerj-org
added a commit
that referenced
this pull request
Aug 11, 2026
Cut from main at 83971fd after the pre-cut PR sweep. Nineteen pull requests had merged since v1.0.0-rc.15 when the sweep started; five more were open and every one got an explicit decision: - merged after review: #305 (autoindex journal-lock release, fork-inherited flock), #307 (dynamic-mapping #292 close-out), #309 (default _search omits embedding companions), #308 (CLA co-author gate, CI-only), #306 (landing hero, already deployed) - excluded: #274 (author-marked WIP, failing checks), #258 (in-progress #204 sweep, failing check) Review residuals from #305/#307/#309 are filed as #310, #311, #312 and disclosed in their CHANGELOG entries rather than left in review text. CHANGELOG: the [Unreleased] section moves under 1.0.0-rc.16 (2026-08-11). Eight merged PRs had filed no entry — #296, #300, #287, #284, #285/#304, #286, #305 — written here, alongside the #309 entry expanded with its copy-path guarantees and disclosed residuals, and the #292/#307 entry extended with the field-budget tightening (multi-fields now count, matching ES total_fields semantics). Both rc.15 known issues (progress-stream forgery, outer-.gitignore reach into nested checkouts) are fixed in this release and the section header says so. Structural repairs: duplicate ### Changed heading merged, two missing blank lines restored (one eaten next to the rc.10 heading by #309's diff). Version: engine workspace 1.0.0-rc.15 -> 1.0.0-rc.16; Cargo.lock refreshed via scoped cargo check -p xerj-common. Tag follows only after CI and the 8-target Release matrix both report success on this exact commit.
xerj-org
added a commit
that referenced
this pull request
Aug 12, 2026
…build on release staleness (#298) Issue #298 pins ROADMAP.md as the authoritative "where the project is going" document and advertises that parts of it are machine-checked. At HEAD (31d1fd5, the `release: cut 1.0.0-rc.16` commit) the file had drifted from every promise the issue makes: - "Last reviewed: … against v1.0.0-rc.15" while CHANGELOG.md already records 1.0.0-rc.16 as cut. - "## Next release — v1.0.0-rc.16" described the release that had just shipped: five PRs listed "in flight" (#290 #281 #261 #296 #297) had merged; #258 alone remains open. - Closed issues listed as open roadmap items: #293 (fixed by #304), and GA gate entries #270 (PR #300), #269 (PR #308), #207 (PR #261), plus the closed #204 umbrella. - The ES-YAML count read 1,365/1,368 against a measured 1,366/1,369. All 13 existing docs_capability_lists tests were green the whole time: they pin capability counts, but nothing tied the roadmap to the release history, so a cut that forgets to roll the roadmap forward passed CI. Fix, in the same shape as the #211 capability-list guard: - Two new tests in docs_capability_lists.rs pin ROADMAP.md to CHANGELOG.md's latest `## [<version>]` heading: `the_roadmap_next_release_has_not_already_been_cut` (the next-release heading must name a version strictly newer than the latest cut) and `the_roadmap_review_line_names_the_latest_cut_release` (the review line must name the latest cut release). Both FAIL before this ROADMAP.md refresh (verified: 13 passed / 2 failed at HEAD) and pass after it. Cutting rc.18 without re-reviewing ROADMAP.md now fails the build. - ROADMAP.md re-reviewed against v1.0.0-rc.16 and main: rc.16 recorded as cut with the roadmap items it retired; next-release section now points at the v1.0.0-rc.17 milestone with the real open set (#294 carried over; residuals #310 #311 #312; PRs #258 #313 #274 in flight); GA gate list drops the shipped items (#270 #269 #207) and reflects the closed #204 umbrella; conformance count updated. - The same stale 1,365/1,368 count fixed in README.md (badge + prose), landing/llms.txt and landing/llms-full.txt. The fresh number traces to a verified run: CI job 93861837729 on 31d1fd5 — "1366 passed · 0 failed · 3 skipped · 1369 total". GitHub-side, per the issue's "every open issue is triaged onto a milestone": milestone v1.0.0-rc.17 created, #294/#310/#311/#312 moved onto it, milestone v1.0.0-rc.16 closed. Gate: cargo fmt --check clean, clippy clean, docs_capability_lists 15 passed / 0 failed. No engine runtime code is touched (the new file is test-only), so the ES-YAML suite is unchanged from the green run above; PR CI re-runs it. Files: - engine/crates/xerj-engine/tests/docs_capability_lists.rs (release freshness section, end of file) - ROADMAP.md, README.md, landing/llms.txt, landing/llms-full.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #270. Refs #191 (closes the
constant_scoreexclusion recorded there), refs #267, refs #188.Both defects reproduced at
origin/main(c21d27a) with the new tests before the fix.1.
constant_scoreon a mixed corpus (bounded page had no segment documents)40 flushed + 300 unflushed identical docs,
{"constant_score":{"filter":{"match_all":{}}}}:Root cause is upstream of the two
hydrate_prefiltered_unsortedbreaks the issue pointed at. The wrapper defeatsis_match_all, andtry_shortcut_counthas no arm for a bareMatchAll(MatchAll totals are normally handled by the flag), socount_authoritativewas false and the stored scan ran in exact-counting mode: it tallies past a full collector but never materialises into it. With the memtable walked first, the bounded page was all memtable documents. The flagged breaks are gated oncount_authoritative, get per-segment headroom (scan_limit), and feed the #267 cross-segment re-merge — they were unreachable in the failing configuration and are untouched (zero new work on the B7 hot loop).Fix:
is_match_all_effectivepeels top-levelconstant_score/boostedchains for the count/bounds decisions only (count_authoritative, the finallive_doc_count()total overwrite). Scoring keeps the raw flag; a top-levelConstant's scores are overwritten with the wrapper boost before the page sort regardless.2. Five post-sort re-sorts tied by
_idaloneThe bool-text IDF rescore, the near-zero-BM25 TF-IDF fallback, and the three
request.rescoresorts. Any of them firing on a tied hit set reordered the page into_idASC. Measured for the issue's own repro (40 flushed + 600 unflushed,{"match":{"body":"listpack"}}): every score exactly 1.0 (bits 1065353216) via the TF-IDF fallback — at ~640 identical docs BM25 IDF collapses under the 0.001 threshold and the fallback'stf.sqrt()*(1+ln(N/df))is exactly 1.0 — pagemem0000…beforeseg0000….Fix: the main sort's decorate-sort-undecorate is hoisted into
Index::sort_hits_page_order(score DESC, seq_no ASC, _id ASC) and the main default sort plus all five re-sorts route through it. Prior art (retrieved via thexerj-searchreference corpus; approach only, no code copied): LuceneHitQueue.java:76-82; tantivycollector/sort_key/sort_by_score.rs:105-109(heap) vs:187(final sort) — same key both places; quickwitquickwit-search/src/collector.rs:1131-1156ties byGlobalDocAddressin both comparators.Tests (all fail at HEAD, pass here)
tests/tied_score_resorts.rs(new): TF-IDF fallback, bool-text IDF rescore, chained-rescore — full-materialisation arrival order + bounded-prefix where the design supports it.constant_scoreadded to the mixed-corpus case intests/tied_score_page_order.rs(HEAD:size:1→["mem000"]).Gate
xerj-enginesuite: green.cargo fmt --checkclean;cargo clippy -p xerj-engine --all-targetsclean.Perf (uncached —
query_cachecleared per iteration; 25k identical docs, size:10, N=200, release, interleaved base/fixed binaries)Forward-order runs showed apparent
term/match_allswings; reversing the measurement order showed those were cache-warming side effects of the precedingconstant_scoreno longer doing an O(N) scan — not per-shape regressions.🤖 Generated with Claude Code