Two performance changes and two correctness fixes, all measured.

Smaller indexes

doc_values on the mapping is now honoured. It used to be accepted, echoed back by GET _mapping, and ignored — aggregating and sorting on a "doc_values": false field both succeeded, where Elasticsearch errors. The sidecar builder was schema-free and filed a column for every _source string it saw.

The default now follows ES: no doc-values for text, annotated_text, match_only_text, search_as_you_type, semantic_text, binary, object and nested. An explicit "doc_values": true on a text field still builds the column and still returns whole-value buckets.

On a 500k-document corpus with one analyzed text field, force-merged to a single segment:

The .dv artifact falls 95.9%; every other artifact is byte-identical. Force-merge gets 2.33x faster because merge re-encodes the sidecar and there are 55 MB fewer bytes to build.

How much you get depends on your corpus. This removes 90–99% of the doc-values sidecar, and that sidecar is anywhere from 2% to 37% of index size depending on how large your text bodies are relative to _source. It was 35.6% on the corpus above and 6.97% across a 4.2 GB set of source-code indexes, where stored _source dominates instead. Do not expect 1.553x everywhere.

A terms aggregation on a text field also got 1.34x faster (3414 → 2547 ms) by losing its column, because walking a column whose every entry is a whole document body was slower than the brute path it now falls back to.

Faster full-text queries

The per-segment FTS reader is cached instead of rebuilt on every query. FtsIndexReader::open sat inside the per-segment search loop and performed two zstd decompressions per field — the whole .post blob and the whole .meta array — into owned buffers, then discarded them. On one production field that was about 50 ms and 41 MB per (segment, field), per query.

Readers are now cached per (segment, field-set), charged against the segment hydration budget under a new fts_reader category visible in _nodes/stats, and evicted at merge completion. Segments are immutable so nothing needs invalidating; if the budget refuses the charge the reader is returned uncached and behaviour falls back to the previous path rather than growing without bound.

Measured p50 on 500k documents with the query cache disabled:

Reproduced across three runs to within 2%. Brute-scan families (function_score, boosting) are unchanged, because their cost is scanning and scoring rather than opening a reader.

BM25 scores are now index-wide

This changes absolute _score values. Read this section before upgrading if you record or compare scores.

Overwriting a document with identical content could move it from last place to first. Statistics were per scoring arm: the memtable normalised against only its own documents, and each segment against only its own. A document alone in the memtable was length-normalised against itself, so the long-field penalty that made it low-scoring disappeared.

Concretely, a document that should have been near the bottom came back first at 1.6x the true top hit, purely because it had been re-indexed.

Statistics are now folded once per search across every segment plus the memtable and applied to all arms, which is what Lucene does. Ranking within a single-arm index is unchanged and pinned by a test, but any index with more than one live arm will see absolute scores move. That is the fix working: scores stop depending on flush timing.

For scale: adversarial review measured 3.2x and 13.3x score drift from segment count alone on the old behaviour. If you pin a numeric min_score, re-tune it. _explain now reports index-wide N and document frequency.

Known limits, all tracked:

• Multi-term expansions (prefix, wildcard, fuzzy, and the trailing term of match_phrase_prefix) still use local document frequency, because their term sets resolve per arm.
• The union falls back to per-arm statistics on very wide queries (fields x terms x segments above 4096).
• size:0 with min_score and no aggregations still filters on per-arm scores while size:N uses index-wide, so the same threshold means two different things depending on size (#193).
• The scalar document count excludes ghosts on the new path where the old one included them, slightly raising IDF on indices with tombstoned or superseded versions (#193).

Also fixed

xerj autoindex no longer aborts an entire run on one file. A byte-length guard was followed by a character-boundary slice, so any SQL-dump statement whose 12th byte fell inside a multi-byte character panicked — and because the workspace builds with panic = "abort", one bad file discarded every other file's work.

A test-only self-deadlock on tokio's blocking pool that made CI fail intermittently on 2-core runners.

Known issues

Tied scores are not broken deterministically: a bounded page and a full page can disagree about which of several exactly-equal documents is returned (#191). Index-wide statistics make exact ties more common, so this is more visible than before.

Verification

ES-YAML conformance 1365 passed / 0 failed / 3 skipped. Full workspace suite 1983 passed / 0 failed / 30 ignored. Benchmark harness and raw results are in demo/playbooks/rc12/; re-run with demo/playbooks/rc12/measure_rc12.sh <label> 500000.

What's Changed

• test(engine): bound the publication test barriers so a missed point fails fast by @xerj-org in #158
• chore(release): bump the workspace version to 1.0.0-rc.10 by @xerj-org in #159
• fix(engine): skip tombstone-only segments in agg corpus by @buger in #161
• perf(onnx): add reproducible fused FP32 model recipe by @buger in #154
• docs: reference pages for the second brain, scripting, snapshots and the security model by @xerj-org in #165
• fix(engine): memtable circuit breaker ignores cgroup memory limit by @Vinz2168 in #163
• Feat/embedding execution identity by @buger in #160
• docs: codify contribution review discipline by @buger in #168

Full Changelog: v1.0.0-rc.10...v1.0.0-rc.11

What's Changed

• fix(api): treat OpenSearch's flat_object as an alias for flattened by @Vinz2168 in #86
• fix(engine): parse_date_ms accepts no-colon numeric zone offsets by @Vinz2168 in #91
• feat(engine): add local functions and lambdas to Painless by @Vinz2168 in #88
• docs(readme): rewrite for humans, install first, video instead of the fast GIF by @xerj-org in #94
• feat(engine): support getDayOfWeekEnum().getDisplayName() in Painless by @Vinz2168 in #92
• fix(api): enforce the real ES runtime-field type allowlist by @xerj-org in #106
• fix(engine): fast_aggs resolves .keyword to its parent's column by @Vinz2168 in #99
• fix(api): send X-OpenSearch-Version on responses to OpenSearch callers by @Vinz2168 in #101
• fix(api): merge keyed (object-shaped) bucket aggs across multi-index search by @Vinz2168 in #103
• security(console): key the auth rate limiter on the socket, not a header by @xerj-org in #112
• fix(query): resolve named ES date formats for ignore_malformed validation by @Vinz2168 in #89
• security(cluster): authenticate control frames, fail closed when unconfigured by @xerj-org in #113
• fix(engine): fast_aggs bails to brute force for unresolvable nested fields by @Vinz2168 in #104
• fix(api): apply the request-time script guard on _msearch, and bound the bucket_script evaluator ahead of #95 by @xerj-org in #115
• fix(painless): surface script resource-limit trips instead of returning a wrong score by @xerj-org in #116
• fix(engine): resolve <field>.keyword in fast_aggs predicates instead of matching nothing by @xerj-org in #114
• docs(engine): correct the record on why the bucket_script expression bound exists by @xerj-org in #117
• test(engine): remove the process-wide state and the residual races from the engine suite by @xerj-org in #118
• feat(query): add script query support by @Vinz2168 in #87
• perf(aggs): bound the zone-offset probe to a six-byte window at the end by @xerj-org in #119
• fix(api): one script-guard definition for every search entry point by @xerj-org in #124
• fix(engine): bucket_script resolves a bare-string _count buckets_path by @Vinz2168 in #105
• fix(aggs): the columnar histogram paths honour config.limits.max_buckets by @xerj-org in #125
• fix(api): surface script resource-limit trips on _rank_eval, _explain and pivot transforms by @xerj-org in #126
• docs(changelog): cut the 1.0.0-rc.10 section by @xerj-org in #127
• fix(engine): bail to brute for multi-field suffixes at any depth by @xerj-org in #129
• fix(engine): query_string with no default_field searches every text field, bounded and deadline-aware by @xerj-org in #130
• security(graph): make a brain a real per-tenant boundary, incl. detached ML-datafeed tasks by @xerj-org in #131
• fix(server): pin prod tokio worker stack to 4 MiB; correct the Painless headroom doc by @xerj-org in #136
• fix(engine): cap exec_terms on the columnar fast path, agreeing with brute by @xerj-org in #134
• fix(engine): Fix flush publication failure and cancellation recovery by @buger in #135
• fix(painless): per-evaluation work budget to bound the flat-script CPU DoS by @xerj-org in #133
• fix(engine): bail to brute for uncolumned nested-object and suppressed-array fields by @xerj-org in #139
• fix(engine): transactional merge publication and rollback by @xerj-org in #140
• fix(engine): serialize collection publication for stable reads by @xerj-org in #141
• fix(console): dashboard panels never authenticate to the data plane by @Vinz2168 in #142
• fix(console): keep the data-sources proxy out of the brain namespace by @xerj-org in #149
• fix(engine): keep document publication free of cancellation and error poison by @xerj-org in #150
• security(snapshot): stop snapshot and restore from reaching another tenant by @xerj-org in #152
• security(api): bound the Painless work one scripted update may do by @xerj-org in #153
• docs(changelog): record the rc.10 security and correctness fixes by @xerj-org in #155
• fix(engine): judge columnar coverage per segment, not across the set (#143) by @xerj-org in #151

Full Changelog: v1.0.0-rc.9...v1.0.0-rc.10

What's Changed

• security: Phase-2 hardening — close #73/#80/#71, advance #76 by @xerj-org in #83
• fix: Windows could never boot + autoindex WAL-fd bound, gated by a per-platform CI matrix by @xerj-org in #84
• release: cut v1.0.0-rc.9 — cross-platform correctness, plus the test coverage that found four more bugs by @xerj-org in #90

Full Changelog: v1.0.0-rc.8...v1.0.0-rc.9

What's Changed

• fix(console): show only the surfaces the engine has data for (brain-only launch) by @xerj-org in #81
• fix(autoindex): raise RLIMIT_NOFILE (Too many open files crash) + tree-sitter AST code extraction by @xerj-org in #82

Full Changelog: v1.0.0-rc.7...v1.0.0-rc.8

What's Changed

• fix(console): show only the surfaces the engine has data for (brain-only launch) by @xerj-org in #81
• fix(autoindex): raise RLIMIT_NOFILE (Too many open files crash) + tree-sitter AST code extraction by @xerj-org in #82

Full Changelog: v1.0.0-rc.7...v1.0.0-rc.8

What's Changed

• fix(engine): Fix raw and turbo vector publication ordering by @buger in #67
• fix(engine) Publish semantic companion vectors to HNSW without weakening passage correctness by @buger in #68
• fix(security): close 6 network-reachable DoS/path-traversal vectors (Phase 1) by @therandomsecurityguy in #69
• perf(embedding): add bounded ONNX throughput controls by @buger in #70

New Contributors

• @therandomsecurityguy made their first contribution in #69

Full Changelog: v1.0.0-rc.6...v1.0.0-rc.7

What's Changed

• fix(engine): Fix raw and turbo vector publication ordering by @buger in #67
• fix(engine) Publish semantic companion vectors to HNSW without weakening passage correctness by @buger in #68
• fix(security): close 6 network-reachable DoS/path-traversal vectors (Phase 1) by @therandomsecurityguy in #69
• perf(embedding): add bounded ONNX throughput controls by @buger in #70

New Contributors

• @therandomsecurityguy made their first contribution in #69

Full Changelog: v1.0.0-rc.6...v1.0.0-rc.7

What's Changed

• fix(api): decompress gzip-encoded request bodies (breaks real ES clients incl. Filebeat) by @Vinz2168 in #57
• feat(storage) Add bounded selected-row and field projection for stored V2 sections by @buger in #58
• fix(engine) Bound Painless expression depth before evaluation by @buger in #59
• feat(cache): Bound all segment hydration caches with one process-wide budget by @buger in #56
• feat(engine): kNN + aggregations in a single request (rc.6) + calltree.ai case study by @xerj-team in #60
• docs(example): hybrid retrieval for RAG chatbots (drop-in for pgvector + BM25) by @xerj-team in #62

• feat(search): return the winning semantic passage by @buger in #63

• refactor(vector): read HNSW through a static storage view by @buger in #65
• docs(case-study): auto-replicate Postgres → XERJ (CDC) + hybrid search (daily.dev) by @xerj-team in #61
• Merge PR #64: HNSW reused-ID entry fix, resolved against the storage-view refactor by @xerj-org in #66
• fix(vector): Fix HNSW entry metadata and stale hits when an external ID is reused by @buger in #64

New Contributors

• @xerj-org made their first contribution in #66

Full Changelog: v1.0.0-rc.5...v1.0.0-rc.6

What's Changed

• Fix/autoindex text family split by @xerj-team in #3
• feat(ai): prototype bounded ONNX embeddings by @buger in #2
• docs(demo): real AI-agent usage scenarios; deprecate stale sales kit by @xerj-team in #4
• feat(ai): wire experimental ONNX through autoindex by @buger in #5
• docs: verified example — embed via any OpenAI-compatible API by @xerj-team in #6
• docs: XERJ + Google AI — EmbeddingGemma (verified), Gemini API, ADK by @xerj-team in #7
• fix(api): register missing HTTP verbs on several ES-compat endpoints by @Vinz2168 in #8
• fix(api): field_caps resolves real glob wildcards, not just bare */_all by @Vinz2168 in #10
• fix(api): /_xpack reports real auth state instead of hardcoded true by @Vinz2168 in #9
• fix(api): _update_by_query/_delete_by_query honor wait_for_completion=false by @Vinz2168 in #11
• fix(index): keep semantic vectors out of full-text indexes by @buger in #12
• fix(api): _bulk reports real per-item _seq_no/_version, not a timestamp by @Vinz2168 in #13
• fix(api): add POST /_index_template/_simulate (body-only template preview) by @Vinz2168 in #14
• research: AST + graph + FTS for AI vulnerability finding at scale by @xerj-team in #15
• fix(api): add GET /_security/privilege (application privileges) by @Vinz2168 in #16
• fix(api): add POST /_security/profile/_activate (fixes Kibana login 500) by @Vinz2168 in #17
• fix(onnx): make lazy session initialization cancellation-safe by @buger in #18
• fix(autoindex): parse PDFs in isolated workers, and use pdf_oxide for more complex PDF parsing by @buger in #20
• fix(autoindex): make bulk request timeout configurable by @buger in #22
• fix(api): accept HTTP Basic auth (Kibana's interactive login realm) by @Vinz2168 in #26
• fix(api): add GET/POST /_security/user/_has_privileges by @Vinz2168 in #25
• feat(api): auto-sense OpenSearch vs Elasticsearch clients per request by @Vinz2168 in #28
• fix(api): complete OpenSearch Dashboards compatibility (product header + _cat/templates/{pattern}) by @Vinz2168 in #29
• Fix/semantic search deadline by @buger in #19
• fix(engine): serialize single-document publication by id by @buger in #27
• fix(autoindex): deduplicate content with crash-safe replacement by @buger in #21
• fix(engine): dynamic mapping infers ISO date strings as date, not text by @Vinz2168 in #24
• fix(api): resolve wildcard/comma index specs on 9 more endpoints by @Vinz2168 in #23
• fix(api): resolve aliases on single-index endpoints; persist aliases across restart by @Vinz2168 in #30
• fix(api): explicit top-level _source param wins over stored_fields suppression by @Vinz2168 in #31
• fix(query): match_phrase/match_phrase_prefix accept scalar (bool/number) query values by @Vinz2168 in #32
• fix(engine): term/match_phrase queries on boolean fields undercounted to 0 by @Vinz2168 in #33
• fix(storage): bound per-shard WAL buffers to 64 KiB by @buger in #34
• fix(api): GET /_xpack respects --compat-version, was hardcoded 8.13.0 by @Vinz2168 in #35
• fix(api): POST /{index}/_update never returned a get block by @Vinz2168 in #36
• fix(api): GET _field_caps never listed declared multi-fields by @Vinz2168 in #37
• fix(engine): geo_point aggs/queries reject string-encoded lat/lon by @Vinz2168 in #38
• Prevent malformed raw documents and source split-brain before WAL publication by @buger in #39
• perf(engine): full-corpus agg path deep-cloned the memtable under lock by @Vinz2168 in #42
• perf(engine): full-corpus agg path re-decoded segments on every query by @Vinz2168 in #43
• fix(engine): term/match_phrase on any .keyword multi-field matched 0 docs by @Vinz2168 in #44
• fix(query): empty match/match_phrase/match_phrase_prefix query -> 400 by @Vinz2168 in #45
• fix(engine): match_phrase/match_phrase_prefix matched 0 on array fields by @Vinz2168 in #46
• fix(engine): terms aggregation ignores script, always returns empty buckets by @Vinz2168 in #50
• docs: WordPress security-audit case study + AST-graph vulnerability detectors by @xerj-team in #41
• fix(engine): Painless doc['field'].value has no date accessor methods by @Vinz2168 in #51
• fix(engine): term/terms on keyword ARRAYS match only the first element by @xerj-team in #40
• perf(engine): concurrent Kibana dashboard bursts stalled the scan path by @Vinz2168 in #48
• fix(server): admin API key regenerates on every restart instead of persisting by @Vinz2168 in #52
• feat(server): add private pprof debugging toolkit by @buger in #53
• Feat/ingest memory instrumentation by @buger in #54

New Contributors

• @xerj-team made their first contribution in #3
• @buger made their first contribution in #2
• @Vinz2168 made their first contribution in #8

Full Changelog: v1.0.0-rc.4...v1.0.0-rc.5

Full Changelog: v1.0.0-rc.3...v1.0.0-rc.4

Full Changelog: v1.0.0-rc.2...v1.0.0-rc.3

Full Changelog: v1.0.0-rc.2...v1.0.0-rc.3

Full Changelog: v1.0.0-rc.2...v1.0.0-rc.3