Tags: xerj-org/xerj
Tags
v1.0.0-rc.12 Smaller indexes, faster full-text queries, and a scoring fix that made results depend on machine topology. doc_values on the mapping is now honoured. On a 500k-doc benchmark corpus the index went 154,413,167 -> 99,424,772 bytes and force-merge 139.13s -> 59.69s. The saving removes 90-99% of the doc-values sidecar, which is 2-37% of index size depending on how large your text bodies are relative to _source. The per-segment FTS reader is cached instead of rebuilt per query: match on text 166 -> 58 ms, prefix 171 -> 69, wildcard 181 -> 75, fuzzy 306 -> 195, match_phrase 496 -> 340. BM25 statistics are now index-wide rather than per scoring arm. Previously an overwritten document could move from last place to first, and segment-resident scores moved 3.2x and 13.3x with segment count alone, so the same corpus and query returned different scores depending on how many CPUs were present at flush time. THIS CHANGES ABSOLUTE _score VALUES on any index with more than one live scoring arm; re-tune any pinned numeric min_score. Also: autoindex no longer aborts an entire run on one multibyte SQL statement. ES-YAML 1365 passed / 0 failed / 3 skipped. Full suite 1983 passed / 0 failed. Known issues: #191 (tied scores are not broken deterministically), #193 (min_score uses per-arm scores at size:0; scalar N drops ghosts).
xerj v1.0.0-rc.11 Eight commits since v1.0.0-rc.10. The version bump lands in this tag, so the binary reports rc.11 — v1.0.0-rc.10 was tagged before its bump and every published rc.10 binary reports rc.9. - autoindex pins the server's embedding execution identity (#160) - memtable circuit breaker reads the cgroup memory limit, not host RAM (#163) - skip tombstone-only segments in the agg corpus (#161) - reproducible fused FP32 ONNX model recipe (#154) - reference docs: second brain, scripting, snapshots, security model (#165) - contribution review discipline (#168) - bound the publication test barriers so a missed point fails fast (#158) Held from this release: the #170 and #172 autoindex extractor fixes, blocked on #178.
xerj v1.0.0-rc.10 Six defects found in a pre-release review and fixed before this cut. A client disconnect during a document PUT could permanently poison an index: the publication guard was held across the only await in the interval, so a dropped request left it Pending and every later read and write on that index failed until the process restarted. The guard is now taken before the interval, which contains no await at all. The Console data-sources proxy reached reserved second-brain namespaces. Its filter matched .xerj_ and missed .xerj-memory-, and the Console router is merged outside the authz layer, so any authenticated session of any role could read another tenant's brain. Snapshot and restore reached another tenant. Wildcards were waved through before grants were consulted, and neither verb passes the visibility funnel, so a non-superuser restore over .xerj-memory-* rolled every tenant's brain back to the backup instant. Scripted updates ran unbounded Painless work. The per-request budget covered _search only, so cost multiplied by statement count and again by hit count, with wait_for_completion=false detaching it entirely. Columnar aggregations dropped a segment's documents. Coverage was judged across the segment set rather than per segment, so a terms aggregation could silently omit an array-suppressed segment and return a plausible smaller number. Thanks to @Nicolas0315 for four measured performance reports, tracked for the release after this one.
xerj v1.0.0-rc.9 — the cross-platform correctness release The Windows binaries published as rc.4 through rc.8 could not start: fsync_dir was File::open(dir) + sync_all() with no platform gate, and obtaining a directory handle that way always fails on Windows with ERROR_ACCESS_DENIED, so every index creation errored and the console bootstrap made that fatal at boot. Nothing caught it because no CI job had ever run the binary anywhere except Ubuntu. A Windows runner now boots the server, writes a document, reads it back and autoindexes 400 datasets on every pull request. Also closes the self-contained half of the post-audit security backlog (#73 snapshot escape, #80 index-name boundary, #76 field limit, magic-link redemption and cluster/info disclosure, #71 ONNX windowing memory), plus a create-time file-descriptor exhaustion reachable from an index-create body, found while writing the tests rc.8 shipped without. Test coverage grew from 1,420 to 1,486 Rust test functions, with 23 offline tests for THE MAP's bounded-graph claims and a CI gate that runs the second-brain, MCP and autoindex use-case harnesses that were manual until now. Writing those tests found four more real bugs, all fixed here: inline script/style contents indexed as prose, non-deterministic XML record election, a core dump on `xerj … | head`, and a test that had been measuring the CI runner's core count rather than the code. Full suite 1,489 passed / 0 failed on real multi-core hardware — it could not pass at all before this release. Conformance 1360 / 0 / 3. Held out deliberately, each with a posted review: #85, #86, #87, #88,
fix(server): raise RLIMIT_NOFILE with a concrete rlim_max so macOS ac… …cepts it rc.8 raised the FD limit by passing the inherited hard limit straight back as rlim_max, but macOS's launchd hard limit is RLIM_INFINITY and setrlimit(RLIMIT_NOFILE) rejects that outright. The raise was therefore a silent no-op on macOS: the soft limit stayed at 256 and `xerj autoindex` still crashed with "Too many open files" near the start on large trees (reported against a 413-dataset, 1.4 GB WooCommerce clone; ~16 descriptors per inferred index). Pass a concrete rlim_max (== rlim_cur), stepping down for the macOS kern.maxfilesperproc ceiling until one is accepted. Extracted the planning into a pure nofile_plan() and unit-tested the RLIM_INFINITY path, which cannot be exercised at runtime on Linux (the kernel caps the NOFILE hard limit, so it is never actually infinite there).
fix(autoindex): bound docx decompression + never index dotfiles/secrets Two findings from the dev-second-brain use-case experiment (brain arm), verified against the rc.7 binary. F-A1 (DoS, corpus-planting) — a crafted .docx whose `word/document.xml` inflates to hundreds of MB inside one paragraph made autoindex peak at 1.68 GB RSS from an 815 KB file: `para` only flushed at `</w:p>`, and a single giant `<w:t>` run was materialized whole by quick-xml. Fix: read `document.xml` through a `Take` that caps the DECOMPRESSED stream at 72 MiB (aligned with the existing 64 MiB body cap — reading more is discarded anyway), plus bound `para` growth. Measured after the fix: the same 400 MB-inflate bomb peaks at 187 MB and is now bounded BY the cap, not the bomb size (a 40 MB-inflate variant peaks lower, at 123 MB — proving the cap engages). F-A2 (secret exposure) — the walker had no ignore rules and happily indexed `.env` (API tokens), `.git`, `.ssh`, `.aws` and other dotfiles into a queryable brain that has no per-brain authorization. For the "point it at my project folder" use case that is a real exposure. Fix: `filter_entry` prunes hidden files and directories before descending (so `.git/` is skipped whole), with the root exempt so a brain over a dot-named folder still works. Two regression tests. The remaining per-brain-authorization gap (any authenticated caller can read any brain) is a larger design item, tracked separately. fmt + clippy clean; xerj-autoindex 127/127.
v1.0.0-rc.5 — the real-client compatibility release Query classes that silently matched zero documents now return what ES returns: booleans, .keyword multi-fields, match_phrase on arrays and on scalar query values, and keyword arrays before a flush. A real Kibana or OpenSearch Dashboards instance boots, logs in and saves objects against XERJ end to end, and a dashboard firing several panel queries at once no longer stalls the node. ES-YAML conformance holds at 1360 passed / 0 failed / 3 skipped. Known limits are stated in the changelog rather than implied: the keyword-array fix covers the memtable half only, and three zero-hit defects found while validating this release are listed unfixed.
PreviousNext