fix: Windows could never boot + autoindex WAL-fd bound, gated by a per-platform CI matrix - #84
Merged
Merged
Conversation
…the macOS FD limit
`xerj autoindex` on a large repo (WooCommerce: 413 datasets) still crashed
with "Too many open files (os error 24)" on macOS. The rc.8 RLIMIT_NOFILE
raise was necessary but insufficient: macOS caps the raise (kern.maxfilesperproc
/ OPEN_MAX) below the need, so raising alone cannot win.
Root cause (measured): 99.8% of server fds were per-index WAL files. Each
index eagerly opened one WAL file PER ingest shard, and the ingest-shard count
scales with CPU cores (~8-16). Segments hold zero persistent fds. So 413
datasets x ~16 = ~6,875 WAL fds — past any low macOS ceiling.
Fix: a persisted per-index `index.xerj_ingest_shards` setting threaded to the
store config at create time and honored on reopen. `store_config_from` takes a
per-index WAL-shard override (else the global engine.ingest_shards); es_compat
create_index routes through create_index_with_settings (which now applies index
templates identically, so no template regression) so request settings reach
store-config; `xerj autoindex` creates every index with a single WAL shard.
Routing-safe: WAL replay is disk-self-describing (discovers root and s{N}/
layouts) and the count is persisted-at-create/honored-on-open so writes never
split across layouts.
Verified on the real WooCommerce clone under a 1024-fd cap: peak server fds
6,875 -> 664, full ingestion (37,665 records), 0 EMFILE, docs survive a restart
(WAL layout consistent). Full xerj-engine suite + ES-compat conformance (1360/0)
pass.
Also add a per-OS CI gate (autoindex-fd-smoke on ubuntu/macOS/windows): boots
the binary and autoindexes a 400-dataset corpus under a constrained descriptor
budget (soft 256 / hard 4096, between the pre-fix ~6,400 and post-fix ~600
needs), failing on any EMFILE. macOS runners default to `ulimit -n 256` — the
exact condition this regressed under — so it reproduces the bug and would have
caught it; nothing in CI previously RAN the binary on macOS/Windows.
`fsync_dir` opened the directory with `std::fs::File::open`, which on
Windows always fails with ERROR_ACCESS_DENIED (os error 5) — std cannot
pass FILE_FLAG_BACKUP_SEMANTICS, so a directory handle is unobtainable.
It is called from `IndexStore::save_snapshot`, so *every* index creation
returned an I/O error, and the unconditional xerj-console bootstrap made
that fatal at startup:
Error: xerj-console bootstrap
Caused by: internal: create .xerj_users: storage error:
I/O error: Access is denied. (os error 5)
Every published Windows binary since the durability chain landed in
297be60 (2026-07-12 — shipped in rc.4 through rc.8) has therefore been
unable to create an index or boot at all, while xerj.org/get.ps1 kept
installing it. Nothing caught it because no CI job had ever executed the
binary on Windows; the new per-platform smoke job is what surfaced it.
Windows exposes no directory-flush primitive (FlushFileBuffers is not
defined for directory handles), so the Windows body returns Ok(()).
Durability there rests on the file-level sync_all the callers already
perform plus NTFS metadata journalling of the rename — weaker than the
Unix guarantee, and now documented as such rather than silently failing.
Also assert a create-index / index-doc / search round-trip in the
per-platform smoke script. Autoindex alone would have caught this, but
only as an opaque "server exited during boot"; the explicit round-trip
names the failing operation on whichever platform breaks next.
THE MAP shipped on two load-bearing claims — "13 groups at any scale"
and "byte-identical across runs" — with no automated test in any
language behind either. The only exercise was a manual Puppeteer script
needing a live server, a token argv and a real Chrome, so in practice
nothing checked them.
The clustering pipeline is pure, dependency-free ES module code, so its
contract is checkable offline in ~600ms. 23 cases over four corpus
scales (8 / 12 / 40 / 120 folders — straddling the 12-cluster cap):
* clusters.length never exceeds MAP_TOP_CLUSTERS + 1, with at most
one pooled "everything else" body
* every file-node lands in exactly one group, and the membership
lists agree with clusterOfFile in both directions
* fetched links reconcile: Σ merged-link constituents + selfLoops
equals the input row count, which is the invariant the honesty row
on the UI is computed from
* the same rows produce a byte-identical partition across runs
* bundles stay under the C·(C−1)/2 pairwise bound
* as-of replay: a link retired before the queried instant counts
toward the total but not toward live, and replaying to before its
retirement makes it live again
All pass against the current pipeline — these are regression guards,
not bug reports. Wired as a `ux-tests` job (node --test, no build, no
server).
…ount
`cargo test --workspace` fails on any developer machine with real core
count and passes on CI, which is why nobody has seen it:
reindex_pages_past_10k_via_keyset
assertion `left == right` failed: every source doc must be reindexed
left: 0
right: 10050
The test seeds via `index_batch_turbo(batch, parallel = true, ..)` and
then asserts on the reindex output without refreshing. Turbo ingest
routes docs to memtable shards BY WORKER THREAD, so on a 2-core runner
every doc lands in one shard and is incidentally visible to the search
the reindex pages over — 10050/10050, green. With real core count the
docs scatter across unpublished memtable shards and the same search
sees nothing at all, so the destination gets 0. The sanity assertion
above it passes either way because `live_doc_count` reads the version
map, not the search surface.
Refreshing the source before the reindex makes the test measure paging
— which is what it is named for — instead of the host's CPU count.
Verified green under both `taskset -c 0,1` and the full core count.
The product path is NOT affected: driving the real HTTP surface end to
end (bulk 10050 docs, POST /_reindex with size 1000) reports
`total: 10050, created: 10050, batches: 11` and the destination counts
10050, both with and without an explicit `_refresh` first. This is the
test reaching under the API, not a reindex defect.
Pre-existing — reproduced on plain main (7e050d0), not introduced by
the FD or hardening work. Same blind spot as the rc.4 doc-values bug:
2-core CI runners cannot see shard-scatter behaviour.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three things, all tied together by the same root gap: nothing in CI had ever run the binary on a non-Linux platform.
1. The autoindex file-descriptor exhaustion (the original PR)
Each index eagerly opened one WAL file per ingest shard, and
num_wal_shardsscales with CPU cores — so a repo inferring 400+ datasets held ~6,900 descriptors and died withToo many open fileson macOS, whose default soft limit is 256. RaisingRLIMIT_NOFILEwas not enough: macOS caps the raise below what 400 indices need.Fix: a persisted per-index
index.xerj_ingest_shardssetting;xerj autoindexcreates every index with a single WAL shard. Measured on a real WooCommerce clone (1.1 GB, 14,185 files): 6,875 → 664 descriptors, 412 datasets indexed, zero EMFILE.2. A per-platform CI gate — which immediately found something worse
New
autoindex-fd-smokejob on[ubuntu, macos, windows]. macOS runners default toulimit -n 256, the exact condition the bug regressed under, so it reproduces the original failure and gates every future rc.On its first run it failed on Windows — but not on descriptors. The server could not boot at all:
3. The Windows fix
xerj_common::fsio::fsync_dirwasFile::open(dir)+sync_all()with no platform gate. On Windows, opening a directory handle always fails withERROR_ACCESS_DENIED— std cannot passFILE_FLAG_BACKUP_SEMANTICS.IndexStore::save_snapshotcalls it, so every index creation failed, and the unconditional console bootstrap made that fatal at startup.It landed in
297be60(2026-07-12), which means every published Windows binary from rc.4 through rc.8 could not create an index or boot, whilexerj.org/get.ps1kept installing it.Windows exposes no directory-flush primitive at all (
FlushFileBuffersis undefined for directory handles), so the Windows body returnsOk(()). Durability there rests on the callers' file-levelsync_allplus NTFS metadata journalling — weaker than the Unix guarantee, and now documented as such instead of silently failing every call.The smoke script also gained an explicit create-index / index-doc / search round-trip, so the next per-platform break names the failing operation instead of surfacing as an opaque "server exited during boot".
Also here
ux-testsjob). The bounded-graph pipeline shipped on two load-bearing claims — "13 groups at any scale" and "byte-identical across runs" — with no automated test in any language. 23 offline cases over four corpus scales now cover the cluster bound, partitioning, link conservation, determinism, the bundle bound, and as-of replay.reindex_pages_past_10k_via_keysetfails on any real machine (left: 0, right: 10050) and passes on 2-core CI, because turbo ingest routes docs to memtable shards by worker thread and the test never refreshed. The_reindexproduct path is unaffected — verified end to end over HTTP attotal: 10050, created: 10050, batches: 11, with and without an explicit refresh.Verification
All eight CI checks green, including the Windows FD smoke that started this. Full workspace suite on the merged tree: 1474 passed, 0 failed on real multi-core hardware. ES-compat conformance: 1360 passed / 0 failed / 3 skipped.