Skip to content

fix: Windows could never boot + autoindex WAL-fd bound, gated by a per-platform CI matrix - #84

Merged
xerj-org merged 4 commits into
mainfrom
fix/autoindex-fd-exhaustion
Jul 31, 2026
Merged

fix: Windows could never boot + autoindex WAL-fd bound, gated by a per-platform CI matrix#84
xerj-org merged 4 commits into
mainfrom
fix/autoindex-fd-exhaustion

Conversation

@xerj-org

@xerj-org xerj-org commented Jul 31, 2026

Copy link
Copy Markdown
Owner

Three things, all tied together by the same root gap: nothing in CI had ever run the binary on a non-Linux platform.

1. The autoindex file-descriptor exhaustion (the original PR)

Each index eagerly opened one WAL file per ingest shard, and num_wal_shards scales with CPU cores — so a repo inferring 400+ datasets held ~6,900 descriptors and died with Too many open files on macOS, whose default soft limit is 256. Raising RLIMIT_NOFILE was not enough: macOS caps the raise below what 400 indices need.

Fix: a persisted per-index index.xerj_ingest_shards setting; xerj autoindex creates every index with a single WAL shard. Measured on a real WooCommerce clone (1.1 GB, 14,185 files): 6,875 → 664 descriptors, 412 datasets indexed, zero EMFILE.

2. A per-platform CI gate — which immediately found something worse

New autoindex-fd-smoke job on [ubuntu, macos, windows]. macOS runners default to ulimit -n 256, the exact condition the bug regressed under, so it reproduces the original failure and gates every future rc.

On its first run it failed on Windows — but not on descriptors. The server could not boot at all:

Error: xerj-console bootstrap
Caused by: internal: create .xerj_users: storage error: I/O error: Access is denied. (os error 5)

3. The Windows fix

xerj_common::fsio::fsync_dir was File::open(dir) + sync_all() with no platform gate. On Windows, opening a directory handle always fails with ERROR_ACCESS_DENIED — std cannot pass FILE_FLAG_BACKUP_SEMANTICS. IndexStore::save_snapshot calls it, so every index creation failed, and the unconditional console bootstrap made that fatal at startup.

It landed in 297be60 (2026-07-12), which means every published Windows binary from rc.4 through rc.8 could not create an index or boot, while xerj.org/get.ps1 kept installing it.

Windows exposes no directory-flush primitive at all (FlushFileBuffers is undefined for directory handles), so the Windows body returns Ok(()). Durability there rests on the callers' file-level sync_all plus NTFS metadata journalling — weaker than the Unix guarantee, and now documented as such instead of silently failing every call.

The smoke script also gained an explicit create-index / index-doc / search round-trip, so the next per-platform break names the failing operation instead of surfacing as an opaque "server exited during boot".

Also here

  • THE MAP contract tests (ux-tests job). The bounded-graph pipeline shipped on two load-bearing claims — "13 groups at any scale" and "byte-identical across runs" — with no automated test in any language. 23 offline cases over four corpus scales now cover the cluster bound, partitioning, link conservation, determinism, the bundle bound, and as-of replay.
  • A test that was measuring the runner's core count. reindex_pages_past_10k_via_keyset fails on any real machine (left: 0, right: 10050) and passes on 2-core CI, because turbo ingest routes docs to memtable shards by worker thread and the test never refreshed. The _reindex product path is unaffected — verified end to end over HTTP at total: 10050, created: 10050, batches: 11, with and without an explicit refresh.

Verification

All eight CI checks green, including the Windows FD smoke that started this. Full workspace suite on the merged tree: 1474 passed, 0 failed on real multi-core hardware. ES-compat conformance: 1360 passed / 0 failed / 3 skipped.

xerj-org added 4 commits July 31, 2026 20:18
…the macOS FD limit

`xerj autoindex` on a large repo (WooCommerce: 413 datasets) still crashed
with "Too many open files (os error 24)" on macOS. The rc.8 RLIMIT_NOFILE
raise was necessary but insufficient: macOS caps the raise (kern.maxfilesperproc
/ OPEN_MAX) below the need, so raising alone cannot win.

Root cause (measured): 99.8% of server fds were per-index WAL files. Each
index eagerly opened one WAL file PER ingest shard, and the ingest-shard count
scales with CPU cores (~8-16). Segments hold zero persistent fds. So 413
datasets x ~16 = ~6,875 WAL fds — past any low macOS ceiling.

Fix: a persisted per-index `index.xerj_ingest_shards` setting threaded to the
store config at create time and honored on reopen. `store_config_from` takes a
per-index WAL-shard override (else the global engine.ingest_shards); es_compat
create_index routes through create_index_with_settings (which now applies index
templates identically, so no template regression) so request settings reach
store-config; `xerj autoindex` creates every index with a single WAL shard.
Routing-safe: WAL replay is disk-self-describing (discovers root and s{N}/
layouts) and the count is persisted-at-create/honored-on-open so writes never
split across layouts.

Verified on the real WooCommerce clone under a 1024-fd cap: peak server fds
6,875 -> 664, full ingestion (37,665 records), 0 EMFILE, docs survive a restart
(WAL layout consistent). Full xerj-engine suite + ES-compat conformance (1360/0)
pass.

Also add a per-OS CI gate (autoindex-fd-smoke on ubuntu/macOS/windows): boots
the binary and autoindexes a 400-dataset corpus under a constrained descriptor
budget (soft 256 / hard 4096, between the pre-fix ~6,400 and post-fix ~600
needs), failing on any EMFILE. macOS runners default to `ulimit -n 256` — the
exact condition this regressed under — so it reproduces the bug and would have
caught it; nothing in CI previously RAN the binary on macOS/Windows.
`fsync_dir` opened the directory with `std::fs::File::open`, which on
Windows always fails with ERROR_ACCESS_DENIED (os error 5) — std cannot
pass FILE_FLAG_BACKUP_SEMANTICS, so a directory handle is unobtainable.
It is called from `IndexStore::save_snapshot`, so *every* index creation
returned an I/O error, and the unconditional xerj-console bootstrap made
that fatal at startup:

    Error: xerj-console bootstrap
    Caused by: internal: create .xerj_users: storage error:
               I/O error: Access is denied. (os error 5)

Every published Windows binary since the durability chain landed in
297be60 (2026-07-12 — shipped in rc.4 through rc.8) has therefore been
unable to create an index or boot at all, while xerj.org/get.ps1 kept
installing it. Nothing caught it because no CI job had ever executed the
binary on Windows; the new per-platform smoke job is what surfaced it.

Windows exposes no directory-flush primitive (FlushFileBuffers is not
defined for directory handles), so the Windows body returns Ok(()).
Durability there rests on the file-level sync_all the callers already
perform plus NTFS metadata journalling of the rename — weaker than the
Unix guarantee, and now documented as such rather than silently failing.

Also assert a create-index / index-doc / search round-trip in the
per-platform smoke script. Autoindex alone would have caught this, but
only as an opaque "server exited during boot"; the explicit round-trip
names the failing operation on whichever platform breaks next.
THE MAP shipped on two load-bearing claims — "13 groups at any scale"
and "byte-identical across runs" — with no automated test in any
language behind either. The only exercise was a manual Puppeteer script
needing a live server, a token argv and a real Chrome, so in practice
nothing checked them.

The clustering pipeline is pure, dependency-free ES module code, so its
contract is checkable offline in ~600ms. 23 cases over four corpus
scales (8 / 12 / 40 / 120 folders — straddling the 12-cluster cap):

  * clusters.length never exceeds MAP_TOP_CLUSTERS + 1, with at most
    one pooled "everything else" body
  * every file-node lands in exactly one group, and the membership
    lists agree with clusterOfFile in both directions
  * fetched links reconcile: Σ merged-link constituents + selfLoops
    equals the input row count, which is the invariant the honesty row
    on the UI is computed from
  * the same rows produce a byte-identical partition across runs
  * bundles stay under the C·(C−1)/2 pairwise bound
  * as-of replay: a link retired before the queried instant counts
    toward the total but not toward live, and replaying to before its
    retirement makes it live again

All pass against the current pipeline — these are regression guards,
not bug reports. Wired as a `ux-tests` job (node --test, no build, no
server).
…ount

`cargo test --workspace` fails on any developer machine with real core
count and passes on CI, which is why nobody has seen it:

  reindex_pages_past_10k_via_keyset
  assertion `left == right` failed: every source doc must be reindexed
    left: 0
   right: 10050

The test seeds via `index_batch_turbo(batch, parallel = true, ..)` and
then asserts on the reindex output without refreshing. Turbo ingest
routes docs to memtable shards BY WORKER THREAD, so on a 2-core runner
every doc lands in one shard and is incidentally visible to the search
the reindex pages over — 10050/10050, green. With real core count the
docs scatter across unpublished memtable shards and the same search
sees nothing at all, so the destination gets 0. The sanity assertion
above it passes either way because `live_doc_count` reads the version
map, not the search surface.

Refreshing the source before the reindex makes the test measure paging
— which is what it is named for — instead of the host's CPU count.
Verified green under both `taskset -c 0,1` and the full core count.

The product path is NOT affected: driving the real HTTP surface end to
end (bulk 10050 docs, POST /_reindex with size 1000) reports
`total: 10050, created: 10050, batches: 11` and the destination counts
10050, both with and without an explicit `_refresh` first. This is the
test reaching under the API, not a reindex defect.

Pre-existing — reproduced on plain main (7e050d0), not introduced by
the FD or hardening work. Same blind spot as the rc.4 doc-values bug:
2-core CI runners cannot see shard-scatter behaviour.
@xerj-org xerj-org changed the title fix(autoindex): bound per-index WAL fds (macOS EMFILE) + per-platform CI gate fix: Windows could never boot + autoindex WAL-fd bound, gated by a per-platform CI matrix Jul 31, 2026
@xerj-org
xerj-org merged commit 053c68c into main Jul 31, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant