perf(embedding): add bounded ONNX throughput controls - #70

Summary

This change adds two opt-in ONNX throughput controls while preserving the current default behavior:

• embedding.onnx_scheduling_window controls how many passages the engine collects before handing one scheduling window to the ONNX backend. Default: 64. Valid range: 1..=4096.
• embedding.onnx_session_pool_size controls how many independent ONNX Runtime sessions are constructed for one shared model configuration. Default: 1. Valid range: 1..=2.

The default remains one serialized ONNX session with a 64-passage caller window. This PR does not change the default lexical embedding mode, select a smaller model, quantize model weights, change the 384-dimensional vector contract, or alter pooling and normalization.

Defaults are subject to change, once all benchmarks are finished.

Why

The backend already grouped similar token lengths into bounded microbatches, but the engine exposed only a source-level 64-passage scheduling window and one serialized ONNX Runtime session. FinanceBench profiling showed that ONNX inference was the dominant ingest stage and that the single session did not keep the available CPU cores busy.

A larger caller window gives the length-aware planner a better population to group. An optional second session allows two complete scheduling windows to execute concurrently. Both controls are bounded and explicit because larger windows retain more passage data and a second session changes the memory/concurrency tradeoff.

Configuration

The existing behavior requires no configuration:

[embedding]
onnx_scheduling_window = 64
onnx_session_pool_size = 1

The measured FinanceBench candidate used:

[embedding]
mode = "onnx-experimental"
onnx_model_path = "/models/all-MiniLM-L6-v2/model.onnx"
onnx_tokenizer_path = "/models/all-MiniLM-L6-v2/tokenizer.json"
onnx_scheduling_window = 512
onnx_session_pool_size = 2
onnx_intra_threads = 8

The session-pool setting is part of the shared backend identity, so handles with different pool sizes cannot accidentally share one initialized backend.

Runtime behavior

• Pool construction reads and verifies the model and tokenizer once, then constructs exactly the requested number of independent sessions.
• Pool construction is atomic. If any session fails to initialize, already-created members are dropped and no partial pool is published.
• A scheduling call leases one session for the entire native inference call.
• With pool size two, the engine launches at most two complete windows together, waits for both native calls to finish, and consumes results in original input order.
• A failed window retains the existing per-document-field retry behavior.
• Admission call and byte permits live inside the blocking closure. Cancelling the async HTTP waiter cannot release capacity or return a session before native inference actually returns.

End-to-end benchmark

Both A/B experiments used one binary per corpus and changed only runtime configuration between the control and candidate. Every run was fresh and passed exact count, map, mapping, ordered hit ID, exact score, selected source, page-local provenance, persisted f32 vector-bit, and restart-equivalence gates.

The FB20 control runs were 84.330 s and 87.973 s. The candidate runs were 73.764 s and 76.266 s. Both candidate runs were faster than both controls.

The causal number for this PR's dual-session comparison is 12.93% lower FB20 end-to-end time, not 23.61%. The 23.61% figure compares the 75.015 s candidate with an older 98.198 s W64 observation from a different experiment epoch. It is useful historical context for the combined W64-to-W512-plus-dual progression, but it is not a controlled same-binary attribution and is not used as the headline.

Correctness evidence

FB4 preserved all 2,190 × 384 persisted f32 vectors with aggregate SHA-256 73a9c9596724895fc639221d8e8ac56bff7bed019a23aee7395fe98f870aac6f.

FB20 preserved all 4,553 × 384 persisted f32 vectors with aggregate SHA-256 eb8dd923f34f026eeff68729810f6f083cf3886d883d6720e631c8804cc5250b.

FB20 also preserved the 20-path to 18-unique-content to 17-accepted-document accounting, two duplicate aliases, the pinned extraction rejection, exact company/year/quarter routing, grounded lexical and semantic answers, and identical results after restart.

Resource tradeoffs

The memory result is mixed, not a universal improvement. The second session remains opt-in because FB4 server RSS increased materially and the narrower same-timestamp combined measurement increased on both corpora. Final disk usage was effectively unchanged on FB4 and increased 0.46% on FB20.

Benchmark provenance

• Validated prototype commit: 64afad2404fb04238869d411187edda0d0e83e95
• Validated prototype tree: 47d28dd6216b47d61d38b27eabc4730ddec1b04a
• Validated binary SHA-256: e40ed27468e0db084e6eb58df2874ea31c6aeb8daf4f4e7e864189bc1d9e5859
• FB4 report: /workspace/.tmp/fb4-dual-session-ab-run-20260729/RESULTS.md
• FB4 binding SHA-256: 7a522b87f8f4736826e34ff95f7e55550479461b6f071d16646577081d4baaf9
• FB4 raw-results SHA-256: 0159cc82d886d0c27bcb430ef79f6f429149f1410c8118c963721f4328b9b01d
• FB20 report: /workspace/.tmp/fb20-dual-session-ab-run-20260729/RESULTS.md
• FB20 binding SHA-256: 9b28515b1634ecedbfc03f0b793d25f68dbe0d3cbe47ae21f39d27253e5eec13
• FB20 raw-results SHA-256: 0d488bf11c98fe524a77d3ff02fedc20d0d77b225c0e2e45ff77bf0329340f28
• Host page cache state: mixed/uncontrolled in both experiments

The contribution branch is a minimal extraction onto current main; it intentionally does not include the prototype's unrelated phase-attribution stack. The benchmark hashes above therefore identify the exact measured prototype, while the focused tests below cover the extracted production behavior.

Tests

• cargo fmt --all --check
• cargo test -p xerj-ai --features onnx-experimental — 40 passed / 1 ignored
• explicit real-asset two-session vector-bit equivalence test with the bound 87 MiB model — passed
• cargo test -p xerj-common config::tests:: — 14 passed
• cargo test -p xerj-engine --features onnx-experimental semantic_embedding_window_tests:: — focused order/window tests passed
• cargo clippy -p xerj-ai --all-targets --features onnx-experimental -- -D warnings
• cargo clippy -p xerj-common --all-targets -- -D warnings
• cargo clippy -p xerj-engine --lib --features onnx-experimental -- -D warnings
• exact-commit release server plus full ES-YAML suite — 1360 passed / 0 failed / 3 skipped

Limitations

• Pool size two is not the default because its memory behavior depends on corpus and workload shape.
• The measured model was FP32 all-MiniLM-L6-v2-compatible ONNX. This PR does not claim the same speedup for arbitrary operator-supplied models.
• The A/B covers 4- and 20-PDF FinanceBench subsets, not the full 368-PDF corpus and not TB scale.
• The observed improvement is approximately 13%, not 20%, 50%, or 10x end to end.
• A larger window improves scheduling opportunity but can retain more passage text and vectors until a call completes.