feat(ai): wire experimental ONNX through autoindex - #5

What this changes

This is the end-to-end integration follow-up to merged
#2.

PR #2 deliberately landed the reviewable foundation:

• process-wide Candle model sharing;
• an opt-in xerj-ai FP32 ONNX Runtime backend;
• safe single-session inference;
• bounded, length-aware/token-budgeted microbatching;
• focused scheduler/model-sharing tests and a benchmark example;
• honest experimental documentation.

It did not expose ONNX through the server or autoindex.

This follow-up makes that already-merged backend usable through XERJ's real
agent-first workflow:

start XERJ with an explicit ONNX model
  -> xerj autoindex <folder>
  -> xerj autoindex map
  -> semantic query

Beyond #2, an explicitly built GNU/Linux glibc server can now:

• select onnx-experimental through CLI, environment, or TOML configuration;
• lazily activate the backend only for schemas containing semantic_text;
• batch semantic embeddings during ordinary bulk/autoindex ingestion;
• persist the vector-space identity with every ONNX semantic index;
• reject incompatible restarts, writes, mappings, or supplied vectors;
• share global call/byte admission with every compatible index handle;
• surface overload as retryable HTTP 429;
• preserve autoindex journal correctness and resume unfinished files;
• expose the mode honestly through server help, autoindex help, and generated
  data maps;
• keep ordinary startup logs quiet while emitting one auditable first-use
  activation event.

This remains experimental and off by default:

• XERJ's default is still the built-in deterministic lexical feature hash. It
  does not become neural because this PR lands.
• --embed-mode neural still selects the existing Candle backend.
• ONNX requires an explicit build feature, runtime mode, local model path, and
  matching tokenizer path.
• XERJ does not download the ONNX model automatically.
• The bundled ONNX Runtime artifacts do not support XERJ's musl targets.

The Candle-sharing implementation and the underlying ONNX inference/scheduler
are background supplied by merged #2, not new code introduced by this
follow-up. Their verified measurements remain below because they explain the
integration's motivation and runtime choices.

Copy-paste end-to-end workflow

1. Export the exact supported model shape

This is deliberately not a generic "run any ONNX model" interface. Export
the FP32 sentence-transformers/all-MiniLM-L6-v2 feature-extraction graph:

python3 -m venv /tmp/xerj-onnx-export
/tmp/xerj-onnx-export/bin/pip install 'optimum[onnxruntime]' onnx

/tmp/xerj-onnx-export/bin/optimum-cli export onnx \
  --model sentence-transformers/all-MiniLM-L6-v2 \
  --task feature-extraction \
  /tmp/xerj-minilm-onnx

/tmp/xerj-onnx-export/bin/python - <<'PY'
import onnx

path = "/tmp/xerj-minilm-onnx/model.onnx"
model = onnx.load(path)
print("inputs:", [
    (value.name, [dim.dim_value for dim in value.type.tensor_type.shape.dim])
    for value in model.graph.input
])
print("outputs:", [
    (value.name, [dim.dim_value for dim in value.type.tensor_type.shape.dim])
    for value in model.graph.output
])
PY

sha256sum \
  /tmp/xerj-minilm-onnx/model.onnx \
  /tmp/xerj-minilm-onnx/tokenizer.json

The interface must contain:

• input_ids: int64;
• attention_mask: int64;
• token_type_ids: int64;
• either last_hidden_state or token_embeddings;
• a rank-three token embedding output;
• output width exactly 384;
• tokenizer.json from the same model/export.

XERJ applies attention-mask mean pooling and row-wise L2 normalization itself
and truncates at 512 tokens. A semantic mapping declaring a width other than
384 is rejected. Unknown input/output names or a different model shape are not
guessed.

2. Build and start the opt-in server

cd engine
cargo build --release -j 32 -p xerj-server --features onnx-experimental

target/release/xerj \
  --insecure \
  --data-dir /tmp/xerj-onnx-data \
  --embed-mode onnx-experimental \
  --onnx-model /tmp/xerj-minilm-onnx/model.onnx \
  --onnx-tokenizer /tmp/xerj-minilm-onnx/tokenizer.json

Equivalent environment variables are:

export XERJ_EMBED_MODE=onnx-experimental
export XERJ_ONNX_MODEL=/tmp/xerj-minilm-onnx/model.onnx
export XERJ_ONNX_TOKENIZER=/tmp/xerj-minilm-onnx/tokenizer.json

CLI flags win over environment configuration.

Startup validates that:

• the binary was built with onnx-experimental;
• both asset paths point to readable files;
• admission limits are nonzero and internally consistent.

The server hashes the assets for the backend configuration but does not load
and optimize the approximately 90 MiB graph until a real semantic field needs
inference. Non-semantic indices therefore do not pay model-loading cost.

3. Autoindex a folder

target/release/xerj autoindex /path/to/corpus \
  --url http://localhost:9200 \
  --prefix finance-onnx \
  --fresh

target/release/xerj autoindex map \
  --url http://localhost:9200 \
  --prefix finance-onnx

ONNX runs only for fields inferred as semantic_text, normally a sufficiently
long body-like field. A short or structured dataset can legitimately infer no
semantic field.

Before attributing an indexing result to ONNX, check all three:

1. autoindex --dry-run or the created mapping contains semantic_text;
2. autoindex map reports semantic=true and a semantic_field;
3. the server prints the first-inference activation log described below.

This distinction prevents accidentally benchmarking the lexical or
non-semantic path and calling it ONNX.

4. Query the discovered semantic field

If autoindex map reports body:

curl -s http://localhost:9200/finance-onnx-*/_search \
  -H 'content-type: application/json' \
  -d '{
    "query": {
      "semantic": {
        "field": "body",
        "query": "Which quarter had the largest operating-margin decline?",
        "k": 10
      }
    },
    "size": 5
  }'

Query text is embedded through the same pinned backend and vector identity as
the stored documents.

What was validated live

A clean debug server was exercised through the actual server, autoindex, map,
semantic query, and shutdown paths.

Startup without semantic work

• startup completed in 277 ms;
• all 14 normal system indices initialized;
• there were zero ONNX backend configuration/activation messages;
• there were zero ONNX Runtime optimizer or allocator messages.

This verifies lazy activation and the log filter on a real server start. An
additional release initialization observation reached 390 ms, but that process
then failed to bind because another server already occupied the ports; it is
not used as the startup result.

Real autoindex and semantic retrieval

The live corpus contained three finance text records. The ordinary command,
not a direct library harness, produced:

• 3/3 records indexed;
• 0 junk records;
• 4.9 seconds autoindex wall time;
• data map: semantic=true;
• data map: semantic_field=body;
• exactly one ONNX backend-configuration message;
• exactly one first-inference activation message.

The semantic query returned:

• all three records;
• the correct Q2 record at rank one;
• top score 0.787737;
• reported query time 11 ms.

Clean shutdown flushed in 113 ms.

This is a smoke test proving that the feature is wired through the product
workflow and that lazy loading/log behavior is comprehensible. Three tiny
records are not a throughput benchmark, and the 4.9 seconds include fixed
startup/first-model-use effects.

Semantic ingestion architecture

Batch document embeddings before storage

The regular bulk path previously treated neural document embedding too much
like per-document work. The integrated path now identifies semantic index
operations, prepares their text/chunks, embeds compatible work as batches, and
then commits document/vector results while preserving original bulk-item
ordering and per-item errors.

The backend receives multiple texts so ONNX Runtime can amortize transformer
execution. The scheduler:

bounded input set
  -> measure post-truncation token lengths
  -> group similar lengths
  -> enforce max documents and padded-token budget
  -> run one safe ONNX session per microbatch
  -> restore exact original text/document order

Defaults:

Both document count and padded tokens are required. On heterogeneous text,
one long row can otherwise make every short row in the same batch cost as a
long row.

One safe shared session

Every complete ONNX configuration shares one process-wide lazy session.
Configuration identity includes paths, content fingerprints, thread settings,
scheduler limits, and admission limits.

The public safe ort API used here requires mutable session access. XERJ keeps
one session behind a mutex rather than bypassing Rust's API with raw pointers or
an unsafe impl Sync.

Consequences:

• individual Session::run calls are serialized;
• aggregate throughput comes from safe microbatching;
• compatible indices do not load duplicate sessions;
• asset contents are hashed into the shared identity;
• different model/tokenizer contents never reuse the same session merely
  because their paths match.

Immediately before the first load, XERJ rereads and rehashes both files. If
either changed after configuration, startup/inference refuses to silently
activate a different vector space from the same path.

Vector-space identity and restart safety

Every ONNX semantic index persists an embedding_identity.json marker:

{
  "version": 1,
  "backend": "onnx-experimental",
  "model_sha256": "<sha256>",
  "tokenizer_sha256": "<sha256>",
  "dimensions": 384,
  "pooling": "attention-mask-mean+l2-normalize",
  "max_tokens": 512
}

The marker is created when a semantic field/index becomes ONNX-backed, before
vectors can be silently mixed.

On reopen or later writes, XERJ refuses:

• a different model hash;
• a different tokenizer hash;
• a different backend;
• incompatible dimensions/pooling/token limit;
• enabling ONNX in place on a populated semantic index with no identity marker;
• caller-supplied derived vectors/chunks whose model identity cannot be
  verified.

The corrective path is explicit: restore the original assets, or create a
fresh index/vector space, normally by rerunning autoindex --fresh under a new
prefix.

This is intentionally stricter than "the model filename is unchanged." A
vector index is only meaningful when query and document vectors share the same
model, tokenizer, truncation, pooling, normalization, and dimensions.

Admission, overload, and resumability

The shared model/session also owns global admission controls:

Admission is checked before model load or tokenization where possible.
Rejected work therefore does not consume transformer memory merely to discover
that the node is overloaded.

The implementation distinguishes:

• too many texts in one call;
• one call exceeding its byte limit;
• the global concurrent-call limit being full;
• the global in-flight byte budget being full;
• impossible/zero startup configuration;
• a padded-token budget unable to fit one truncated row;
• changed assets;
• missing assets;
• an incompatible ONNX graph/output width.

Transient admission failure maps to retryable HTTP 429, including per-item
bulk errors. It is not flattened into an opaque 500.

autoindex treats a failed semantic bulk correctly:

• it reports the status and corrective reason;
• it does not mark the affected source file complete in the resume journal;
• the operator can reduce pressure, adjust configuration, or wait;
• rerunning the same command resumes unfinished work.

That is important for large corpora: overload must not create a false
"successfully indexed" journal state.

Logs and diagnostics

The ONNX path is quiet until it is actually used.

On the first real semantic inference, it emits one concise activation event
containing:

• model SHA-256;
• tokenizer SHA-256;
• dimensions;
• confirmation that the experimental backend loaded verified assets.

This gives operators an auditable way to prove which vector space produced an
index without flooding normal startup logs.

ONNX Runtime messages below warning are hidden by default. For diagnosis:

XERJ_ONNX_LOG=info target/release/xerj ...
# or: debug, verbose, trace

This avoids exposing optimizer/allocator chatter during ordinary XERJ startup
while retaining an explicit troubleshooting switch.

Error messages provide both the reason and next action, including:

• rebuild with --features onnx-experimental;
• pass --onnx-model and --onnx-tokenizer;
• restore the original hashed assets;
• create a fresh prefix/reindex rather than mixing vector spaces;
• split oversized work;
• retry overload with backoff;
• correct invalid admission limits.

Performance evidence

There are three different measurements below. They answer different questions
and must not be multiplied.

Controlled backend comparison: 12.90x embedding throughput

The strongest backend benchmark used a deterministic 128-document mixed-length
finance-like corpus and an identical length-aware batch plan for Candle and
ONNX:

• same sentence-transformers/all-MiniLM-L6-v2 semantics;
• same tokenizer;
• FP32;
• 512-token truncation;
• attention-mask mean pooling;
• row-wise L2 normalization;
• same original order;
• seven batches: 64, 18, 14, 10, 8, 8, 6;
• at most 4,096 padded token slots per batch;
• 25,778 padded token slots total;
• warmup;
• five runs with alternating backend order;
• no competing build/analysis workload.

The ONNX arm included its real double-tokenization scheduling cost. Candle used
the identical precomputed grouping, so the runtime comparison did not give
ONNX an easier input shape.

Equivalent wording:

• ONNX completed the embedding work in about 7.75% of Candle's wall time;
• elapsed time was about 92.25% lower;
• throughput was about 1,190% higher when expressed as increase over
  Candle;
• ONNX used 2.92x fewer CPU-seconds per document.

CPU time for the complete 128-document arm:

The wall-time gain exceeds the CPU-efficiency gain because ONNX Runtime used
the host's cores/kernel implementations more effectively. ONNX was configured
with 16 intra-op threads for the 16-CPU test host.

Quality checks:

• minimum same-document Candle/ONNX cosine: 0.9999991655;
• query-vector cosine approximately 1.0000006 (floating-point rounding);
• identical query top-10 order;
• 128/128 vectors restored to their original positions;
• no missing, non-finite, wrong-width, or cross-position result.

This is an embedding-layer benchmark. It excludes extraction, HTTP parsing,
lexical indexing, persistence, journaling, HNSW construction, and other
autoindex work. It is not evidence that the complete product is 12.90x faster.

Contextual scheduler smoke: 1.80x

A separate committed-backend smoke embedded 256 mixed-length documents:

• minimum scheduled/singleton cosine: 0.9999990463;
• all 256 results preserved input order.

A later post-log-filter smoke measured 163.59 scheduled docs/s but did not run
the singleton arm, so it is not presented as another speedup comparison.

The 1.80x result only shows that length-aware batching helps this ONNX workload.
It is contextual evidence, not the Candle-versus-ONNX headline and not
end-to-end indexing.

Why length-aware batching was retained

In a broader 256-document scheduling experiment:

Naive FIFO batching was 29.6% slower than singleton because padding
inflated work. This is why the product integration does not simply select
"batch 64" and assume bigger is faster.

Background from merged #2: Candle multi-index sharing

Merged #2's Candle ownership fix is useful even if ONNX remains experimental.
It is not part of this follow-up's 18-file integration delta, but its
before/after result is included to make the overall backend ownership model
reviewable in one place.

Controlled before/after with 16 identically configured neural indices:

Warm ingestion, eight documents per request:

Five repetitions are sufficient to reject a large concurrency regression, but
not to claim the observed 3–4% concurrent gain as intrinsic. The defensible
result is the large model-load/memory reduction with no observed throughput
loss at useful concurrency.

Identical documents across all 16 persisted indices produced bit-for-bit equal
384-dimensional Candle vectors; maximum component delta was 0.0.

Binary, runtime, and model cost

Measured stripped server binaries:

The static ONNX Runtime archive used during compilation was approximately
90.6 MiB, but static linking and dead-code elimination mean it is not copied
wholesale into the executable. The measured dual-backend binary increase is
18.75 MiB.

The verified FP32 ONNX model is approximately 90.4 MiB, comparable to the
existing approximately 90 MiB Candle safetensors model:

• ONNX-only deployment: one ONNX model asset;
• Candle-only deployment: one safetensors model asset;
• dual-backend cache: may contain both formats and therefore approximately
  double model-asset storage.

The measured GNU/Linux build statically links ONNX Runtime and does not require
a deployed libonnxruntime.so.

The upstream ort-sys prebuilt artifact matrix does not cover XERJ's
x86_64/aarch64 musl release targets. Standard musl binaries must remain
Candle-only unless the project adopts a reproducible ONNX Runtime musl build
or explicitly narrows the ONNX target matrix.

Benefits

• Real autoindex -> map -> semantic query integration.
• Same local FP32 MiniLM semantics with very close tested vector equivalence.
• Safe Rust ONNX invocation; no shared-session aliasing bypass.
• Length-aware batching avoids the measured naive-padding regression.
• Bounded documents, padded tokens, calls, per-call bytes, and global bytes.
• Retryable 429 overload behavior rather than opaque internal errors.
• Resume journal does not falsely mark rejected source files complete.
• Persisted model/tokenizer/vector-space identity prevents silent corruption.
• Asset hashes are rechecked immediately before lazy load.
• One shared session per complete configuration.
• Quiet ordinary startup and one auditable first-use activation message.
• No model download or unexpected network access at server startup.
• Non-semantic indices avoid loading the ONNX graph.
• Existing default lexical and Candle modes keep their meaning.
• Builds on feat(ai): prototype bounded ONNX embeddings #2's already-merged Candle multi-index sharing without changing its
  supported behavior.

Costs and limitations

• The mode is experimental and requires a feature build.
• Only GNU/Linux glibc packaging is currently demonstrated.
• The binary grows 18.75 MiB when both Candle and ONNX are present.
• Operators must export/provide and retain the correct local assets.
• The interface supports one exact MiniLM-compatible graph contract, not
  arbitrary ONNX models.
• ort 2.0.0-rc.12 is still a release candidate.
• One safe session serializes runtime invocations; throughput depends on
  batching.
• Scheduling measures token lengths and then tokenizes again for inference.
• The current batching path is designed around bounded bulk/offline work; it is
  not a general online queue with deadlines, per-tenant fairness, or
  cancellation.
• The 12.90x result is embedding-only.
• The three-document product smoke is too small for throughput conclusions.
• No full FinanceBench end-to-end ONNX versus Candle autoindex time has been
  measured yet.
• Full-corpus RSS, peak memory, and disk growth have not been compared in
  separate end-to-end backend processes.
• Controlled vector/ranking equivalence does not replace a full FinanceBench
  evidence Recall@k and near-tie ANN evaluation.
• Scanned-PDF OCR and PDF extraction quality are separate from the embedding
  runtime and are not fixed by ONNX.

Rejected/deferred approaches

• Unsafe concurrent access to one ort::Session.
• Session pools that duplicate runtime/model state without a measured
  end-to-end benefit.
• Fixed universal batch 64.
• Naive FIFO batch 16.
• Unbounded batches based only on document count.
• 24/32 runtime threads on a 16-CPU host; neither gave stable gains.
• Unconditional collection sleeps for already-backlogged autoindex work.
• Graph surgery for pooling/L2 without profiling evidence.
• Quantization before an independent quality gate.
• Silent backend/model switching for existing indices.
• Automatic guessing of graph inputs/outputs.

Validation

The follow-up was prepared from current upstream base:

88f21b8 Merge pull request #4 from xerj-team/docs/agent-scenarios

Its integration delta modifies 18 existing files across xerj-ai,
xerj-autoindex, xerj-common, xerj-engine, xerj-server, the lockfile,
and the already-landed experimental guide. It does not re-add the standalone
prototype commit from #2.

The integrated source adds tests for:

• scheduler bounds and order restoration;
• impossible padded-token budgets;
• call and byte admission;
• admission shared across handles;
• rejection before model load/tokenization;
• same paths with different hashes never sharing a session;
• ONNX dimension validation;
• identity creation and compatible reopen;
• model/tokenizer mismatch;
• backend mismatch;
• marker-less populated semantic-index refusal;
• later-added semantic fields being pinned before first write;
• caller-supplied unverified derived vector/chunk refusal;
• admission errors mapping to HTTP 429;
• bulk preserving per-item ONNX 429;
• a non-vector schema not touching ONNX assets;
• Candle process-wide model sharing and lifecycle.

The final integrated diff passed:

cd engine

cargo test --release -j 32 -p xerj-ai \
  --features neural,onnx-experimental
# 36 passed, 0 failed, 1 ignored

# Scoped autoindex suite (expanded on the current upstream base)
# 29 passed, 0 failed

# ONNX-enabled engine test targets
# unit tests:           134 passed, 0 failed
# battle:                10 ignored
# chaos:                 10 passed, 0 failed
# es_compat:             65 passed, 0 failed
# integration:           93 passed, 0 failed
# multi_match:            1 passed, 0 failed
# node_lock:              1 passed, 0 failed
# perf:                    7 ignored
# product:               10 passed, 0 failed
# storage_hardening:      2 passed, 0 failed
# ttl:                     2 passed, 0 failed
# shard_router:            2 passed, 0 failed

# ONNX-enabled server tests
# 7 passed, 0 failed

# Repository formatting/whitespace gates
cargo fmt -p xerj-ai -- --check
git diff --check

The integrated server path was additionally exercised live as described above:

server start
  -> autoindex 3/3
  -> map reports body semantic field
  -> first-use model activation
  -> semantic query ranks Q2 first
  -> clean flush/shutdown

The mandatory ES-YAML hard gate was rerun on the final integrated diff:

1,360 passed
0 failed
3 skipped
1,363 total

Reviewer checklist

1. Is the exact model contract narrow and explicit enough?
2. Does the identity marker cover every parameter that determines vector
   compatibility?
3. Are marker-less populated semantic indices rejected at the right boundary?
4. Can any derived-vector or bulk path bypass identity validation?
5. Do 429 admission failures preserve bulk item order/status and autoindex
   resumability?
6. Are the default call/byte/token limits safe across supported machines?
7. Is lazy initialization correct under concurrent first use?
8. Are logs sufficient to prove ONNX was actually used without exposing noisy
   runtime internals?
9. Should the experimental feature remain excluded from all standard release
   artifacts until musl and full FinanceBench gates pass?

Follow-up gates

Before changing ONNX from experimental:

1. Run full FinanceBench end-to-end on identical extracted input with Candle
   and ONNX.
2. Report complete autoindex wall time, semantic passages/s, documents/s,
   tokens/s, CPU-seconds/document, p50/p95/p99, RSS/HWM, and disk growth.
3. Score document/evidence Recall@1/5/10, ranking changes, and ANN near ties.
4. Test restart, interruption, resume, cancellation, and overload on a
   multi-hour corpus.
5. Stress simultaneous indexing and semantic queries.
6. Validate corrupt/truncated graphs and tokenizers without process aborts.
7. Decide and document GNU-only versus reproducible musl support.
8. Add backend/session/admission metrics and operator diagnostics.
9. Re-run the complete ES-YAML conformance hard gate.

Honest claim boundary

Supported:

The experimental ONNX backend now works through XERJ's real
autoindex -> map -> semantic query loop. A live three-record run indexed
3/3 records, identified body as semantic, activated one verified shared
ONNX model, and ranked the correct Q2 record first. Separately, on a
controlled mixed-length xerj-ai workload, scheduled FP32 ONNX completed
the embedding portion in 1.097 seconds versus 14.152 seconds for identically
batched Candle: 12.90x throughput, 92.25% less elapsed time, and 2.92x fewer
CPU-seconds per document with equivalent tested vectors/ranking.

Not yet supported:

XERJ autoindex is 12.90x faster on FinanceBench.

The full FinanceBench end-to-end time has not been measured.