feat(ai): prototype bounded ONNX embeddings - #2
Merged
Conversation
Neural indices previously loaded one Candle model per index, multiplying startup work and resident memory. The experimental ONNX investigation also lived outside a reviewable product change, so its safe execution policy and measured boundaries were not available in-tree. Share Candle models process-wide by complete NeuralConfig while retaining lazy loading and releasing registry entries after the last user drops. Add an opt-in xerj-ai ONNX FP32 backend using the safe ort API, one shared session, length-aware microbatches, padded-token and queue bounds, input-order restoration, focused planner tests, and a live benchmark example. The default feature set and --embed-mode neural remain Candle; no server or CLI ONNX mode is introduced. The controlled embedding-layer comparison measured 116.671 docs/s for scheduled ONNX versus 9.045 docs/s for identically planned Candle (12.90x), with minimum same-document cosine 0.9999991655. This is not an end-to-end indexing claim. Measured stripped server deltas were +18.75 MiB for dual backends and +16.43 MiB for ONNX-only. Bundled ort artifacts do not support the project's musl release targets, so the feature remains explicitly experimental. Validation: default xerj-ai release build and 25 tests; neural release build and 3 model-sharing tests; ONNX release build and 3 scheduler tests; combined neural+ONNX suite (31 passed, 1 ignored); live 256-document example (163.96 scheduled docs/s, 1.87x over singleton, minimum cosine 0.99999905); package formatting and diff checks.
7 tasks
xerj-org
added a commit
that referenced
this pull request
Jul 31, 2026
…#82) Two user-reported problems autoindexing large source trees (django, redis), plus a real new capability. 1. `Too many open files (os error 24)` mid-run. Every index holds segment mmaps, a WAL and a merge task, so discovering a repo (django infers ~74 datasets → ~74 indices) opens thousands of descriptors. On a default macOS soft limit (256) the server exhausted them at the 2nd index. Reproduced against a real django clone under `ulimit -n 256`. Fix: raise RLIMIT_NOFILE to the hard limit at server startup (step down for the macOS kernel cap; best-effort). Django + Redis now index cleanly under a 256 soft limit (was: crash at index #2). 2. `<jemalloc>: option background_thread currently supports pthread only` on every macOS launch. `background_thread:true` is Linux-only; gate it to `target_os = "linux"` and keep the decay policy elsewhere. 3. Source code was indexed as plain text. Add an AST extractor (extract/code.rs) using tree-sitter for python, javascript, typescript, tsx, rust, go, java, c, c++, ruby, php, c#, bash — routed by extension via a new `Family::Code`. Each file carries `language`, a structured `symbols` array {name,kind,line}, a searchable `defs` list ("class User", "def save") and the full source, so `class Model` retrieves the file that defines it. 13 per-language tests + the full 140-test autoindex suite pass; verified on real django (Python) and redis (C) clones.
This was referenced Aug 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Neural indices previously loaded one Candle model per index, multiplying startup work and resident memory. The experimental ONNX investigation also lived outside a reviewable product change, so its safe execution policy and measured boundaries were not available in-tree.
Share Candle models process-wide by complete NeuralConfig while retaining lazy loading and releasing registry entries after the last user drops. Add an opt-in xerj-ai ONNX FP32 backend using the safe ort API, one shared session, length-aware microbatches, padded-token and queue bounds, input-order restoration, focused planner tests, and a live benchmark example. The default feature set and --embed-mode neural remain Candle; no server or CLI ONNX mode is introduced.
The controlled embedding-layer comparison measured 116.671 docs/s for scheduled ONNX versus 9.045 docs/s for identically planned Candle (12.90x), with minimum same-document cosine 0.9999991655. This is not an end-to-end indexing claim. Measured stripped server deltas were +18.75 MiB for dual backends and +16.43 MiB for ONNX-only. Bundled ort artifacts do not support the project's musl release targets, so the feature remains explicitly experimental.
Validation: default xerj-ai release build and 25 tests; neural release build and 3 model-sharing tests; ONNX release build and 3 scheduler tests; combined neural+ONNX suite (31 passed, 1 ignored); live 256-document example (163.96 scheduled docs/s, 1.87x over singleton, minimum cosine 0.99999905); package formatting and diff checks.