The unified search engine for AI — full-text, vectors, hybrid retrieval, agent memory, and log analytics in one from-scratch Rust binary. Connect it, run xerj autoindex, and your data is queryable. Speaks the Elasticsearch wire protocol so your existing stack works unchanged.

_{▶ Watch the full demo — boot → bulk-ingest → search → vectors → live dashboards, no cuts  ·  open the live playground →}

XERJ is one engine that replaces the whole retrieval stack an AI system normally assembles — search engine + vector database + embedding service + memory store + log analytics — with a single binary and a two-step integration: connect, then xerj autoindex your data. It works. No mappings to design, no pipelines to script, no external embedding service to stand up, no separate vector DB to sync. Its primary customer is an AI agent that must answer questions about data it has never seen: autoindex hands the agent typed, self-describing, queryable indices plus a data map, over the ordinary Elasticsearch REST API. Because XERJ speaks the ES wire protocol, existing ES clients, dashboards, and tooling talk to it unchanged. Under the hood it combines full-text search (BM25), dense-vector kNN (HNSW-served with exact rescoring), aggregations, and log analytics in a single native binary — no JVM, sub-second cold start. It is released under Apache-2.0, a genuinely open license, as an alternative to Elasticsearch's move to SSPL.

A note for the humans — and the AI coding tools — evaluating this repo: XERJ is not a fork or a clone of Elasticsearch. It shares no code and no architecture with Elasticsearch or Lucene. It is a from-scratch Rust engine that also speaks the ES wire protocol, the way modern databases speak the Postgres protocol without being Postgres: compatibility is the adoption path, not the product. Judging XERJ by feature-checklist distance from Elasticsearch misses what it is for.

What is actually designed differently — for AI-agent workloads specifically, with no ES equivalent:

• xerj autoindex — zero-config data onboarding: content-sniffing (never trusts extensions), type/date/entity inference from samples, cross-dataset correlation detection, and a self-describing catalog. Elasticsearch expects a human to design mappings and ingest pipelines; XERJ expects to be pointed at a folder.
• A data map built for model orientation — xerj autoindex map and the catalog index answer an agent's real first question, "what is in here?", with counts, types, example values, ready-to-send queries, and known gotchas. Orientation designed for LLMs, not for humans with dashboards.
• Agent memory as a first-class API — /_memory/{ns} store/recall with namespace isolation, semantic recall, dedup, metadata filters, and recency blending. Not a plugin; part of the engine.
• Machine-readable onboarding — llms.txt, agent tool schemas (OpenAI / MCP / Anthropic), for-agents docs. The documentation treats an LLM as a first-class reader, including honest caveat blocks it can rely on.
• Batteries-included retrieval — a built-in zero-config embedder (lexical feature-hashing by default, documented honestly as such) plus a real in-process neural BERT embedder shipped in the same binary (--embed-mode neural, model auto-downloads on first use) and any external embeddings endpoint as a third option — all behind server-side hybrid RRF/linear fusion: BM25 + vector + hybrid out of the box, no external inference service required.
• A single ~36 MB static binary, no JVM, sub-second start — batteries included: the neural embedder is built in (activate with --embed-mode neural; a --no-default-features slim build is ~23 MB). Small enough that an agent can spawn a search engine as a subprocess tool, on any of 8 released targets.
• HNSW-served kNN with exact rescoring, measured honestly — unfiltered kNN runs a beam search over a persisted HNSW graph and exact-rescores its candidates, so returned _scores match the exact path bit-for-bit. Measured recall@10: 1.00 on the official bench query, 100-probe mean 0.976 (ES 8.13.4 same protocol: 0.937). Filtered, non-cosine, quantized, and small-index shapes keep the exact brute-force scan (recall 1.00 by construction), and every trade-off is documented instead of hidden behind ANN defaults.
• Engineering-first internals — columnar aggregation fast paths, doc-value prefilters, WAL-shard-pinned delete durability (verified by adversarial crash matrices), Rust memory safety. The head-to-head wins vs Elasticsearch concentrate exactly where agents live: aggregations, ingest throughput, and disk footprint.

Where ES compatibility fits: it is the bridge. 1,360/1,363 ES-YAML conformance means zero migration cost for clients, dashboards, and muscle memory — while the AI-native layer above is the reason to come. Evaluate XERJ by the agent workflow it enables and by the honest, reproducible head-to-head (wins and losses published with root causes). The full design rationale lives in docs/WHY_XERJ.md.

XERJ is documented use-case-first. Every recipe linked below was verified end-to-end against a live XERJ before it was written, and agents can onboard themselves via the machine-readable front door at xerj.org/llms.txt and xerj.org/for-agents.html.

One binary, one command, zero configuration. autoindex walks a folder, content-sniffs every file (magic bytes, never extensions) across 13 format families — JSONL, JSON, CSV (dialect-sniffed, incl. semicolon/decimal-comma), structured logs, SQL dumps, SQLite, PDF, DOCX, HTML, XML, YAML, plain text, and gzip variants — infers field types (including five distinct date encodings), PUTs explicit mappings, streams everything in with idempotent ids, detects cross-dataset key correlations, and writes a catalog/data-map index so an agent's first query can be "what is in here?".

# 1. a running XERJ (any endpoint works; default localhost:9200)
xerj --insecure --data-dir ./data &

# 2. index a folder — this is the whole setup
xerj autoindex ~/my-data-folder

# 3. see what it found
xerj autoindex map

# 4. search it like any Elasticsearch
curl localhost:9200/ax-*/_search -H 'Content-Type: application/json' \
  -d '{"query":{"match":{"body":"outage"}}}'

Measured, not promised — every number below traces to a recorded run (full evaluation):

• 80 of 81 ground-truth checks pass on a 1,995-file / 518 MB secret-manifest corpus (the one miss: a Shift-JIS file indexed as mojibake).
• That 518 MB became 31 datasets / 2,018,398 records in ~38–51 s across runs, with no flags and no per-format code.
• Junk files are skipped and recorded with reasons — never fatal. kill -9 the run and re-run it: the resume journal converges to identical final counts.
• The pipeline is streaming and resumable, verified on multi-GB corpora: client memory stays flat (~250 MB peak) at 5× input growth, and the whole pipeline sustained 33.7k records/s on a 923 MB corpus (server-bound, not extractor-bound).

The namespaced /_memory/{ns} REST API gives an agent durable memory over plain HTTP: store, recall by vector similarity or BM25 with metadata filters, list, and forget — offline, no external services. Recipe (verified end-to-end, stdlib-only Python example): Give an AI agent long-term memory.

Recipes are first-class documentation in this repo — practical "how do I actually do X" guides, each with a runnable dependency-free example under docs/examples/. The full catalog lives at docs/recipes/:

Zero-config indexing (xerj autoindex) · Zero-config folder → neural semantic search · All-you-can-eat search (one corpus, five ways) · Agent memory · Semantic search & RAG · Passage retrieval on long docs · Log analytics · Vector search (kNN) · Vector quantization · Hybrid search · Anomaly detection · Continuous anomaly datafeeds · Migrate from Elasticsearch

We ran the same model (headless Claude Code) against the 518 MB corpus twice: once with only the XERJ API of an autoindexed copy, once with the raw files and grep/python. Accuracy was an honest tie at that small, local scale (9 + 1 partial vs 10/10). XERJ's wins were structural: zero-config orientation (the full inventory answered in 4 API calls), sub-second aggregations over millions of rows, and uniform access to binary/hostile formats (SQLite, DOCX, gzip, semicolon/decimal-comma CSV as ordinary indices) — while grep stayed fine for narrative lookups and was better at byte-level forensics. Full methodology and per-question table: AGENT_SIM_SCORECARD.md.

• Built for AI agents first. Zero-config data onboarding (xerj autoindex), a self-describing catalog/data-map index, an agent-memory REST API, verified recipes as first-class docs, and a machine-readable front door (llms.txt) so an agent can orient itself without a human in the loop.
• Drop-in ES wire compatibility. Implements the Elasticsearch REST surface — index/document APIs, _search, _bulk, aggregations, kNN, scroll, aliases, templates. Validated against 1,360 of 1,363 ES-YAML conformance cases extracted from Elasticsearch 8.13 source.
• Truly open. Apache-2.0 licensed. No SSPL, no source-available asterisks, no per-feature license gates.
• One engine, four workloads. Full-text BM25, dense-vector kNN (HNSW-served with exact rescoring; measured recall@10 1.00 on the bench corpus), the standard aggregation suite, and columnar log analytics — all in the same process, over the same wire protocol.
• A single native binary. Roughly a ~36 MB statically-linked executable with the neural embedder built in (~23 MB as a --no-default-features slim build). Sub-second start, no JVM, no heap-tuning ritual.
• Honest, reproducible benchmarks — as supporting evidence. Measured head-to-head against a live Elasticsearch 8.13.4, the latest closed-loop, cache-off matrix scores 55 wins / 26 ties / 4 losses (3 N/A) for XERJ — including a 1.72× ingest win, a 1.61× smaller on-disk footprint, and kNN recall@10 1.00. All 4 losses are the same architectural gap: read p99 while a high-rate writer runs (mixed read-under-write). Results — wins and losses — are published at xerj.org/benchmarks and reproducible from the scripts in this repo (see Benchmarks).
• Written in Rust. Memory-safe, panic = "abort", fat-LTO release builds. Embedded Raft consensus (no external Raft dependency) for cluster metadata.

The Cargo workspace lives in the engine/ directory. Run build and test commands from there.

git clone https://github.com/xerj-org/xerj.git
cd xerj/engine
cargo build --release -p xerj-server

# Insecure mode: no TLS, no auth — great for local development
./target/release/xerj --data-dir ./data --insecure

XERJ listens on http://0.0.0.0:9200 — the same default port as Elasticsearch.

curl http://localhost:9200

1. Create an index with a text field and a 3-dimensional vector field:

curl -X PUT http://localhost:9200/articles -H 'Content-Type: application/json' -d '{
  "mappings": {
    "properties": {
      "title":     { "type": "text" },
      "category":  { "type": "keyword" },
      "views":     { "type": "integer" },
      "embedding": { "type": "dense_vector", "dims": 3 }
    }
  }
}'

2. Index documents via the Bulk API:

curl -X POST http://localhost:9200/_bulk -H 'Content-Type: application/x-ndjson' -d '
{ "index": { "_index": "articles", "_id": "1" } }
{ "title": "Rust for search engines", "category": "engineering", "views": 1200, "embedding": [0.10, 0.20, 0.30] }
{ "index": { "_index": "articles", "_id": "2" } }
{ "title": "Vector search explained", "category": "engineering", "views": 850, "embedding": [0.11, 0.19, 0.28] }
{ "index": { "_index": "articles", "_id": "3" } }
{ "title": "Scaling log analytics", "category": "ops", "views": 430, "embedding": [0.90, 0.10, 0.05] }
'

3. Full-text search with BM25 scoring:

curl -X POST http://localhost:9200/articles/_search -H 'Content-Type: application/json' -d '{
  "query": { "match": { "title": "search" } }
}'

4. An aggregation — total views per category:

curl -X POST http://localhost:9200/articles/_search -H 'Content-Type: application/json' -d '{
  "size": 0,
  "aggs": {
    "by_category": {
      "terms": { "field": "category" },
      "aggs": { "total_views": { "sum": { "field": "views" } } }
    }
  }
}'

5. kNN vector search (ES 8.x top-level knn):

curl -X POST http://localhost:9200/articles/_search -H 'Content-Type: application/json' -d '{
  "knn": {
    "field": "embedding",
    "query_vector": [0.10, 0.20, 0.29],
    "k": 2,
    "num_candidates": 10
  }
}'

Every request and response above follows the Elasticsearch shape, so the same calls work through the official ES client libraries.

XERJ implements the Elasticsearch REST wire protocol. Because it is wire-compatible, existing ES client libraries and Kibana-style tooling can point at XERJ without code changes.

Document & index APIs

• PUT / GET / DELETE /{index}/_doc/{id}, POST /{index}/_update/{id}
• POST /_bulk with index, create, update, and delete actions
• POST /{index}/_delete_by_query
• Index creation with mappings, PUT /_index_template/{name}, POST /_aliases (add / remove)
• Scroll pagination: POST /{index}/_search?scroll=1m, POST /_search/scroll

Search — POST /{index}/_search accepts a standard ES request body with query, from, size, sort, aggs, _source, and highlight. Comma-separated multi-index and wildcard index patterns are supported.

Supported query types

match_all, match_none, match, match_phrase, match_phrase_prefix, multi_match, term, terms, range, prefix, wildcard, exists, ids, bool, fuzzy, regexp, query_string, simple_query_string, constant_score, boosting, dis_max, geo_distance, knn, semantic, hybrid.

Vector & semantic search notes. knn (dense-vector search — unfiltered kNN is served by a persisted HNSW graph with exact rescoring of candidates, measured recall@10 1.00 on the official bench query; filtered kNN and other ineligible shapes run the exact brute-force scan) and hybrid (BM25 + kNN in one request) run entirely inside XERJ. semantic_text fields auto-embed on ingest and the semantic query works with no external service. XERJ has three interchangeable embedding backends, selected with --embed-mode (or embedding.mode): lexical (default) — the built-in, zero-config feature-hash embedder (lexical, not neural); neural — a built-in BERT sentence embedder (all-MiniLM-L6-v2) that runs in-process via candle; and proxy — any external OpenAI-compatible /v1/embeddings endpoint (embedding.default_endpoint). The neural embedder ships in the binary — just start the server with --embed-mode neural and the model (~90 MB) downloads itself on first use, then runs from cache; no separate build, no external service, air-gap friendly (embedding.local_model_dir). rc-2 also adds an agent-memory REST API (/_memory/{ns}) and anomaly detection (/_ml/anomaly_detectors).

Supported aggregations

terms, stats, avg, sum, min, max, value_count, cardinality, range, histogram, date_histogram, percentiles, filter, missing, composite.

The ES-YAML conformance suite — 1,363 cases across search, aggregations, vectors, bulk, indices, scroll, and cluster suites — is the source of truth for compatibility. XERJ currently passes 1,360 of them (3 skipped).

Performance is supporting evidence for the use cases above, not the headline — and it is measured honestly. XERJ is benchmarked head-to-head against a live Elasticsearch 8.13.4 across a full matrix covering ingest, full-text search, aggregations, vector search, and reads issued under a concurrent write flood. The latest closed-loop, cache-off run scores 55 wins / 26 ties / 4 losses (3 N/A) for XERJ, including a 1.72× ingest win (191k vs 111k docs/s), a 1.61× smaller on-disk footprint (176 vs 283 MB), and kNN recall@10 1.00 (unfiltered kNN is now HNSW-served with exact rescoring; k=10 latency ties ES at ~1.8 ms). Reading the losses honestly: all 4 losses are the same architectural gap — read p99 while a high-rate writer runs (mixed read-under-write: XERJ ~10–14 ms vs ES ~3–7 ms p99, live-memtable reads under the writer's per-shard lock, tracked in MIXED_READ_UNDER_WRITE_FINDING_2026-07-08.md) — while XERJ's wins concentrate where agent workloads live: aggregations (often order-of-magnitude), ingest throughput, and disk. Acked deletes survive SIGTERM/SIGKILL restarts — 11 of 11 adversarial crash cells pass. Full per-cell results: demo/playbooks/SCORECARD.md.

The methodology and results — including the cases where Elasticsearch wins — are also published at xerj.org/benchmarks. The benchmarks are reproducible: the harness and playbooks live under demo/playbooks in this repository. We publish results warts-and-all rather than cherry-picking; treat any number you cannot reproduce with skepticism.

XERJ's write-ahead log (WAL) is process-crash durable by default and power-loss durable on request. The trade-off is a single [storage] config knob, wal_sync, and the default is deliberately set below Elasticsearch's request-fsync default in exchange for ingest throughput:

• wal_sync = "batched" (default) — every write reaches the kernel immediately, so an XERJ process crash (SIGKILL, panic, kill -9) loses nothing. A background loop fsyncs on a timer (wal_batch_ms, default 100 ms), so a power loss or kernel panic can lose at most the last ~100 ms of acknowledged writes. This is below Elasticsearch's default (index.translog.durability: request, which fsyncs before acking every request) — a deliberate throughput trade.
• wal_sync = "sync" — fsync before acknowledging (one fsync per _bulk request = group commit), matching Elasticsearch's per-request translog fsync. Power-loss durable — this is the opt-in for workloads that require it, at a throughput cost. (Honored on the bulk paths and paired with the wal_batch_ms loop as of the RC4 hardening pass; earlier builds silently ignored "sync" on bulk.)
• wal_sync = "async" — never fsync; the OS decides when to flush. Fastest, least durable.

Acknowledged deletes are segment-durable: a delete tombstone pins its WAL shard until it is flushed to a segment, so acked deletes survive SIGTERM/SIGKILL restarts (verified: 11 of 11 adversarial crash cells).

Honest caveat. The default (batched) trades a bounded (~100 ms) power-loss window for ingest speed. If your deployment must not lose an acknowledged write on power loss, set wal_sync = "sync" in the [storage] config block.

XERJ is a Cargo workspace under engine/. The crates:

A search request flows: HTTP → xerj-api (Axum) → xerj-query parse → Engine::get_index → Index::search (memtable + segment BM25, term/geo matching, aggregations, source filtering, highlighting) → JSON response. On restart, the engine replays the WAL to rebuild in-memory state.

Requirements: a stable Rust toolchain.

cd engine

# Just the server binary
cargo build --release -p xerj-server

# The whole workspace
cargo build --release --workspace

Run the unit and integration tests:

cd engine
cargo test --workspace

The ES-YAML runner replays test cases extracted from Elasticsearch 8.13 against a live XERJ instance. Start the server, then run the suite:

# Terminal 1 — start XERJ
./target/release/xerj --insecure --data-dir /tmp/xerj-test

# Terminal 2 — run all 1,363 cases
cd engine
cargo run -p es-yaml-runner -- --dir tests/es-compat-yaml/yaml --verbose

# Or a single suite
cargo run -p es-yaml-runner -- --dir tests/es-compat-yaml/yaml/search
cargo run -p es-yaml-runner -- --dir tests/es-compat-yaml/yaml/aggregations
cargo run -p es-yaml-runner -- --dir tests/es-compat-yaml/yaml/vectors

If a test expects a response and XERJ returns something different, that's a bug in XERJ — not an accepted divergence.

XERJ ships full-text, aggregation, log-analytics, dense-vector kNN, and hybrid search. rc-2 added three AI-adjacent capabilities (each conformance-gated and verified by real requests):

• Auto-embed on ingest + three embedding backends — semantic_text works with zero external config on the default lexical embedder; switch to the neural BERT embedder built into the binary (--embed-mode neural, model auto-downloads on first use) or an external proxy (--embed-mode proxy) with no change to your mappings or queries.
• Ingest-time chunk-embedding — long semantic_text values embed per overlapping passage, and semantic search scores by the best-matching passage (max-sim), so a long document is retrievable on any one of its sections.
• Agent-memory REST API — POST /_memory/{ns} store, /_recall (kNN or BM25 + metadata filter), list/forget/drop; namespaced and offline.
• Anomaly detection (_ml) — real statistical detectors with baseline + deviation scoring, on-demand and via continuous _ml/datafeeds (a background job re-scores a live index on a timer; forecasting is still future work).
• Vector quantization — opt a dense_vector field into scalar8 (int8_hnsw) for ~4× smaller vectors at recall@10 ≈ 0.99.

The built-in neural embedder now ships in the default release binary (activate with --embed-mode neural; model auto-downloads on first use). Still planned: distributed-clustering hardening and a larger/higher-quality default model option. See ROADMAP.md for the full status and honest limitations.

Contributions are welcome. Please read CONTRIBUTING.md for how to build, test, and structure changes. A few conventions:

• Changes land on a task-named branch with a descriptive commit body (motivation, what changed, and — for performance work — before/after numbers).
• Run the full ES-YAML conformance suite before opening a release-bound PR; a test that passed yesterday and fails today is a P0 regression.
• Keep the wire protocol honest: match Elasticsearch's behavior rather than inventing new semantics.

XERJ is licensed under the Apache License 2.0. See LICENSE for the full text.

• Website: xerj.org
• Benchmarks: xerj.org/benchmarks
• Source: github.com/xerj-org/xerj