AI-native search, vector, and log engine. Rust/C. Single binary. Zero GC. Byte-perfect.

xerj is an Elasticsearch-compatible search engine that replaces ES clusters with a single binary.

# Download and run
./xerj --data-dir ./data --insecure

# Or with Docker
docker run -v xerj-data:/data -p 9200:9200 xerj --insecure

# Create an index
curl -X PUT http://localhost:9200/my-index

# Index a document
curl -X PUT http://localhost:9200/my-index/_doc/1 \
  -H 'Content-Type: application/json' \
  -d '{"title": "Hello", "content": "World"}'

# Search
curl -X POST http://localhost:9200/my-index/_search \
  -H 'Content-Type: application/json' \
  -d '{"query": {"match": {"content": "world"}}}'

Head-to-head against Elasticsearch 8.13.0 on the same machine, identical HTTP wire payloads, single-node, tmpfs data dir, both freshly started (full report):

xerj wins on every operational and read-path metric. ES still wins bulk-ingest throughput on warm Lucene IndexWriter (perf-backlog item to close — sharded flush thresholds + doc-values borrow refactor).

• Broad ES 8.x wire compatibility — point an existing client at port 9200 and most of it just works. Wide, not total: the measured coverage and the gaps are in demo/playbooks/ES_COMPATIBILITY.md
• BM25 full-text search with highlighting, aggregations, and every query type in Query Types Supported below — plus two more that are recognised and deliberately refused with a 400 rather than answered wrongly
• Vector search with HNSW, filtered ANN, and inline embedding
• SQL API — query with SQL syntax
• WAL persistence — crash recovery, data survives restarts
• Security by default — auth + TLS on by default
• Zero dependencies — single static binary, no JVM, no Lucene

cargo build --release -p xerj-server

The resulting binary is at target/release/xerj.

See xerj.default.toml for the commonly-tuned settings, and crates/xerj-common/src/config.rs for all 115 — every one of them optional.

The minimal config to get started:

[server]
data_dir = "/var/lib/xerj"
es_compat_port = 9200

[auth]
enabled = false   # NOT the default — the default is true

auth.enabled defaults to true; enabled = false above is an explicit opt-out, appropriate only for a trusted private network or local development. Leave it out (or set it to true) and the server mints an admin key on first boot, prints it, and writes it to <data_dir>/admin.key. Every client then has to send it — including xerj autoindex, which never reads the key out of xerj.toml; give it --api-key or XERJ_API_KEY:

export XERJ_API_KEY="$(cat /var/lib/xerj/admin.key)"
xerj autoindex ~/my-project

Pass a config file with --config:

xerj --config /etc/xerj/xerj.toml

Complete and machine-checked. The list is generated from xerj_query::parser::SUPPORTED_QUERY_TYPES, that constant is pinned to parse_query's dispatch table by a unit test, and this file is compared against it by crates/xerj-engine/tests/docs_capability_lists.rs — so a query type cannot be added or removed without the build failing until the docs follow.

Acceptance is not a fidelity claim. A type listed here parses, plans and executes; it is not a promise that every parameter matches Elasticsearch in every corner. Per-type gaps are tracked in ROADMAP.md, and the ES-YAML conformance suite is the measured answer.

Full-text — match, match_phrase, match_phrase_prefix, match_bool_prefix, multi_match, combined_fields, query_string, simple_query_string, intervals, more_like_this

Term-level — term, terms, terms_set, range, prefix, wildcard, regexp, fuzzy, exists, ids, script

Universal — match_all, match_none

Compound and scoring — bool, boosting, constant_score, dis_max, function_score, script_score, distance_feature, rank_feature, pinned

Vector, semantic and hybrid — knn, semantic, hybrid

Geo — geo_distance, geo_bounding_box, geo_polygon, geo_shape

Span — span_term, span_near, span_or, span_not, span_first, span_containing, span_within

Document structure — nested

Other — percolate, type, wrapper

Recognised and deliberately rejected with a 400:

has_child, has_parent

XERJ never materialises a parent/child join, so a has_child that ran would match against flat documents and return silently wrong hits. It refuses at parse time instead, with a message that names the alternative (denormalize the relationship). Any other query type answers unknown query type.

Complete and machine-checked the same way, from xerj_engine::aggs::SUPPORTED_AGG_TYPES. No probabilistic sketch sits in the metric path — cardinality is a true distinct count, not an HLL estimate, and terms doc_count is precise.

There are two deliberate exceptions. The first is the sampling family, which is a sample by definition: run_sampler sorts the matched documents by _score and keeps the first shard_size (default 200), so every sub-aggregation under sampler or random_sampler is computed over that slice rather than the whole match set. diversified_sampler truncates the same way and additionally caps how many documents may share a field value (max_docs_per_value, default 1). random_sampler shares the sampler implementation and does not read ES's probability.

The second is percentiles with the hdr option, which returns HdrHistogram-quantized values on purpose, so that ES's own outputs reproduce. The default tdigest path is not a t-digest at all: it sorts every matched value and interpolates linearly between the neighbours, reading the whole value set (O(N) memory) instead of a sketch.

Metric — avg, sum, min, max, stats, extended_stats, value_count, cardinality, percentiles, percentile_ranks, median_absolute_deviation, matrix_stats, string_stats, boxplot, top_metrics, top_hits, scripted_metric

Bucket — terms, multi_terms, rare_terms, significant_terms, significant_text, range, date_range, ip_range, ip_prefix, histogram, variable_width_histogram, date_histogram, auto_date_histogram, filter, filters, missing, composite, adjacency_matrix, time_series, global

Sampling — sampler, random_sampler, diversified_sampler

Scope — nested, reverse_nested

Geo — geo_bounds, geo_centroid, geo_distance, geohash_grid, geotile_grid

Pipeline — avg_bucket, sum_bucket, min_bucket, max_bucket, stats_bucket, extended_stats_bucket, percentiles_bucket, derivative, cumulative_sum, serial_diff, moving_avg, moving_fn, bucket_script, bucket_selector, bucket_sort

Bucket aggregations take nested sub-aggregations via an inner aggs key.

HTTP Request
    → xerj-api  (Axum handler, es_compat.rs)
    → xerj-query  parse_request() — raw JSON → SearchRequest (QueryNode tree)
    → Engine::get_index() — looks up named index
    → Index::search()
         → memtable scan  (in-memory BM25 via FtsMemtable)
         → segment scan   (on-disk FTS via FtsIndexReader + BM25)
         → doc_matches_query() for term-level / geo queries
         → run_aggs() for aggregations
         → apply_source_filter(), apply_highlight()
    → SearchResult → JSON response

Every crate under crates/ appears in this table — crates/xerj-engine/tests/docs_capability_lists.rs fails if one is missing or if the table names a crate that no longer exists. Five of them (xerj-autoindex, xerj-console-api, xerj-cluster, xerj-mcp, xerj-wasm) were absent until issue #211. The conformance runner lives outside crates/, at tests/es-compat-yaml.

Each index is stored as a directory under data_dir/:

data/
  my-index/
    wal/          WAL segments (crash-safe write buffer)
    seg-000001/   Immutable on-disk segment (after first flush)
    seg-000002/
    schema.json   Field mapping definition

Documents are written to the WAL and an in-memory memtable first. The memtable is flushed to a durable segment when it exceeds flush_size_mb (default 512 MiB) or flush_interval_secs (default 30 s). Segments are periodically merged in the background.

By default, xerj requires an API key on every request. On first startup, the admin key is auto-generated and written to <data_dir>/admin.key:

curl -H "Authorization: ApiKey <key>" http://localhost:9200/_cluster/health

Use --insecure to disable auth and TLS in development environments.

xerj is open source. No ERU licensing. No per-node fees. No feature gates.

All capabilities — vector search, full-text search, aggregations, SQL, log ingestion, embedding proxy — are available in the single open-source binary with no paid tiers.

xerj terms aggregations are exact, not estimates.

Elasticsearch uses the HyperLogLog++ algorithm for cardinality and can return approximate bucket counts for terms aggregations under high cardinality. xerj computes every aggregation over the full document set with precise counts — there is no sampling, no approximation, and no doc_count_error_upper_bound.

xerj supports up to 16,384 dimensions per vector field — 4× Elasticsearch's limit of 4,096. This accommodates current and next-generation embedding models:

Set max_dimensions in xerj.toml (default is 16,384).

xerj delegates embedding generation to the configured external model API (e.g. OpenAI, Ollama, or a self-hosted model). Token limits are therefore model-specific, not xerj-specific:

• OpenAI text-embedding-ada-002 — 8,191 tokens per input
• OpenAI text-embedding-3-* — 8,191 tokens per input
• Ollama nomic-embed-text — model-dependent (typically 2,048–8,192)

xerj's chunker (xerj-ai) splits long documents into model-appropriate windows before calling the embedding API. Configure the endpoint and model in [embedding] in your config file.

POST /{index}/_update_by_query performs actual field updates on matching documents. This is the recommended pattern for denormalization cascades:

# Update all orders where customer.id = "C123" to reflect a name change
curl -X POST http://localhost:9200/orders/_update_by_query \
  -H 'Content-Type: application/json' \
  -d '{
    "query": { "term": { "customer.id": "C123" } },
    "script": { "source": "ctx._source.customer.name = params.name",
                "params": { "name": "Acme Corp" } }
  }'

xerj is purpose-built as a data substrate for SIEM (Security Information and Event Management) deployments:

• Log ingestion — syslog (RFC 3164/5424), OTLP, and structured JSON ingest via POST /{index}/syslog and POST /{index}/otlp
• Time-series retention — configurable per-index TTL with automatic deletion
• Full-text + vector search — correlate logs with threat intelligence embeddings
• Aggregations — exact terms and date_histogram for alert thresholds
• Single binary — deploy on-prem or in isolated networks with no external deps

1. Fork the repository and create a feature branch.
2. Run cargo check and cargo test before opening a PR.
3. Keep commits focused; one logical change per commit.
4. All public APIs must have doc-comments.

# Check all crates compile
cargo check

# Run tests
cargo test

# Run a specific crate's tests
cargo test -p xerj-engine

# Lint
cargo clippy -- -D warnings

Apache 2.0 — see LICENSE.