perf(engine): full-corpus agg path deep-cloned the memtable under lock - #42

Summary

search_inner's need_full_corpus block (used whenever a request has aggs and the columnar fast-agg path doesn't apply) built its corpus by deep-cloning the entire memtable while holding the per-shard RwLock read guard, causing severe latency under concurrent dashboard load even on tiny datasets.

Root cause

memtable.all_docs_with_sources() clones every buffered document's JSON tree, and the clone happens inside the s.read() critical section (per-shard parking_lot::RwLock). Under concurrent load — several Kibana dashboard panel queries firing within milliseconds of each other, all touching the same memtable-resident index — this serialises the shard lock across every concurrent request, each paying a full O(doc) clone under it. Measured 1.2–2.7s just for this phase on a ~1000-doc memtable, scaling with the size of the concurrent burst.

The exact same class of bug was already fixed for the fast-agg path via all_docs_with_sources_arc() (Arc-sharing: an O(1) refcount bump under the lock instead of an O(doc) deep clone) — see that function's own doc comment ("was deep-cloning the entire memtable per agg request... ~100-300ms/query at 1e5 buffered docs"). This need_full_corpus path just never got the same treatment.

Fix

Arc-share out of the memtable under the lock, then deep-clone into the owned Value run_aggs_with_all needs after the lock is released. The expensive clone no longer holds up concurrent readers/writers on the same shard.

Test plan

• cargo build --release -p xerj-engine -p xerj-api -p xerj-server
• cargo fmt --check / cargo clippy --no-deps — clean
• cargo test --release -p xerj-engine --lib — 156/156 passed
• 15 concurrent identical date_histogram aggregation queries against a real ~14k-doc memtable-resident index (kibana_sample_data_logs): went from 1.2–2.7s (with slow_query warnings logged) to a consistent ~150–170ms with zero warnings

🤖 Generated with Claude Code