perf(engine): concurrent Kibana dashboard bursts stalled the scan path - #48

Summary

Two independent, compounding costs in the brute-force scan path, both specifically hit by a real Kibana dashboard firing several panel queries at once — this is what a user reported as "xerj feels slow when Kibana sends multiple parallel requests," one of the last blockers before trusting xerj as a real Kibana/OSD backend.

Bug 1 — unconditional per-doc _source clone

scan_stored_section_into (the brute-force query-matching loop) deep-cloned every scanned document's _source to splice _id in, even though that injection is only ever needed for a deeply-nested Ids clause (e.g. function_score → ids) — a top-level Ids query is already handled separately, without this clone at all.

Fix: query_needs_id_injection() — an exhaustive match over every QueryNode variant, computed once per scan (not per doc). The match has no wildcard arm on purpose: adding a future QueryNode variant that can nest a sub-query without updating this function is a compile error, so a missed arm can only ever cost the pre-fix clone again, never silently produce a wrong match. The common case (term/match/match_phrase/query_string/bool with no nested ids — the overwhelming majority of real queries) now matches directly against the borrowed source, no clone at all.

Bug 2 — cold segment decode has no single-flight protection

The raw-fallback decode branch (reached whenever neither decoded_stored_cache nor stored_slices_cache is warm for a segment — i.e. every first query against it, most visibly right after a restart) opens and decompresses the segment with no single-flight protection. N concurrent requests racing the same cold segment each independently pay the full open+decompress.

Measured ~4.5s per request, uniformly, across a real 24-way concurrent burst — every request finishing in the same narrow time window is the signature of N-way redundant work, not N independent per-request costs.

Fix: wrapped the decode in the same per-segment mutex (stored_slices_build_locks) the sorted path's stored_slices_for already uses for this exact reason (see its own doc comment: "up to 64 in-flight queries race the same cold segment... one ~30ms decode became a ~1s 64-query stall episode") — with a cache re-check after acquiring the lock so only the first waiter actually decodes; everyone else gets the freshly-published result.

Test plan

• cargo build --release -p xerj-engine -p xerj-api -p xerj-server
• cargo fmt --check / cargo clippy --no-deps -- -D warnings — clean
• cargo test --release -p xerj-engine --lib — 156/156 passed
• Full ES-compat YAML conformance suite: 1360 passed, 0 failed, 3 skipped — no regressions (checked closely for ids-query failures given bug 1 touches that logic)
• Real-world verification against a real Kibana OSS 7.10.2 instance + real eCommerce sample data: a 24-concurrent-request burst of query_string wildcard + match_phrase filters, fired immediately after a fresh restart (worst case — fully cold caches):
  • Before: ~4.5s per request, uniform, zero variance; 32 slow_query warnings logged
  • After: 100–165ms per request; zero slow_query warnings — a ~30–45x improvement on the cold-start case every Kibana dashboard hits after a restart
  • A warm-cache repeat of the identical burst (every subsequent dashboard load) completes in 1–5ms

🤖 Generated with Claude Code