fix(api): merge keyed (object-shaped) bucket aggs across multi-index search - #103

Summary

This is the actual root cause of an OpenSearch Dashboards Timeline (Timelion) panel that rendered empty against xerj. Multi-index search runs the same aggregation independently per index and merges the per-index results — merge_bucket_agg, the merger for multi-bucket aggregations, only handled array-shaped buckets (terms, date_histogram, ...). A keyed multi-bucket agg — filters in its default named-filter form, or terms/range/date_range with keyed: true — represents buckets as a JSON object (bucket name → bucket body), not an array. For that shape the function always returned None (.as_array() on an object is None).

The caller's fallback path made the identical mistake (also called .as_array() on the keyed buckets object), so should_replace was always false. Net effect: across N indices, only the first index's contribution to a keyed bucket ever survived — every other index's matching documents were silently dropped from the merged aggregation, even though hits.total summed correctly across those same indices. The query matching was fine; only the aggregation merge was broken.

How this was found

Root-causing an OpenSearch Dashboards Timeline panel across three layers, in order:

1. First found and fixed query_string picking only the first text field with no default_field (fix(engine): query_string with no default_field searches every text field #102) — real, but insufficient alone.
2. Then found X-OpenSearch-Version missing from xerj's responses (fix(api): send X-OpenSearch-Version on responses to OpenSearch callers #101) — real, also insufficient alone.
3. With both applied, the exact real Timelion query (captured live via temporary request logging, removed before this PR) still returned doc_count: 0 for the target index's filters bucket on a multi-index (_all) search — while the identical query on a single index returned the correct count. That isolated it to the merge step, not query execution.

Traced to merge_bucket_agg: reproduced directly — filters + field-less query_string, run on _all across 26 indices, one of which alone holds ~14,074 matching documents, returned doc_count: 0 for that filter bucket. Whichever index xerj happened to process first (almost never the one actually being searched for — most of the 26 indices are internal .xerj_*/.kibana_1 system indices) "won", and the real index's contribution — sent later in iteration order — was discarded by the broken merge.

Fix

Added a keyed-object merge path to merge_bucket_agg, parallel to the existing array path: sum doc_count per matching object key, recursively merge nested sub-aggregations the same way the array path already does (reusing merge_metric_agg / merge_bucket_agg / merge_single_bucket_agg), and add any key present in only one side. Falls through to the pre-existing array path unchanged for every non-keyed aggregation — zero behavior change for terms, date_histogram, histogram, range (non-keyed), composite, etc.

Test plan

• cargo fmt --check
• cargo clippy -p xerj-api --all-targets -- -D warnings
• cargo test -p xerj-api --lib (112/112)
• 3 new regression tests: keyed_filters_agg_sums_doc_count_across_indices, keyed_filters_agg_merges_nested_sub_aggregation, keyed_filters_agg_adds_new_key_from_second_index
• Live verification, full chain: rebuilt a combined binary with this fix + fix(api): send X-OpenSearch-Version on responses to OpenSearch callers #101 + fix(engine): query_string with no default_field searches every text field #102 + the two other fixes from the same session (fix(engine): bucket_script resolves TSVB filter_ratio's _count + null/ternary script #95, fix(engine): fast_aggs resolves <field>.keyword to its parent's column #99), redeployed against the real long-running test environment, re-ran the exact captured live Timelion request — doc_count went from 0 to the correct 3769, avg(bytes) buckets went from all-null to 143/171 real values. Confirmed in the browser: the "(Timeline) Avg bytes over time" panel — empty since before this investigation started — now renders a populated line chart, matching its TSVB sibling panel.

🤖 Generated with Claude Code

https://claude.ai/code/session_014PRCbyt7Y2HDyhG1tbQTeL