fix(engine): fast_aggs bails to brute force for unresolvable nested fields - #104
Merged
Merged
Conversation
5 tasks
…ields **Stacked on xerj-org#99** (needs `SegEntry::col` from that PR — this branch is based on it, not on a fresh `upstream/main`). A genuinely nested JSON field (`geo.dest`, `machine.ram`, the doubly-nested `machine.os.keyword`) has no column of its own in any segment — and, contrary to my first attempt at this fix, neither does its parent object (`geo`, `machine`): fast_aggs' `.dv` columns only ever cover scalar leaf fields, so an object-typed field never gets a column either, live or synthetic. xerj-org#99 fixed the adjacent case (a text field's `.keyword` multi-field sharing its parent's column), but that fix only strips a single trailing `.keyword` suffix — it doesn't help a genuinely nested path with no `.keyword` suffix (`geo.dest`), or a doubly-nested one where stripping `.keyword` still doesn't reach a real column (`machine.os.keyword` -> `machine.os`, which also isn't a column; nothing under `machine` is). Every fast-path executor (`exec_terms`, `exec_metric_top`, `exec_cardinality`, ...) keys off `params["field"]`, and each independently treated "no column found" as "field genuinely absent" — a valid ES outcome for a sparse/dynamic field — instead of "fast_aggs structurally cannot serve this field, escalate to the brute-force path that walks `_source` directly and resolves nested paths correctly regardless of storage layout." Found live across three of the shipped sample dashboards: - `geo.dest` terms (`(Horizontal Bar)`/`(VisBuilder) Top destination count`, `(Region Map) Destination count`) — empty - `machine.ram` avg (`(Vega) Average machine RAM` — threw "Cannot convert undefined or null to object"; `(Goal) Average machine RAM` — showed 0% instead of ~64%) - `machine.os.keyword` terms (`(Heatmap)`/`(Vega) Source vs OS`, `[Logs] Visitors by OS`) — empty In every case the TSVB sibling panel for the SAME field showed correct data, because TSVB's request shape happens to bypass the buggy fast path entirely — proving the data and the aggregation logic were both fine; only the fast-path field resolution was broken. Fixed with a single guard at `exec_agg`'s dispatch point (every concrete `exec_*` function is called from here, keyed on `params["field"]`): `field_needs_brute_fallback` returns true when a dotted field resolves via neither the exact name nor the `.keyword`-stripped fallback, AND its top-level root is mapped `object`/`nested` in the index's SCHEMA (not, as a first attempt tried and a live debug trace disproved, whether the root happens to exist as a physical column — it never does, since `.dv` columns only ever cover scalar leaves). The schema snapshot is taken once per request under the existing async `schema.read().await` lock (the same pattern already used for `bool_fields`) and handed to the sync executor as a `HashSet<String>` of object/nested field names. That combination positively identifies "real nested path fast_aggs can't reach" and excludes "field genuinely never indexed" (whose root isn't schema-mapped at all — a true absence, correctly served as empty by the fast path with no fallback needed). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014PRCbyt7Y2HDyhG1tbQTeL
xerj-org
force-pushed
the
fix/nested-field-brute-fallback
branch
from
August 1, 2026 21:25
e0ddf90 to
9fb1a53
Compare
xerj-org
added a commit
that referenced
this pull request
Aug 2, 2026
…121) `fast_aggs::exec_terms` built its term map with no bucket cap of any kind, only a `size`-derived OUTPUT cap. An operator who lowered `config.limits.max_buckets` to protect memory got that protection only on queries that missed the columnar path: measured at max_buckets=37 over 200 distinct terms, the fast path (a 12k-doc index) materialised all 200 (top5=300 + sum_other_doc_count=11,700) while the brute `run_terms` stopped at 37. That is the OOM vector the cap exists to close — the half PR #125 left open after fixing the histogram executors. The two executors must AGREE past the cap. Brute `run_terms` does not error there; it keeps the first `max_buckets` distinct terms in doc-iteration order, drops the rest, and reports a `sum_other_doc_count` computed only over the terms it kept, so the dropped terms vanish silently and the total conceals them (order-dependent). Mirroring that truncation onto the fast path would copy the bug, not the contract; erroring would make the fast path disagree with a brute path that still returns a body. So `exec_terms` BAILS to brute the moment its distinct-term count would exceed the cap, checked as the map grows (per segment and after the memtable) so the map is bounded to ~`max_buckets`. After the bail the request is answered by the very `run_terms` a test measures the fast path against, so the two AGREE past the cap by construction — the same shape as the #104 / #120 bail-to-brute fallbacks. An index whose distinct count is at or under the cap is served columnarly as before, and the default-cap (65,536) common case is untouched. Test (its own binary, one function, like agg_bucket_cap.rs, because the cap lives in a process-wide static): a 10,050-doc index over 200 terms at cap 37 is no longer served columnarly (`fast_path_aggs_served` does not tick) and honours the cap (37 buckets, not 200); a 600-doc brute index of the same shape agrees on the count; and an at-cap index (exactly 37 terms) is still served columnarly, guarding against over-bailing. Reverting only fast_aggs.rs fails it: served==1, 200 buckets against max_buckets=37.
xerj-org
added a commit
that referenced
this pull request
Aug 2, 2026
xerj-org
added a commit
that referenced
this pull request
Aug 2, 2026
…brute (#134) * fix(aggs): the columnar terms path honours config.limits.max_buckets (#121) `fast_aggs::exec_terms` built its term map with no bucket cap of any kind, only a `size`-derived OUTPUT cap. An operator who lowered `config.limits.max_buckets` to protect memory got that protection only on queries that missed the columnar path: measured at max_buckets=37 over 200 distinct terms, the fast path (a 12k-doc index) materialised all 200 (top5=300 + sum_other_doc_count=11,700) while the brute `run_terms` stopped at 37. That is the OOM vector the cap exists to close — the half PR #125 left open after fixing the histogram executors. The two executors must AGREE past the cap. Brute `run_terms` does not error there; it keeps the first `max_buckets` distinct terms in doc-iteration order, drops the rest, and reports a `sum_other_doc_count` computed only over the terms it kept, so the dropped terms vanish silently and the total conceals them (order-dependent). Mirroring that truncation onto the fast path would copy the bug, not the contract; erroring would make the fast path disagree with a brute path that still returns a body. So `exec_terms` BAILS to brute the moment its distinct-term count would exceed the cap, checked as the map grows (per segment and after the memtable) so the map is bounded to ~`max_buckets`. After the bail the request is answered by the very `run_terms` a test measures the fast path against, so the two AGREE past the cap by construction — the same shape as the #104 / #120 bail-to-brute fallbacks. An index whose distinct count is at or under the cap is served columnarly as before, and the default-cap (65,536) common case is untouched. Test (its own binary, one function, like agg_bucket_cap.rs, because the cap lives in a process-wide static): a 10,050-doc index over 200 terms at cap 37 is no longer served columnarly (`fast_path_aggs_served` does not tick) and honours the cap (37 buckets, not 200); a 600-doc brute index of the same shape agrees on the count; and an at-cap index (exactly 37 terms) is still served columnarly, guarding against over-bailing. Reverting only fast_aggs.rs fails it: served==1, 200 buckets against max_buckets=37. * doc(aggs): drop the now-false exec_terms 'no bucket cap' note This PR gives exec_terms a bucket cap, so the FAST_PATH_AGGS_SERVED comment that cited it as a still-uncapped divergence is now false. Replaced with the current list (#99/#104/#114/#120 closed, #128 nested-object still open).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #99 (needs
SegEntry::col; this branch is based on that PR's branch, notmain).A genuinely nested JSON field (
geo.dest,machine.ram, the doubly-nestedmachine.os.keyword) has no column of its own anywhere infast_aggs' columnar storage —.dvcolumns only ever cover scalar leaf fields, so even the parent object (geo,machine) never gets a column either. #99 fixed the adjacent case (a text field's.keywordmulti-field sharing its parent's column via a single trailing-suffix strip), but that doesn't help a genuinely nested path with no.keywordsuffix at all (geo.dest), or a doubly-nested one where stripping.keywordstill doesn't land on a real column (machine.os.keyword->machine.os, also not a column).Every fast-path executor (
exec_terms,exec_metric_top,exec_cardinality, ...) keys offparams["field"], and each independently treated "no column found" the same as "field genuinely absent" — a valid ES outcome for a sparse/dynamic field — instead of "fast_aggs structurally cannot serve this field, escalate to the brute-force path that walks_sourcedirectly and resolves nested paths correctly regardless of storage layout." Net effect: silent empty buckets / null metrics for real, populated fields.Found live
Auditing every shipped OpenSearch Dashboards sample dashboard turned up 7 broken panels across two dashboards, all sharing this one root cause:
geo.destterms:(Horizontal Bar),(VisBuilder),(Vega) Top destination count,(Region Map) Destination count— all emptymachine.ramavg:(Vega) Average machine RAM— threw "Cannot convert undefined or null to object";(Goal) Average machine RAM— showed 0% instead of the real ~64%machine.os.keywordterms:(Heatmap)/(Vega) Source vs OS,[Logs] Visitors by OS— all emptyIn every case the TSVB sibling panel for the same field showed correct data, because TSVB's request shape happens to bypass the buggy fast path entirely — proving the data and the aggregation logic were both fine; only fast-path field resolution was broken.
First attempt was wrong — worth documenting
My first fix attempted to detect this by checking whether the dotted field's top-level root existed as a physical column in
SegEntry(the same struct.keyword-stripping in #99 reads). All 5 synthetic unit tests passed. It did nothing live: a debug trace showedcol("geo")returnsNoneon the real index despitegeobeing mappedobject— object-typed fields never get a.dvcolumn at all, not even under their own name, so "does the root resolve as a column" can never distinguish a genuinely nested path from a genuinely absent one. Root storage layout, not synthetic test data, was the thing to check against.Fix
field_needs_brute_fallbacknow checks the schema instead: a dotted field resolving to no column (neither the exact name nor.keyword-stripped) whose top-level root is mappedobject/nestedis a genuine nested path fast_aggs structurally cannot reach — bail to brute force. A field whose root isn't schema-mapped at all is a true absence, correctly served empty by the fast path with no fallback needed.The schema snapshot (
HashSet<String>of object/nested field names) is taken once per request under the existing asyncschema.read().awaitlock, mirroring the pre-existingbool_fieldssnapshot used for the same reason (sync executor, async schema lock), and handed toFastCtxalongside it.Test plan
cargo fmt --check -p xerj-engine -p xerj-commoncargo clippy -p xerj-engine -p xerj-common --all-targets -- -D warningscargo test -p xerj-engine --lib(2 pre-existing, unrelated failures confirmed present on unmodified86bc050too — macOS/tmpsymlink path resolution insnapshot_path_security_tests, nothing to do with this change)field_needs_brute_fallback, including a case proving the physical-column short-circuit (from fix(engine): fast_aggs resolves <field>.keyword to its parent's column #99) still wins when it appliescurlthatgeo.destterms/machine.ramavg/machine.os.keywordterms all return real data (matching brute-force-path values exactly). Confirmed in the browser across all 7 previously-broken panels on[Logs] Chart and Visualization demoand[Logs] Web Traffic— every one now renders and matches its TSVB sibling.🤖 Generated with Claude Code
https://claude.ai/code/session_014PRCbyt7Y2HDyhG1tbQTeL