Skip to content

fix(engine): fast_aggs bails to brute force for unresolvable nested fields - #104

Merged
xerj-org merged 1 commit into
xerj-org:mainfrom
Vinz2168:fix/nested-field-brute-fallback
Aug 1, 2026
Merged

fix(engine): fast_aggs bails to brute force for unresolvable nested fields#104
xerj-org merged 1 commit into
xerj-org:mainfrom
Vinz2168:fix/nested-field-brute-fallback

Conversation

@Vinz2168

@Vinz2168 Vinz2168 commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

Stacked on #99 (needs SegEntry::col; this branch is based on that PR's branch, not main).

A genuinely nested JSON field (geo.dest, machine.ram, the doubly-nested machine.os.keyword) has no column of its own anywhere in fast_aggs' columnar storage — .dv columns only ever cover scalar leaf fields, so even the parent object (geo, machine) never gets a column either. #99 fixed the adjacent case (a text field's .keyword multi-field sharing its parent's column via a single trailing-suffix strip), but that doesn't help a genuinely nested path with no .keyword suffix at all (geo.dest), or a doubly-nested one where stripping .keyword still doesn't land on a real column (machine.os.keyword -> machine.os, also not a column).

Every fast-path executor (exec_terms, exec_metric_top, exec_cardinality, ...) keys off params["field"], and each independently treated "no column found" the same as "field genuinely absent" — a valid ES outcome for a sparse/dynamic field — instead of "fast_aggs structurally cannot serve this field, escalate to the brute-force path that walks _source directly and resolves nested paths correctly regardless of storage layout." Net effect: silent empty buckets / null metrics for real, populated fields.

Found live

Auditing every shipped OpenSearch Dashboards sample dashboard turned up 7 broken panels across two dashboards, all sharing this one root cause:

  • geo.dest terms: (Horizontal Bar), (VisBuilder), (Vega) Top destination count, (Region Map) Destination count — all empty
  • machine.ram avg: (Vega) Average machine RAM — threw "Cannot convert undefined or null to object"; (Goal) Average machine RAM — showed 0% instead of the real ~64%
  • machine.os.keyword terms: (Heatmap)/(Vega) Source vs OS, [Logs] Visitors by OS — all empty

In every case the TSVB sibling panel for the same field showed correct data, because TSVB's request shape happens to bypass the buggy fast path entirely — proving the data and the aggregation logic were both fine; only fast-path field resolution was broken.

First attempt was wrong — worth documenting

My first fix attempted to detect this by checking whether the dotted field's top-level root existed as a physical column in SegEntry (the same struct .keyword-stripping in #99 reads). All 5 synthetic unit tests passed. It did nothing live: a debug trace showed col("geo") returns None on the real index despite geo being mapped object — object-typed fields never get a .dv column at all, not even under their own name, so "does the root resolve as a column" can never distinguish a genuinely nested path from a genuinely absent one. Root storage layout, not synthetic test data, was the thing to check against.

Fix

field_needs_brute_fallback now checks the schema instead: a dotted field resolving to no column (neither the exact name nor .keyword-stripped) whose top-level root is mapped object/nested is a genuine nested path fast_aggs structurally cannot reach — bail to brute force. A field whose root isn't schema-mapped at all is a true absence, correctly served empty by the fast path with no fallback needed.

The schema snapshot (HashSet<String> of object/nested field names) is taken once per request under the existing async schema.read().await lock, mirroring the pre-existing bool_fields snapshot used for the same reason (sync executor, async schema lock), and handed to FastCtx alongside it.

Test plan

🤖 Generated with Claude Code

https://claude.ai/code/session_014PRCbyt7Y2HDyhG1tbQTeL

…ields

**Stacked on xerj-org#99** (needs `SegEntry::col` from that PR — this branch is
based on it, not on a fresh `upstream/main`).

A genuinely nested JSON field (`geo.dest`, `machine.ram`, the
doubly-nested `machine.os.keyword`) has no column of its own in any
segment — and, contrary to my first attempt at this fix, neither does
its parent object (`geo`, `machine`): fast_aggs' `.dv` columns only
ever cover scalar leaf fields, so an object-typed field never gets a
column either, live or synthetic. xerj-org#99 fixed the adjacent case (a text
field's `.keyword` multi-field sharing its parent's column), but that
fix only strips a single trailing `.keyword` suffix — it doesn't help
a genuinely nested path with no `.keyword` suffix (`geo.dest`), or a
doubly-nested one where stripping `.keyword` still doesn't reach a
real column (`machine.os.keyword` -> `machine.os`, which also isn't a
column; nothing under `machine` is).

Every fast-path executor (`exec_terms`, `exec_metric_top`,
`exec_cardinality`, ...) keys off `params["field"]`, and each
independently treated "no column found" as "field genuinely absent" —
a valid ES outcome for a sparse/dynamic field — instead of "fast_aggs
structurally cannot serve this field, escalate to the brute-force
path that walks `_source` directly and resolves nested paths
correctly regardless of storage layout."

Found live across three of the shipped sample dashboards:
- `geo.dest` terms (`(Horizontal Bar)`/`(VisBuilder) Top destination
  count`, `(Region Map) Destination count`) — empty
- `machine.ram` avg (`(Vega) Average machine RAM` — threw "Cannot
  convert undefined or null to object"; `(Goal) Average machine RAM`
  — showed 0% instead of ~64%)
- `machine.os.keyword` terms (`(Heatmap)`/`(Vega) Source vs OS`,
  `[Logs] Visitors by OS`) — empty

In every case the TSVB sibling panel for the SAME field showed
correct data, because TSVB's request shape happens to bypass the
buggy fast path entirely — proving the data and the aggregation logic
were both fine; only the fast-path field resolution was broken.

Fixed with a single guard at `exec_agg`'s dispatch point (every
concrete `exec_*` function is called from here, keyed on
`params["field"]`): `field_needs_brute_fallback` returns true when a
dotted field resolves via neither the exact name nor the
`.keyword`-stripped fallback, AND its top-level root is mapped
`object`/`nested` in the index's SCHEMA (not, as a first attempt
tried and a live debug trace disproved, whether the root happens to
exist as a physical column — it never does, since `.dv` columns only
ever cover scalar leaves). The schema snapshot is taken once per
request under the existing async `schema.read().await` lock (the same
pattern already used for `bool_fields`) and handed to the sync
executor as a `HashSet<String>` of object/nested field names.

That combination positively identifies "real nested path fast_aggs
can't reach" and excludes "field genuinely never indexed" (whose root
isn't schema-mapped at all — a true absence, correctly served as
empty by the fast path with no fallback needed).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PRCbyt7Y2HDyhG1tbQTeL
@xerj-org
xerj-org force-pushed the fix/nested-field-brute-fallback branch from e0ddf90 to 9fb1a53 Compare August 1, 2026 21:25
@xerj-org
xerj-org merged commit 8accf18 into xerj-org:main Aug 1, 2026
9 checks passed
xerj-org added a commit that referenced this pull request Aug 2, 2026
…121)

`fast_aggs::exec_terms` built its term map with no bucket cap of any kind,
only a `size`-derived OUTPUT cap. An operator who lowered
`config.limits.max_buckets` to protect memory got that protection only on
queries that missed the columnar path: measured at max_buckets=37 over 200
distinct terms, the fast path (a 12k-doc index) materialised all 200
(top5=300 + sum_other_doc_count=11,700) while the brute `run_terms` stopped
at 37. That is the OOM vector the cap exists to close — the half PR #125
left open after fixing the histogram executors.

The two executors must AGREE past the cap. Brute `run_terms` does not error
there; it keeps the first `max_buckets` distinct terms in doc-iteration
order, drops the rest, and reports a `sum_other_doc_count` computed only
over the terms it kept, so the dropped terms vanish silently and the total
conceals them (order-dependent). Mirroring that truncation onto the fast
path would copy the bug, not the contract; erroring would make the fast
path disagree with a brute path that still returns a body.

So `exec_terms` BAILS to brute the moment its distinct-term count would
exceed the cap, checked as the map grows (per segment and after the
memtable) so the map is bounded to ~`max_buckets`. After the bail the
request is answered by the very `run_terms` a test measures the fast path
against, so the two AGREE past the cap by construction — the same shape as
the #104 / #120 bail-to-brute fallbacks. An index whose distinct count is
at or under the cap is served columnarly as before, and the default-cap
(65,536) common case is untouched.

Test (its own binary, one function, like agg_bucket_cap.rs, because the cap
lives in a process-wide static): a 10,050-doc index over 200 terms at cap 37
is no longer served columnarly (`fast_path_aggs_served` does not tick) and
honours the cap (37 buckets, not 200); a 600-doc brute index of the same
shape agrees on the count; and an at-cap index (exactly 37 terms) is still
served columnarly, guarding against over-bailing. Reverting only
fast_aggs.rs fails it: served==1, 200 buckets against max_buckets=37.
xerj-org added a commit that referenced this pull request Aug 2, 2026
This PR gives exec_terms a bucket cap, so the FAST_PATH_AGGS_SERVED comment
that cited it as a still-uncapped divergence is now false. Replaced with the
current list (#99/#104/#114/#120 closed, #128 nested-object still open).
xerj-org added a commit that referenced this pull request Aug 2, 2026
…brute (#134)

* fix(aggs): the columnar terms path honours config.limits.max_buckets (#121)

`fast_aggs::exec_terms` built its term map with no bucket cap of any kind,
only a `size`-derived OUTPUT cap. An operator who lowered
`config.limits.max_buckets` to protect memory got that protection only on
queries that missed the columnar path: measured at max_buckets=37 over 200
distinct terms, the fast path (a 12k-doc index) materialised all 200
(top5=300 + sum_other_doc_count=11,700) while the brute `run_terms` stopped
at 37. That is the OOM vector the cap exists to close — the half PR #125
left open after fixing the histogram executors.

The two executors must AGREE past the cap. Brute `run_terms` does not error
there; it keeps the first `max_buckets` distinct terms in doc-iteration
order, drops the rest, and reports a `sum_other_doc_count` computed only
over the terms it kept, so the dropped terms vanish silently and the total
conceals them (order-dependent). Mirroring that truncation onto the fast
path would copy the bug, not the contract; erroring would make the fast
path disagree with a brute path that still returns a body.

So `exec_terms` BAILS to brute the moment its distinct-term count would
exceed the cap, checked as the map grows (per segment and after the
memtable) so the map is bounded to ~`max_buckets`. After the bail the
request is answered by the very `run_terms` a test measures the fast path
against, so the two AGREE past the cap by construction — the same shape as
the #104 / #120 bail-to-brute fallbacks. An index whose distinct count is
at or under the cap is served columnarly as before, and the default-cap
(65,536) common case is untouched.

Test (its own binary, one function, like agg_bucket_cap.rs, because the cap
lives in a process-wide static): a 10,050-doc index over 200 terms at cap 37
is no longer served columnarly (`fast_path_aggs_served` does not tick) and
honours the cap (37 buckets, not 200); a 600-doc brute index of the same
shape agrees on the count; and an at-cap index (exactly 37 terms) is still
served columnarly, guarding against over-bailing. Reverting only
fast_aggs.rs fails it: served==1, 200 buckets against max_buckets=37.

* doc(aggs): drop the now-false exec_terms 'no bucket cap' note

This PR gives exec_terms a bucket cap, so the FAST_PATH_AGGS_SERVED comment
that cited it as a still-uncapped divergence is now false. Replaced with the
current list (#99/#104/#114/#120 closed, #128 nested-object still open).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants