fix(engine): fast_aggs resolves <field>.keyword to its parent's column - #99

…ields

**Stacked on xerj-org#99** (needs `SegEntry::col` from that PR — this branch is
based on it, not on a fresh `upstream/main`).

A genuinely nested JSON field (`geo.dest`, `machine.ram`, the
doubly-nested `machine.os.keyword`) has no column of its own in any
segment — and, contrary to my first attempt at this fix, neither does
its parent object (`geo`, `machine`): fast_aggs' `.dv` columns only
ever cover scalar leaf fields, so an object-typed field never gets a
column either, live or synthetic. xerj-org#99 fixed the adjacent case (a text
field's `.keyword` multi-field sharing its parent's column), but that
fix only strips a single trailing `.keyword` suffix — it doesn't help
a genuinely nested path with no `.keyword` suffix (`geo.dest`), or a
doubly-nested one where stripping `.keyword` still doesn't reach a
real column (`machine.os.keyword` -> `machine.os`, which also isn't a
column; nothing under `machine` is).

Every fast-path executor (`exec_terms`, `exec_metric_top`,
`exec_cardinality`, ...) keys off `params["field"]`, and each
independently treated "no column found" as "field genuinely absent" —
a valid ES outcome for a sparse/dynamic field — instead of "fast_aggs
structurally cannot serve this field, escalate to the brute-force
path that walks `_source` directly and resolves nested paths
correctly regardless of storage layout."

Found live across three of the shipped sample dashboards:
- `geo.dest` terms (`(Horizontal Bar)`/`(VisBuilder) Top destination
  count`, `(Region Map) Destination count`) — empty
- `machine.ram` avg (`(Vega) Average machine RAM` — threw "Cannot
  convert undefined or null to object"; `(Goal) Average machine RAM`
  — showed 0% instead of ~64%)
- `machine.os.keyword` terms (`(Heatmap)`/`(Vega) Source vs OS`,
  `[Logs] Visitors by OS`) — empty

In every case the TSVB sibling panel for the SAME field showed
correct data, because TSVB's request shape happens to bypass the
buggy fast path entirely — proving the data and the aggregation logic
were both fine; only the fast-path field resolution was broken.

Fixed with a single guard at `exec_agg`'s dispatch point (every
concrete `exec_*` function is called from here, keyed on
`params["field"]`): `field_needs_brute_fallback` returns true when a
dotted field resolves via neither the exact name nor the
`.keyword`-stripped fallback, AND its top-level root is mapped
`object`/`nested` in the index's SCHEMA (not, as a first attempt
tried and a live debug trace disproved, whether the root happens to
exist as a physical column — it never does, since `.dv` columns only
ever cover scalar leaves). The schema snapshot is taken once per
request under the existing async `schema.read().await` lock (the same
pattern already used for `bool_fields`) and handed to the sync
executor as a `HashSet<String>` of object/nested field names.

That combination positively identifies "real nested path fast_aggs
can't reach" and excludes "field genuinely never indexed" (whose root
isn't schema-mapped at all — a true absence, correctly served as
empty by the fast path with no fallback needed).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PRCbyt7Y2HDyhG1tbQTeL

…ields (#104)

`fast_aggs` resolved a nested field path against the segment column map and,
when the lookup failed, carried on with a predicate that matched nothing. A
nested field the fast path could not resolve therefore produced a plausible
small number instead of an error or a correct result, while the brute path
computed the right answer for the same request.

The fast path now detects the unresolvable case and hands the aggregation to
the brute-force implementation rather than answering from a resolution it does
not have. Falling back is slower for that request and correct, which is the
right trade for a silent wrong number.

Rebased onto current main; the branch was previously stacked on #99, which has
since merged.

This PR gives exec_terms a bucket cap, so the FAST_PATH_AGGS_SERVED comment
that cited it as a still-uncapped divergence is now false. Replaced with the
current list (#99/#104/#114/#120 closed, #128 nested-object still open).

…brute (#134)

* fix(aggs): the columnar terms path honours config.limits.max_buckets (#121)

`fast_aggs::exec_terms` built its term map with no bucket cap of any kind,
only a `size`-derived OUTPUT cap. An operator who lowered
`config.limits.max_buckets` to protect memory got that protection only on
queries that missed the columnar path: measured at max_buckets=37 over 200
distinct terms, the fast path (a 12k-doc index) materialised all 200
(top5=300 + sum_other_doc_count=11,700) while the brute `run_terms` stopped
at 37. That is the OOM vector the cap exists to close — the half PR #125
left open after fixing the histogram executors.

The two executors must AGREE past the cap. Brute `run_terms` does not error
there; it keeps the first `max_buckets` distinct terms in doc-iteration
order, drops the rest, and reports a `sum_other_doc_count` computed only
over the terms it kept, so the dropped terms vanish silently and the total
conceals them (order-dependent). Mirroring that truncation onto the fast
path would copy the bug, not the contract; erroring would make the fast
path disagree with a brute path that still returns a body.

So `exec_terms` BAILS to brute the moment its distinct-term count would
exceed the cap, checked as the map grows (per segment and after the
memtable) so the map is bounded to ~`max_buckets`. After the bail the
request is answered by the very `run_terms` a test measures the fast path
against, so the two AGREE past the cap by construction — the same shape as
the #104 / #120 bail-to-brute fallbacks. An index whose distinct count is
at or under the cap is served columnarly as before, and the default-cap
(65,536) common case is untouched.

Test (its own binary, one function, like agg_bucket_cap.rs, because the cap
lives in a process-wide static): a 10,050-doc index over 200 terms at cap 37
is no longer served columnarly (`fast_path_aggs_served` does not tick) and
honours the cap (37 buckets, not 200); a 600-doc brute index of the same
shape agrees on the count; and an at-cap index (exactly 37 terms) is still
served columnarly, guarding against over-bailing. Reverting only
fast_aggs.rs fails it: served==1, 200 buckets against max_buckets=37.

* doc(aggs): drop the now-false exec_terms 'no bucket cap' note

This PR gives exec_terms a bucket cap, so the FAST_PATH_AGGS_SERVED comment
that cited it as a still-uncapped divergence is now false. Replaced with the
current list (#99/#104/#114/#120 closed, #128 nested-object still open).