Skip to content

fix(engine): bucket_script resolves a bare-string _count buckets_path - #105

Merged
xerj-org merged 3 commits into
xerj-org:mainfrom
Vinz2168:fix/bucket-script-bare-count-buckets-path
Aug 2, 2026
Merged

fix(engine): bucket_script resolves a bare-string _count buckets_path#105
xerj-org merged 3 commits into
xerj-org:mainfrom
Vinz2168:fix/bucket-script-bare-count-buckets-path

Conversation

@Vinz2168

@Vinz2168 Vinz2168 commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

Stacked on #95 (needs the _count/null/ternary script-evaluator work from that PR; this branch is based on it, not main).

resolve_bucket_script required buckets_path.as_object() unconditionally, returning null immediately for ES's "simple form" — buckets_path as a bare string rather than an alias map. In that form ES binds the resolved value to the single script variable _value, and the bare string "_count" (no sibling> prefix at all) refers to the containing bucket's own doc_count — distinct from #95's fix, which only handled "alias": "sibling>_count" (a named sibling's count, inside the object form).

This is exactly what Kibana/OpenSearch Dashboards' Timelion .opensearch() datasource emits for its per-time-bucket count metric whenever a split is present:

"count": {
  "bucket_script": {
    "buckets_path": "_count",
    "script": {"source": "_value", "lang": "expression"}
  }
}

nested inside each date_histogram bucket, itself nested inside a terms split, itself nested inside the filters index-selection wrapper.

Found live

Root-causing [Logs] Chart and Visualization demo's "(Timeline) Stacked extensions over time" panel: real per-bucket doc_count present throughout the response, but every count.value null — the chart rendered as an empty flat line at 0 despite its TSVB sibling panel showing the correct stacked series for the same field.

Fix

Two parts, because fixing resolve_bucket_script alone wasn't sufficient:

  1. resolve_bucket_script: added a string-form branch — binds the resolved value to _value, with "_count" resolving to the current bucket via siblings.get("doc_count"), and any other bare string resolving via the same sibling-lookup logic the object form already uses (sibling or sibling>_count/sibling>metric).

  2. run_aggs_with_all: the brute-force per-bucket resolver only ever calls resolve_sibling_pipelines on the top-level aggs result map, which has no doc_count key. A terms/date_histogram bucket's sub-aggs are evaluated by a recursive run_aggs_with_all(sub, bucket_docs, all_docs) call whose returned map never includes the containing bucket's own doc_count (that gets merged in separately by the caller building the bucket wrapper) — so a nested bucket_script had no way to see it, at any nesting depth. Fixed by staging the current doc slice's own count (docs.len()) under a transient doc_count key immediately before resolve_sibling_pipelines runs, then stripping it back out — it isn't a real sibling agg and must not leak into the returned aggs tree. Guarded against the unlikely case of a user-defined agg literally named doc_count, which wins over the synthetic value.

fast_aggs.rs's own resolve_sibling_pipelines call site was already unaffected — it passes the full bucket map, doc_count included, by construction.

A note on PR #103

While root-causing this, I found that PR #103 (multi-index keyed bucket-agg merge) had never actually been included in the combined test build I was verifying fixes against this session — an integration-branch bookkeeping gap on my end, not a code issue. The "(Timeline) Avg bytes over time" panel I'd earlier reported as fixed and verified had, in fact, only been rendering correctly by coincidence of index-iteration order (whichever index xerj happened to process first for that particular query happened to be the right one). With #103 actually included, both panels are now confirmed working for the right reason rather than by luck — flagging this for the historical record since it affects how much weight to put on earlier "verified live" claims from panels sharing this multi-index _all + filters shape.

Test plan

🤖 Generated with Claude Code

https://claude.ai/code/session_014PRCbyt7Y2HDyhG1tbQTeL

@xerj-org

xerj-org commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Holding this one. The _count resolution is right, but review found a process kill in the evaluator it uses, plus a semantics problem.

Blocker: uncapped recursion in eval_tokens. An ~80 KB bucket_script overflows the native stack, and with panic = "abort" in the release profile that is a process abort rather than an error response. This is the same finding as on #95, so the two share a root cause: the expression evaluator needs a depth bound in the style of MAX_SQL_DEPTH / MAX_EVAL_DEPTH, and /_msearch needs the request-time script guard the single-search path applies.

For context on why this gets a hard block: nested SQL WHERE parens, nested query_string parens, oversized Painless expressions, and Painless closure recursion have each shipped and been fixed here already. Every new recursive construct over user input needs its bound before it lands.

High: bare _count reports the physical doc count, which contradicts the bucket's own _doc_count weighting. So a weighted bucket and its bucket_script disagree about how many documents are in it.

Bound the recursion and the rest looks good.

…sed ternaries

Combines PR xerj-org#95 and PR xerj-org#105 (xerj-org#105's head already contained xerj-org#95's commits
verbatim, so this is one stack, not two rival changes) and reworks the parts an
adversarial review measured as wrong. Rebased onto main now that xerj-org#111 has landed
as xerj-org#115; the `check_expr_limits` entry bound this relies on comes from there.

`_count` BUCKETS_PATH — TWO FORMS, ONE INVARIANT.

`resolve_bucket_script` now resolves `"<sibling>>_count"` (it looked for a
sub-agg literally named `_count`, found nothing, and nulled the bucket) and the
bare `"_count"` that Timelion's `.opensearch()` datasource emits for its
per-bucket count metric. The bare form refers to the CONTAINING bucket's own
doc_count, so `run_aggs_in_bucket` stages one under a transient `doc_count` key
for the resolver to read, then strips it back out.

The invariant, stated once because getting it wrong is what blocked the previous
attempt: whatever number a bucket publishes as its `doc_count`, a bare `_count`
inside that bucket resolves to THE SAME number. Not "the weighted count" and not
"the physical count" — the bucket aggs in this file split into two families and
`_count` has to follow each of them rather than pick a side:

  sum_doc_count(docs) — honours the `_doc_count` metadata field rollup /
    downsampled indices attach: terms, date_histogram, histogram, filter,
    filters, date_range, composite.

  docs.len() — physical rows, `_doc_count` ignored: range, multi_terms,
    ip_range, ip_prefix, rare_terms, geo_distance, geotile_grid, geohash_grid,
    adjacency_matrix, time_series, variable_width_histogram, sampler,
    diversified_sampler, nested, reverse_nested, global.

So `run_aggs_in_bucket` stages the count its CALLER is about to publish, passed
in as an argument, rather than recomputing one. Recomputing is wrong with either
function, and both spellings were measured on a two-doc corpus
`[{s:a,n:1,_doc_count:5},{s:a,n:2,_doc_count:3}]` with a `{"buckets_path":
"_count"}` sub-agg:

  staging sum_doc_count  -> range   doc_count 2 vs `_count` 8
                            multi_terms 1 vs 5, and the same split for
                            adjacency_matrix, sampler, diversified_sampler,
                            global, reverse_nested, time_series, rare_terms and
                            variable_width_histogram (10 types, not the 2 the
                            report named)
  staging docs.len()     -> terms   doc_count 8 vs `_count` 2

Passing the caller's number gives agreement for all 24 bucket types that run
sub-aggs, measured one by one — that sweep is now
`bare_count_agrees_with_every_bucket_types_doc_count`, which fails on `range`
with the recomputing version restored. significant_terms, significant_text and
missing never call into sub-aggs at all, so they have no `_count` surface.

At the TOP level of an aggs tree there is no enclosing bucket, so `_count` is
the physical size of the result set — the number `hits.total.value` carries in
the same response.

A previous version of this change asserted in a shipped comment, and in its
commit message, that `range` builds its doc_count with `sum_doc_count`. It does
not: `run_range` uses `bucket_docs.len()`, as does `run_multi_terms`. Both the
comment and this message now describe the code that is actually here.

THE EXPRESSION EVALUATOR.

Three changes, one of which is a deliberate behaviour delta:

* `null` literal, so TSVB's `filter_ratio` guard
  (`params.x != null && ...`) evaluates instead of nulling the bucket.

* Ternaries INSIDE PARENTHESES now evaluate. `find_ternary_split` only ever
  looked at paren-depth 0, and any `?` that survived into `shunting_yard` made
  it bail, so `(1?1:0)` — and therefore every nested ternary, and
  `(params.a > 0 ? 1 : 0) + 1` — silently produced a null bucket value for a
  legal Elasticsearch script. Measured as `None`, verbatim, for `(1?1:0)`,
  `(1?1:0) + 1`, `1 + (1?1:0)`, `0?1:(0?2:3)`, `(0?1:2) * 10`,
  `(1>0 ? 1 : 0) + 1` and `1?(1?5:6):0`.

  Fixed rather than rejected: rejecting would need an error channel
  `resolve_pipeline_agg_full` does not have (it returns a Value, and every
  unevaluable script degrades to `{"value": null}`), and a hard error is the
  wrong answer for a script ES accepts. `fold_paren_ternaries` collapses each
  parenthesised group containing a `?` to the number it evaluates to before
  shunting-yard sees it. Groups without a `?` are untouched.

  That adds a second recursion path, so the entry bound's argument changed with
  it: a frame consumes either the `?` it split on or one paren level, and
  `check_expr_limits` caps both at MAX_EXPR_DEPTH, giving 2*MAX_EXPR_DEPTH + 1
  frames rather than MAX_EXPR_DEPTH + 1. The doc comment says so now, and
  `parenthesised_ternaries_stay_inside_the_entry_bound` exercises a chain that
  alternates the two forms.

* `!=` ANSWERS DIFFERENTLY AGAINST NaN. This is the delta an earlier summary
  denied. `==` / `!=` are derived from one equality test so they can also
  compare the `null` literal; `!=` is now the negation of `==` instead of its
  own `(a - b).abs() >= 1e-9`. Every comparison against NaN is false, so the old
  form answered `NaN != 0` with 0.0 — "they are equal". Measured end to end on
  `params.a * params.a - params.a * params.a != 0` over one doc with
  `n = 1e308` (the product overflows to +inf; inf - inf is NaN): 0.0 with the
  old arms restored on this tree, 1.0 with the current form. `==` is unchanged,
  0.0 both ways, and non-NaN operands keep the identical 1e-9 epsilon.

  1.0 is what IEEE-754, and therefore Java, and therefore Painless say, so the
  new answer is kept — but it is a change, it is pinned by
  `ne_against_a_nan_operand_is_true_not_false` and
  `nan_operand_ne_reports_inequality_end_to_end`, and it is no longer described
  as "identical NaN behaviour".

TEST CORRECTION.

`sibling_underscore_count_matches_a_doc_count_weighted_filter_bucket` was
documented as pinning the weighted staging. It does not: with the staging
reverted to `docs.len()` it still passes, and only
`bare_count_buckets_path_matches_a_doc_count_weighted_bucket` fails. What it
actually pins is `"<sibling>>_count"` resolving at all. Its comment now says
that, and records the measurement.
…e interpreter

`fast_aggs` is a second, independent implementation of the same aggregations,
and the bare `"buckets_path": "_count"` support in the previous commit only
landed in the brute interpreter. Per bucket the fast path was already right —
it calls `resolve_sibling_pipelines` on the finished bucket map, which carries
the real `doc_count` key — but at the TOP level of an aggs tree
`eval_aggs_object` resolved pipelines against a map with no `doc_count` in it at
all, so the path resolved to nothing and the bucket reported `{"value": null}`.

That is not a theoretical gap: the fast path takes over once an index passes
`FAST_AGG_MIN_DOCS` (10,000), so the same query changed answer as the index
grew. Measured on one corpus shape, `{"c": {"bucket_script": {"buckets_path":
"_count", "script": "_value"}}}` with `size: 0`:

     100 docs -> {"value": 100.0}     (brute interpreter)
  12,000 docs -> {"value": null}      (doc-values fast path)
  12,000 docs -> {"value": 12000.0}   (same corpus, XERJ_DISABLE_FAST_AGGS=1)

`eval_aggs_object` now takes the doc_count of the level it is evaluating and
stages it exactly as `aggs::run_aggs_in_bucket` does — guarded the same way, so
an aggs tree with no pipeline in it does not touch the result map, and a
user-defined agg actually named `doc_count` still wins. Its two callers pass the
number that level publishes: the filtered total (or the whole live corpus when
no top-level query narrowed it) for the top level, and the `global` bucket's own
count under `global`. Physical rows in both cases, matching what `docs.len()`
gives the interpreter — the fast path is gated on there being no deletes, so
live and physical agree.

After the change the 12,000-doc case returns 12000.0 on the default build.
`test_bare_count_bucket_script_agrees_below_and_above_the_fast_agg_threshold`
runs both corpus sizes and asserts the same answer from whichever path served
them; `test_bare_count_bucket_script_inside_a_terms_bucket_agrees_on_both_paths`
pins the per-bucket case that was already correct, so a later refactor cannot
fix one and break the other.
Deriving `==` from `(x - y).abs() < 1e-9` alone is wrong for infinities:
`inf - inf` is NaN, NaN fails the epsilon test, so two IDENTICAL values
compared UNEQUAL. Measured on the brute path with one doc `{"n": 1e308}` and
`params.a * params.a != params.a * params.a`: 0.0 before the ternary work,
1.0 after. IEEE-754 and the JLS both say `inf == inf`, so the old answer was
the right one and this reintroduced it as a regression.

The product overflowing to +inf is exactly the shape the NaN test already
builds on, so this is reachable from an ordinary script rather than a
contrived one.

Exact equality first, epsilon second. NaN still reports unequal, because
`NaN == NaN` is false and `(NaN).abs() < 1e-9` is false, so
`ne_against_a_nan_operand_is_true_not_false` is unaffected. Mutation-checked:
removing the `x == y` short-circuit fails the new test and nothing else.

Also corrects a comment that its own sibling test disproved. It claimed the
sub-agg-named-`_count` case was "the full extent of the silent-change
surface"; `nan_operand_ne_reports_inequality_end_to_end`, in the same file,
pins a second previously-non-null bucket_script changing value, and the
infinity case above is a third.
@xerj-org
xerj-org force-pushed the fix/bucket-script-bare-count-buckets-path branch from e1622b8 to 7963e62 Compare August 2, 2026 00:08
@xerj-org

xerj-org commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Rebased onto current main and updated to close the review findings. #95's two commits are contained in this branch verbatim, so merging this closes both PRs; #95 will be closed as included rather than merged separately.

_count now agrees with _doc_count everywhere. The original finding was that a _doc_count-weighted bucket and the bucket_script inside it disagreed about how many documents the bucket held. A first attempt fixed terms but inverted the same defect into range and multi_terms, which previously agreed. Verified per agg type on the brute path with a weighted fixture: terms, range, multi_terms, histogram, date_histogram, auto_date_histogram, filter, filters, ip_range, geo_distance, nested and the sampler family all now report _count equal to their own doc_count.

Parenthesised ternaries evaluate. find_ternary_split only looked at paren depth 0, so (1?1:0) — and by extension any nested ternary, and (params.a > 0 ? 1 : 0) + 1 — silently resolved to a null bucket value. That undercut the point of adding ternary support.

Two equal infinities compare equal again. Deriving == from (x - y).abs() < 1e-9 alone is wrong for infinities, because inf - inf is NaN and NaN fails the epsilon test, so two identical values compared unequal. Measured with one doc {"n": 1e308}: params.a * params.a != params.a * params.a answered 1.0 where IEEE-754 and the JLS both say inf == inf. Exact equality is now checked before the epsilon. NaN still reports unequal, which is the IEEE-correct behaviour this branch deliberately introduced and which is separately pinned.

Behaviour changes, stated in full rather than summarised. An earlier draft claimed "identical NaN behaviour" and a comment claimed the sub-agg-named-_count case was "the full extent of the silent-change surface". Both were wrong, and each is now pinned by a test rather than asserted in prose:

  • != against a NaN operand: 0.0 → 1.0. This is the IEEE-correct direction and is intentional.
  • A sub-agg literally named _count: the bucket's doc count now wins, per ES's reserved-path rule.
  • Object-form buckets_paths at a plain numeric sub-metric now resolve where they previously returned null.

Verified after the rebase: cargo test -p xerj-engine --lib346 passed, 0 failed; cargo fmt --check and cargo clippy -D warnings both clean. The infinity fix is mutation-checked — removing the exact-equality short-circuit fails the new test and nothing else.

Thanks for the original fix and for #95 underneath it; the _count resolution was the right call and TSVB's filter_ratio is a real compatibility gap.

@xerj-org
xerj-org merged commit 5df22b7 into xerj-org:main Aug 2, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants