fix(engine): bucket_script resolves a bare-string _count buckets_path - #105
Conversation
|
Holding this one. The Blocker: uncapped recursion in For context on why this gets a hard block: nested SQL High: bare Bound the recursion and the rest looks good. |
…sed ternaries Combines PR xerj-org#95 and PR xerj-org#105 (xerj-org#105's head already contained xerj-org#95's commits verbatim, so this is one stack, not two rival changes) and reworks the parts an adversarial review measured as wrong. Rebased onto main now that xerj-org#111 has landed as xerj-org#115; the `check_expr_limits` entry bound this relies on comes from there. `_count` BUCKETS_PATH — TWO FORMS, ONE INVARIANT. `resolve_bucket_script` now resolves `"<sibling>>_count"` (it looked for a sub-agg literally named `_count`, found nothing, and nulled the bucket) and the bare `"_count"` that Timelion's `.opensearch()` datasource emits for its per-bucket count metric. The bare form refers to the CONTAINING bucket's own doc_count, so `run_aggs_in_bucket` stages one under a transient `doc_count` key for the resolver to read, then strips it back out. The invariant, stated once because getting it wrong is what blocked the previous attempt: whatever number a bucket publishes as its `doc_count`, a bare `_count` inside that bucket resolves to THE SAME number. Not "the weighted count" and not "the physical count" — the bucket aggs in this file split into two families and `_count` has to follow each of them rather than pick a side: sum_doc_count(docs) — honours the `_doc_count` metadata field rollup / downsampled indices attach: terms, date_histogram, histogram, filter, filters, date_range, composite. docs.len() — physical rows, `_doc_count` ignored: range, multi_terms, ip_range, ip_prefix, rare_terms, geo_distance, geotile_grid, geohash_grid, adjacency_matrix, time_series, variable_width_histogram, sampler, diversified_sampler, nested, reverse_nested, global. So `run_aggs_in_bucket` stages the count its CALLER is about to publish, passed in as an argument, rather than recomputing one. Recomputing is wrong with either function, and both spellings were measured on a two-doc corpus `[{s:a,n:1,_doc_count:5},{s:a,n:2,_doc_count:3}]` with a `{"buckets_path": "_count"}` sub-agg: staging sum_doc_count -> range doc_count 2 vs `_count` 8 multi_terms 1 vs 5, and the same split for adjacency_matrix, sampler, diversified_sampler, global, reverse_nested, time_series, rare_terms and variable_width_histogram (10 types, not the 2 the report named) staging docs.len() -> terms doc_count 8 vs `_count` 2 Passing the caller's number gives agreement for all 24 bucket types that run sub-aggs, measured one by one — that sweep is now `bare_count_agrees_with_every_bucket_types_doc_count`, which fails on `range` with the recomputing version restored. significant_terms, significant_text and missing never call into sub-aggs at all, so they have no `_count` surface. At the TOP level of an aggs tree there is no enclosing bucket, so `_count` is the physical size of the result set — the number `hits.total.value` carries in the same response. A previous version of this change asserted in a shipped comment, and in its commit message, that `range` builds its doc_count with `sum_doc_count`. It does not: `run_range` uses `bucket_docs.len()`, as does `run_multi_terms`. Both the comment and this message now describe the code that is actually here. THE EXPRESSION EVALUATOR. Three changes, one of which is a deliberate behaviour delta: * `null` literal, so TSVB's `filter_ratio` guard (`params.x != null && ...`) evaluates instead of nulling the bucket. * Ternaries INSIDE PARENTHESES now evaluate. `find_ternary_split` only ever looked at paren-depth 0, and any `?` that survived into `shunting_yard` made it bail, so `(1?1:0)` — and therefore every nested ternary, and `(params.a > 0 ? 1 : 0) + 1` — silently produced a null bucket value for a legal Elasticsearch script. Measured as `None`, verbatim, for `(1?1:0)`, `(1?1:0) + 1`, `1 + (1?1:0)`, `0?1:(0?2:3)`, `(0?1:2) * 10`, `(1>0 ? 1 : 0) + 1` and `1?(1?5:6):0`. Fixed rather than rejected: rejecting would need an error channel `resolve_pipeline_agg_full` does not have (it returns a Value, and every unevaluable script degrades to `{"value": null}`), and a hard error is the wrong answer for a script ES accepts. `fold_paren_ternaries` collapses each parenthesised group containing a `?` to the number it evaluates to before shunting-yard sees it. Groups without a `?` are untouched. That adds a second recursion path, so the entry bound's argument changed with it: a frame consumes either the `?` it split on or one paren level, and `check_expr_limits` caps both at MAX_EXPR_DEPTH, giving 2*MAX_EXPR_DEPTH + 1 frames rather than MAX_EXPR_DEPTH + 1. The doc comment says so now, and `parenthesised_ternaries_stay_inside_the_entry_bound` exercises a chain that alternates the two forms. * `!=` ANSWERS DIFFERENTLY AGAINST NaN. This is the delta an earlier summary denied. `==` / `!=` are derived from one equality test so they can also compare the `null` literal; `!=` is now the negation of `==` instead of its own `(a - b).abs() >= 1e-9`. Every comparison against NaN is false, so the old form answered `NaN != 0` with 0.0 — "they are equal". Measured end to end on `params.a * params.a - params.a * params.a != 0` over one doc with `n = 1e308` (the product overflows to +inf; inf - inf is NaN): 0.0 with the old arms restored on this tree, 1.0 with the current form. `==` is unchanged, 0.0 both ways, and non-NaN operands keep the identical 1e-9 epsilon. 1.0 is what IEEE-754, and therefore Java, and therefore Painless say, so the new answer is kept — but it is a change, it is pinned by `ne_against_a_nan_operand_is_true_not_false` and `nan_operand_ne_reports_inequality_end_to_end`, and it is no longer described as "identical NaN behaviour". TEST CORRECTION. `sibling_underscore_count_matches_a_doc_count_weighted_filter_bucket` was documented as pinning the weighted staging. It does not: with the staging reverted to `docs.len()` it still passes, and only `bare_count_buckets_path_matches_a_doc_count_weighted_bucket` fails. What it actually pins is `"<sibling>>_count"` resolving at all. Its comment now says that, and records the measurement.
…e interpreter
`fast_aggs` is a second, independent implementation of the same aggregations,
and the bare `"buckets_path": "_count"` support in the previous commit only
landed in the brute interpreter. Per bucket the fast path was already right —
it calls `resolve_sibling_pipelines` on the finished bucket map, which carries
the real `doc_count` key — but at the TOP level of an aggs tree
`eval_aggs_object` resolved pipelines against a map with no `doc_count` in it at
all, so the path resolved to nothing and the bucket reported `{"value": null}`.
That is not a theoretical gap: the fast path takes over once an index passes
`FAST_AGG_MIN_DOCS` (10,000), so the same query changed answer as the index
grew. Measured on one corpus shape, `{"c": {"bucket_script": {"buckets_path":
"_count", "script": "_value"}}}` with `size: 0`:
100 docs -> {"value": 100.0} (brute interpreter)
12,000 docs -> {"value": null} (doc-values fast path)
12,000 docs -> {"value": 12000.0} (same corpus, XERJ_DISABLE_FAST_AGGS=1)
`eval_aggs_object` now takes the doc_count of the level it is evaluating and
stages it exactly as `aggs::run_aggs_in_bucket` does — guarded the same way, so
an aggs tree with no pipeline in it does not touch the result map, and a
user-defined agg actually named `doc_count` still wins. Its two callers pass the
number that level publishes: the filtered total (or the whole live corpus when
no top-level query narrowed it) for the top level, and the `global` bucket's own
count under `global`. Physical rows in both cases, matching what `docs.len()`
gives the interpreter — the fast path is gated on there being no deletes, so
live and physical agree.
After the change the 12,000-doc case returns 12000.0 on the default build.
`test_bare_count_bucket_script_agrees_below_and_above_the_fast_agg_threshold`
runs both corpus sizes and asserts the same answer from whichever path served
them; `test_bare_count_bucket_script_inside_a_terms_bucket_agrees_on_both_paths`
pins the per-bucket case that was already correct, so a later refactor cannot
fix one and break the other.
Deriving `==` from `(x - y).abs() < 1e-9` alone is wrong for infinities:
`inf - inf` is NaN, NaN fails the epsilon test, so two IDENTICAL values
compared UNEQUAL. Measured on the brute path with one doc `{"n": 1e308}` and
`params.a * params.a != params.a * params.a`: 0.0 before the ternary work,
1.0 after. IEEE-754 and the JLS both say `inf == inf`, so the old answer was
the right one and this reintroduced it as a regression.
The product overflowing to +inf is exactly the shape the NaN test already
builds on, so this is reachable from an ordinary script rather than a
contrived one.
Exact equality first, epsilon second. NaN still reports unequal, because
`NaN == NaN` is false and `(NaN).abs() < 1e-9` is false, so
`ne_against_a_nan_operand_is_true_not_false` is unaffected. Mutation-checked:
removing the `x == y` short-circuit fails the new test and nothing else.
Also corrects a comment that its own sibling test disproved. It claimed the
sub-agg-named-`_count` case was "the full extent of the silent-change
surface"; `nan_operand_ne_reports_inequality_end_to_end`, in the same file,
pins a second previously-non-null bucket_script changing value, and the
infinity case above is a third.
e1622b8 to
7963e62
Compare
|
Rebased onto current
Parenthesised ternaries evaluate. Two equal infinities compare equal again. Deriving Behaviour changes, stated in full rather than summarised. An earlier draft claimed "identical NaN behaviour" and a comment claimed the sub-agg-named-
Verified after the rebase: Thanks for the original fix and for #95 underneath it; the |
Summary
Stacked on #95 (needs the
_count/null/ternary script-evaluator work from that PR; this branch is based on it, notmain).resolve_bucket_scriptrequiredbuckets_path.as_object()unconditionally, returning null immediately for ES's "simple form" —buckets_pathas a bare string rather than an alias map. In that form ES binds the resolved value to the single script variable_value, and the bare string"_count"(nosibling>prefix at all) refers to the containing bucket's owndoc_count— distinct from #95's fix, which only handled"alias": "sibling>_count"(a named sibling's count, inside the object form).This is exactly what Kibana/OpenSearch Dashboards' Timelion
.opensearch()datasource emits for its per-time-bucket count metric whenever asplitis present:nested inside each
date_histogrambucket, itself nested inside atermssplit, itself nested inside thefiltersindex-selection wrapper.Found live
Root-causing
[Logs] Chart and Visualization demo's "(Timeline) Stacked extensions over time" panel: real per-bucketdoc_countpresent throughout the response, but everycount.valuenull — the chart rendered as an empty flat line at 0 despite its TSVB sibling panel showing the correct stacked series for the same field.Fix
Two parts, because fixing
resolve_bucket_scriptalone wasn't sufficient:resolve_bucket_script: added a string-form branch — binds the resolved value to_value, with"_count"resolving to the current bucket viasiblings.get("doc_count"), and any other bare string resolving via the same sibling-lookup logic the object form already uses (siblingorsibling>_count/sibling>metric).run_aggs_with_all: the brute-force per-bucket resolver only ever callsresolve_sibling_pipelineson the top-level aggs result map, which has nodoc_countkey. Aterms/date_histogrambucket's sub-aggs are evaluated by a recursiverun_aggs_with_all(sub, bucket_docs, all_docs)call whose returned map never includes the containing bucket's owndoc_count(that gets merged in separately by the caller building the bucket wrapper) — so a nested bucket_script had no way to see it, at any nesting depth. Fixed by staging the current doc slice's own count (docs.len()) under a transientdoc_countkey immediately beforeresolve_sibling_pipelinesruns, then stripping it back out — it isn't a real sibling agg and must not leak into the returned aggs tree. Guarded against the unlikely case of a user-defined agg literally nameddoc_count, which wins over the synthetic value.fast_aggs.rs's ownresolve_sibling_pipelinescall site was already unaffected — it passes the full bucket map,doc_countincluded, by construction.A note on PR #103
While root-causing this, I found that PR #103 (multi-index keyed bucket-agg merge) had never actually been included in the combined test build I was verifying fixes against this session — an integration-branch bookkeeping gap on my end, not a code issue. The "(Timeline) Avg bytes over time" panel I'd earlier reported as fixed and verified had, in fact, only been rendering correctly by coincidence of index-iteration order (whichever index xerj happened to process first for that particular query happened to be the right one). With #103 actually included, both panels are now confirmed working for the right reason rather than by luck — flagging this for the historical record since it affects how much weight to put on earlier "verified live" claims from panels sharing this multi-index
_all+filtersshape.Test plan
cargo fmt --check -p xerj-enginecargo clippy -p xerj-engine --all-targets -- -D warningscargo test -p xerj-engine --lib(296/296; 2 pre-existing unrelated failures confirmed present on unmodified code — macOS/tmpsymlink path resolution insnapshot_path_security_tests)bucket_script_resolves_bare_string_underscore_count_buckets_path, nested inside atermsbucket to match the real shape (bare_countonly means anything inside a containing bucket)curlagainst the exact captured live Timelion query: everytime_bucketsentry'scount.valuenow mirrors itsdoc_count(previously null throughout). Confirmed in the browser: "(Timeline) Stacked extensions over time" — flat empty line at 0 since before this investigation started — now renders the correct stacked area chart, matching its TSVB sibling panel's shape and per-extension proportions.🤖 Generated with Claude Code
https://claude.ai/code/session_014PRCbyt7Y2HDyhG1tbQTeL