Skip to content

fix(api): apply the request-time script guard on _msearch, and bound the bucket_script evaluator ahead of #95 - #115

Merged
xerj-org merged 1 commit into
mainfrom
fix/issue-111
Aug 1, 2026
Merged

fix(api): apply the request-time script guard on _msearch, and bound the bucket_script evaluator ahead of #95#115
xerj-org merged 1 commit into
mainfrom
fix/issue-111

Conversation

@xerj-org

@xerj-org xerj-org commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Closes #111

Two changes with different urgency, stated separately because the issue conflated them and an earlier draft of this branch claimed a process kill that shipped XERJ never had.

Live defect: /_msearch skipped the script guard

/_msearch built its sub-requests straight off parse_request, skipping build_search_request and therefore the request-time script validation the single-search path applies. That is a general bypass of the guard, not something specific to any one script shape.

Confirmed against the real handler: a 240,531-byte _msearch body whose items carry scripts that find_script_limit_violation rejects came back 200 on every item, where _search returns 400 for the same scripts. _search/template and _msearch/template skipped the same guard on their rendered bodies.

All three now run it. _msearch applies it per sub-search item, failing only the offending item with a 400 in ES's per-item error shape and letting the rest of the batch run. Response-array alignment is preserved.

Pre-emptive bound: the evaluator is not recursive today, and is about to be

The issue describes unbounded recursion in eval_tokens. That function does not exist on main. eval_script_expr is tokenize_scriptshunting_yardevaluate_rpn, all three iterative, and ?/: are not even tokens (Tok has only Num/Op/LParen/RParen). Measured on an optimized build, the issue's own 80,001-byte nested-ternary fixture returns None in 4.5 µs. No released version was process-killable through this path.

The recursion arrives with PR #95, which adds eval_tokens and find_ternary_split and re-enters the evaluator once per ?. With #95 applied and this guard bypassed, the same source overflows the worker stack and, because the release profile sets panic = "abort", aborts the process. So the bound lands first and #95 lands onto it.

check_expr_limits now runs on entry to eval_script_expr, before a single token is produced, rejecting a source past 64 KiB, past MAX_EXPR_DEPTH (64) paren nesting, or carrying more than MAX_EXPR_DEPTH ternaries. That matches sql::MAX_SQL_DEPTH and xerj-query's MAX_QUERY_DEPTH. Ternaries are counted rather than depth-tracked because a right-associative chain (a?b:c?d:e) descends one level per ? while never opening a paren.

Blast radius

eval_script_expr has exactly two production callers, bucket_selector and resolve_bucket_script; both now go through the entry bound. On the current tree the new limits are unreachable via HTTP: a bucket_script of N nested parens already 400s at N≥50 from the pre-existing painless request guard, the 64 KiB cap duplicates a 400 that already existed, and the ternary cap is inert because ? is not tokenizable yet. Net user-visible change on this tree: none that could be constructed.

…the bucket_script evaluator ahead of #95

Two changes with different urgency. Stating that plainly because an earlier
draft of this message claimed a process kill that shipped XERJ never had.

LIVE DEFECT — `/_msearch` skipped the script guard.

`/_msearch` built its sub-requests straight off `parse_request`, skipping
`build_search_request` and therefore the request-time script validation the
single-search path applies. That is a general bypass of the guard, not
something specific to any one script shape: a 240 KB `_msearch` body whose
items carry scripts that `_search` rejects with a 400 came back 200 on every
item. `_search/template` and `_msearch/template` skipped the same guard on
their rendered bodies. All three now run it, `_msearch` per sub-search item,
failing only the offending item with a 400 in ES's per-item error shape and
letting the rest of the batch run.

PRE-EMPTIVE BOUND — the evaluator is not recursive today, and is about to be.

On this tree `eval_script_expr` is `tokenize_script` -> `shunting_yard` ->
`evaluate_rpn`, all three iterative; `?` and `:` are not even tokens. An
80,001-byte nested-ternary source returns `None` in 4.5 us on an optimized
build. There is no stack growth to exhaust and no released version was
process-killable through this path.

The recursion arrives with PR #95, which adds `eval_tokens` and
`find_ternary_split` and re-enters the evaluator once per `?`. With #95
applied and this guard bypassed, the same 80,001-byte source overflows the
worker stack and, because the release profile sets `panic = "abort"`, aborts
the process rather than returning an error. So the bound lands first and #95
lands onto it.

`check_expr_limits` now runs on ENTRY to `eval_script_expr`, before a single
token is produced, rejecting a source past 64 KiB, past `MAX_EXPR_DEPTH`
(64) paren nesting, or carrying more than `MAX_EXPR_DEPTH` ternaries, which
matches `sql::MAX_SQL_DEPTH` and `xerj-query`'s `MAX_QUERY_DEPTH`. Ternaries
are counted rather than depth-tracked because a right-associative chain
(`a?b:c?d:e`) descends one level per `?` while never opening a paren. A
violation resolves to the same null bucket value every other unevaluable
script already produces.

Closes #111
@xerj-org
xerj-org merged commit 5594d89 into main Aug 1, 2026
9 checks passed
xerj-org added a commit that referenced this pull request Aug 1, 2026
#117)

The bound landed in #115 with three comments asserting that this evaluator is
recursive and can be made to abort the process. It is not, and it cannot. The
commit message was corrected before that PR merged; the source comments were
not, so main currently documents a crash that no released version had.

What is actually there: `eval_script_expr` is `tokenize_script` ->
`shunting_yard` -> `evaluate_rpn`, three flat loops over a `Vec`, and `?` is
not a token at all (`tokenize_script` returns `None` on it). Measured with the
bound removed on a 2 MiB stack: the 80,001-byte / 20,000-ternary source returns
`None`, and a 1,000,000-deep parenthesised expression returns `Some(1.0)`.
Nothing overflows.

The recursion arrives with the `eval_tokens` / `find_ternary_split` evaluator,
which re-enters once per `?`. The bound is real and worth keeping; it is simply
pre-emptive, landing ahead of that evaluator rather than behind a live defect.
The comments now say so.

No functional change. Only doc comments move.
xerj-org added a commit that referenced this pull request Aug 1, 2026
PR #115 closed a real bypass — `_msearch`, `_search/template` and
`_msearch/template` skip `build_search_request`, so the request-time script
guard never ran there — but it closed it with a walk over the WHOLE
sub-request body, while `_search` guards six specific fields (`query`,
`rescore`, `sort`, `script_fields`, `runtime_mappings`, `aggs`). That made
the multi-search API stricter than `_search`.

Measured on main through the real router, identical body to both endpoints:

  * an over-limit `script` under `highlight`, `suggest`, `docvalue_fields`
    or `collapse.inner_hits`: `_search` 200, `_msearch` 400
  * `aggs` ordinary + `aggregations` over-limit: `_search` 200, `_msearch`
    400 — and neither endpoint even executes that value, since the typed and
    the raw resolver both take the FIRST spelling present
    (`parse_request`: `obj.get("aggs").or_else(|| obj.get("aggregations"))`)

`GuardedField` is now the single definition of what the guard walks. Both
resolvers — `in_search_body` for the typed `EsSearchBody` (`_search`, scroll,
async search) and `in_raw_body` for the raw bodies handed straight to
`parse_request` — are exhaustive matches over it, so a field cannot be
guarded on one path and skipped on another. The `aggs`/`aggregations` pair is
one value in both, resolved the way the executor resolves it, not two keys
checked independently.

Not a loosening of the guard: an over-limit script in any of the six fields
is still rejected on all four entry points, and a hostile `_msearch` item is
still rejected on its own while its siblings run. Top-level `knn` stays
outside the set with a tripwire test: `_search` folds it into the guarded
`query`, and `_msearch` does not execute it at all (a `knn.filter` matching
nothing returns 1 hit there vs 0 on `_search`).

The tests fire one body at `_search`, `_msearch`, `_search/template` and
`_msearch/template` through the real router and require the verdicts to
match; the field table is driven off `GuardedField::ALL`, so a new variant
cannot ship without a fixture every entry point is checked against.
Reverting only the `_msearch` call site to the whole-body walk fails three of
them.
xerj-org added a commit to Vinz2168/xerj that referenced this pull request Aug 2, 2026
…sed ternaries

Combines PR xerj-org#95 and PR xerj-org#105 (xerj-org#105's head already contained xerj-org#95's commits
verbatim, so this is one stack, not two rival changes) and reworks the parts an
adversarial review measured as wrong. Rebased onto main now that xerj-org#111 has landed
as xerj-org#115; the `check_expr_limits` entry bound this relies on comes from there.

`_count` BUCKETS_PATH — TWO FORMS, ONE INVARIANT.

`resolve_bucket_script` now resolves `"<sibling>>_count"` (it looked for a
sub-agg literally named `_count`, found nothing, and nulled the bucket) and the
bare `"_count"` that Timelion's `.opensearch()` datasource emits for its
per-bucket count metric. The bare form refers to the CONTAINING bucket's own
doc_count, so `run_aggs_in_bucket` stages one under a transient `doc_count` key
for the resolver to read, then strips it back out.

The invariant, stated once because getting it wrong is what blocked the previous
attempt: whatever number a bucket publishes as its `doc_count`, a bare `_count`
inside that bucket resolves to THE SAME number. Not "the weighted count" and not
"the physical count" — the bucket aggs in this file split into two families and
`_count` has to follow each of them rather than pick a side:

  sum_doc_count(docs) — honours the `_doc_count` metadata field rollup /
    downsampled indices attach: terms, date_histogram, histogram, filter,
    filters, date_range, composite.

  docs.len() — physical rows, `_doc_count` ignored: range, multi_terms,
    ip_range, ip_prefix, rare_terms, geo_distance, geotile_grid, geohash_grid,
    adjacency_matrix, time_series, variable_width_histogram, sampler,
    diversified_sampler, nested, reverse_nested, global.

So `run_aggs_in_bucket` stages the count its CALLER is about to publish, passed
in as an argument, rather than recomputing one. Recomputing is wrong with either
function, and both spellings were measured on a two-doc corpus
`[{s:a,n:1,_doc_count:5},{s:a,n:2,_doc_count:3}]` with a `{"buckets_path":
"_count"}` sub-agg:

  staging sum_doc_count  -> range   doc_count 2 vs `_count` 8
                            multi_terms 1 vs 5, and the same split for
                            adjacency_matrix, sampler, diversified_sampler,
                            global, reverse_nested, time_series, rare_terms and
                            variable_width_histogram (10 types, not the 2 the
                            report named)
  staging docs.len()     -> terms   doc_count 8 vs `_count` 2

Passing the caller's number gives agreement for all 24 bucket types that run
sub-aggs, measured one by one — that sweep is now
`bare_count_agrees_with_every_bucket_types_doc_count`, which fails on `range`
with the recomputing version restored. significant_terms, significant_text and
missing never call into sub-aggs at all, so they have no `_count` surface.

At the TOP level of an aggs tree there is no enclosing bucket, so `_count` is
the physical size of the result set — the number `hits.total.value` carries in
the same response.

A previous version of this change asserted in a shipped comment, and in its
commit message, that `range` builds its doc_count with `sum_doc_count`. It does
not: `run_range` uses `bucket_docs.len()`, as does `run_multi_terms`. Both the
comment and this message now describe the code that is actually here.

THE EXPRESSION EVALUATOR.

Three changes, one of which is a deliberate behaviour delta:

* `null` literal, so TSVB's `filter_ratio` guard
  (`params.x != null && ...`) evaluates instead of nulling the bucket.

* Ternaries INSIDE PARENTHESES now evaluate. `find_ternary_split` only ever
  looked at paren-depth 0, and any `?` that survived into `shunting_yard` made
  it bail, so `(1?1:0)` — and therefore every nested ternary, and
  `(params.a > 0 ? 1 : 0) + 1` — silently produced a null bucket value for a
  legal Elasticsearch script. Measured as `None`, verbatim, for `(1?1:0)`,
  `(1?1:0) + 1`, `1 + (1?1:0)`, `0?1:(0?2:3)`, `(0?1:2) * 10`,
  `(1>0 ? 1 : 0) + 1` and `1?(1?5:6):0`.

  Fixed rather than rejected: rejecting would need an error channel
  `resolve_pipeline_agg_full` does not have (it returns a Value, and every
  unevaluable script degrades to `{"value": null}`), and a hard error is the
  wrong answer for a script ES accepts. `fold_paren_ternaries` collapses each
  parenthesised group containing a `?` to the number it evaluates to before
  shunting-yard sees it. Groups without a `?` are untouched.

  That adds a second recursion path, so the entry bound's argument changed with
  it: a frame consumes either the `?` it split on or one paren level, and
  `check_expr_limits` caps both at MAX_EXPR_DEPTH, giving 2*MAX_EXPR_DEPTH + 1
  frames rather than MAX_EXPR_DEPTH + 1. The doc comment says so now, and
  `parenthesised_ternaries_stay_inside_the_entry_bound` exercises a chain that
  alternates the two forms.

* `!=` ANSWERS DIFFERENTLY AGAINST NaN. This is the delta an earlier summary
  denied. `==` / `!=` are derived from one equality test so they can also
  compare the `null` literal; `!=` is now the negation of `==` instead of its
  own `(a - b).abs() >= 1e-9`. Every comparison against NaN is false, so the old
  form answered `NaN != 0` with 0.0 — "they are equal". Measured end to end on
  `params.a * params.a - params.a * params.a != 0` over one doc with
  `n = 1e308` (the product overflows to +inf; inf - inf is NaN): 0.0 with the
  old arms restored on this tree, 1.0 with the current form. `==` is unchanged,
  0.0 both ways, and non-NaN operands keep the identical 1e-9 epsilon.

  1.0 is what IEEE-754, and therefore Java, and therefore Painless say, so the
  new answer is kept — but it is a change, it is pinned by
  `ne_against_a_nan_operand_is_true_not_false` and
  `nan_operand_ne_reports_inequality_end_to_end`, and it is no longer described
  as "identical NaN behaviour".

TEST CORRECTION.

`sibling_underscore_count_matches_a_doc_count_weighted_filter_bucket` was
documented as pinning the weighted staging. It does not: with the staging
reverted to `docs.len()` it still passes, and only
`bare_count_buckets_path_matches_a_doc_count_weighted_bucket` fails. What it
actually pins is `"<sibling>>_count"` resolving at all. Its comment now says
that, and records the measurement.
xerj-org added a commit that referenced this pull request Aug 2, 2026
)

* fix(api): one script-guard definition for every search entry point

PR #115 closed a real bypass — `_msearch`, `_search/template` and
`_msearch/template` skip `build_search_request`, so the request-time script
guard never ran there — but it closed it with a walk over the WHOLE
sub-request body, while `_search` guards six specific fields (`query`,
`rescore`, `sort`, `script_fields`, `runtime_mappings`, `aggs`). That made
the multi-search API stricter than `_search`.

Measured on main through the real router, identical body to both endpoints:

  * an over-limit `script` under `highlight`, `suggest`, `docvalue_fields`
    or `collapse.inner_hits`: `_search` 200, `_msearch` 400
  * `aggs` ordinary + `aggregations` over-limit: `_search` 200, `_msearch`
    400 — and neither endpoint even executes that value, since the typed and
    the raw resolver both take the FIRST spelling present
    (`parse_request`: `obj.get("aggs").or_else(|| obj.get("aggregations"))`)

`GuardedField` is now the single definition of what the guard walks. Both
resolvers — `in_search_body` for the typed `EsSearchBody` (`_search`, scroll,
async search) and `in_raw_body` for the raw bodies handed straight to
`parse_request` — are exhaustive matches over it, so a field cannot be
guarded on one path and skipped on another. The `aggs`/`aggregations` pair is
one value in both, resolved the way the executor resolves it, not two keys
checked independently.

Not a loosening of the guard: an over-limit script in any of the six fields
is still rejected on all four entry points, and a hostile `_msearch` item is
still rejected on its own while its siblings run. Top-level `knn` stays
outside the set with a tripwire test: `_search` folds it into the guarded
`query`, and `_msearch` does not execute it at all (a `knn.filter` matching
nothing returns 1 hit there vs 0 on `_search`).

The tests fire one body at `_search`, `_msearch`, `_search/template` and
`_msearch/template` through the real router and require the verdicts to
match; the field table is driven off `GuardedField::ALL`, so a new variant
cannot ship without a fixture every entry point is checked against.
Reverting only the `_msearch` call site to the whole-body walk fails three of
them.

* fix(api): catch a shrinking guard set, and state the aggs precedence rule exactly

Follow-up to the guard-unification commit, closing three things verification
found. No change to what any request answers.

A COMPILE-TIME WIDTH ASSERTION. Adding a `GuardedField` variant was already a
compile error in both exhaustive matches. Removing one was not, and that is the
dangerous direction: every test over the guarded set iterates `ALL`, so they
prove the guard covers whatever `ALL` currently holds rather than six specific
fields. Measured — deleting `Self::Rescore` leaves the entire suite green while
an 80,001-byte `rescore` script goes 400 -> 200 on all four entry points, which
is exactly the bypass #111 closed. `const _: () = assert!(ALL.len() == 6)` now
makes that a build failure; verified by mutation, it fails with
"evaluation panicked: assertion failed: GuardedField::ALL.len() == 6".

THE AGGS PRECEDENCE RULE WAS AMBIGUOUS. Three places said "first spelling
present wins", which reads as document order. `serde_json` preserves key order
workspace-wide, so document order is observable and it is NOT the rule: with
`{"aggregations": <over-limit>, "aggs": <ordinary>}` all four endpoints answer
200 and execute the `aggs` value even though it is written second. The rule is
key-name precedence — `aggs` wins whenever both are present — and the docs now
say that.

THE BLAST RADIUS WAS UNDER-NAMED. The behaviour-delta list named four unguarded
fields plus the aggs pair. The actual change is the whole class: on `_msearch`,
`_search/template` and `_msearch/template`, an over-limit script under ANY key
other than the six guarded ones goes 400 -> 200. Measured on eight further
top-level keys beyond those declared: post_filter, _source, fields, ext,
stored_fields, pit, indices_boost, and a top-level inner_hits object. None is a
script sink, so nothing executes and the direction is the ES-compatible one,
but a reviewer reading the old list would have expected four fields.

Two bodies now disagree between the typed and raw paths where main happened to
agree: a `knn.filter` carrying a script, and `{"aggs": null, "aggregations":
<over-limit>}`. Main's agreement was accidental — the whole-body walk 400'd
everything — and no script executes on the raw paths in either case.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expression evaluator (eval_tokens) has no depth bound, and /_msearch skips the script guard

1 participant