fix(api): apply the request-time script guard on _msearch, and bound the bucket_script evaluator ahead of #95 - #115
Merged
Merged
Conversation
…the bucket_script evaluator ahead of #95 Two changes with different urgency. Stating that plainly because an earlier draft of this message claimed a process kill that shipped XERJ never had. LIVE DEFECT — `/_msearch` skipped the script guard. `/_msearch` built its sub-requests straight off `parse_request`, skipping `build_search_request` and therefore the request-time script validation the single-search path applies. That is a general bypass of the guard, not something specific to any one script shape: a 240 KB `_msearch` body whose items carry scripts that `_search` rejects with a 400 came back 200 on every item. `_search/template` and `_msearch/template` skipped the same guard on their rendered bodies. All three now run it, `_msearch` per sub-search item, failing only the offending item with a 400 in ES's per-item error shape and letting the rest of the batch run. PRE-EMPTIVE BOUND — the evaluator is not recursive today, and is about to be. On this tree `eval_script_expr` is `tokenize_script` -> `shunting_yard` -> `evaluate_rpn`, all three iterative; `?` and `:` are not even tokens. An 80,001-byte nested-ternary source returns `None` in 4.5 us on an optimized build. There is no stack growth to exhaust and no released version was process-killable through this path. The recursion arrives with PR #95, which adds `eval_tokens` and `find_ternary_split` and re-enters the evaluator once per `?`. With #95 applied and this guard bypassed, the same 80,001-byte source overflows the worker stack and, because the release profile sets `panic = "abort"`, aborts the process rather than returning an error. So the bound lands first and #95 lands onto it. `check_expr_limits` now runs on ENTRY to `eval_script_expr`, before a single token is produced, rejecting a source past 64 KiB, past `MAX_EXPR_DEPTH` (64) paren nesting, or carrying more than `MAX_EXPR_DEPTH` ternaries, which matches `sql::MAX_SQL_DEPTH` and `xerj-query`'s `MAX_QUERY_DEPTH`. Ternaries are counted rather than depth-tracked because a right-associative chain (`a?b:c?d:e`) descends one level per `?` while never opening a paren. A violation resolves to the same null bucket value every other unevaluable script already produces. Closes #111
xerj-org
added a commit
that referenced
this pull request
Aug 1, 2026
#117) The bound landed in #115 with three comments asserting that this evaluator is recursive and can be made to abort the process. It is not, and it cannot. The commit message was corrected before that PR merged; the source comments were not, so main currently documents a crash that no released version had. What is actually there: `eval_script_expr` is `tokenize_script` -> `shunting_yard` -> `evaluate_rpn`, three flat loops over a `Vec`, and `?` is not a token at all (`tokenize_script` returns `None` on it). Measured with the bound removed on a 2 MiB stack: the 80,001-byte / 20,000-ternary source returns `None`, and a 1,000,000-deep parenthesised expression returns `Some(1.0)`. Nothing overflows. The recursion arrives with the `eval_tokens` / `find_ternary_split` evaluator, which re-enters once per `?`. The bound is real and worth keeping; it is simply pre-emptive, landing ahead of that evaluator rather than behind a live defect. The comments now say so. No functional change. Only doc comments move.
xerj-org
added a commit
that referenced
this pull request
Aug 1, 2026
PR #115 closed a real bypass — `_msearch`, `_search/template` and `_msearch/template` skip `build_search_request`, so the request-time script guard never ran there — but it closed it with a walk over the WHOLE sub-request body, while `_search` guards six specific fields (`query`, `rescore`, `sort`, `script_fields`, `runtime_mappings`, `aggs`). That made the multi-search API stricter than `_search`. Measured on main through the real router, identical body to both endpoints: * an over-limit `script` under `highlight`, `suggest`, `docvalue_fields` or `collapse.inner_hits`: `_search` 200, `_msearch` 400 * `aggs` ordinary + `aggregations` over-limit: `_search` 200, `_msearch` 400 — and neither endpoint even executes that value, since the typed and the raw resolver both take the FIRST spelling present (`parse_request`: `obj.get("aggs").or_else(|| obj.get("aggregations"))`) `GuardedField` is now the single definition of what the guard walks. Both resolvers — `in_search_body` for the typed `EsSearchBody` (`_search`, scroll, async search) and `in_raw_body` for the raw bodies handed straight to `parse_request` — are exhaustive matches over it, so a field cannot be guarded on one path and skipped on another. The `aggs`/`aggregations` pair is one value in both, resolved the way the executor resolves it, not two keys checked independently. Not a loosening of the guard: an over-limit script in any of the six fields is still rejected on all four entry points, and a hostile `_msearch` item is still rejected on its own while its siblings run. Top-level `knn` stays outside the set with a tripwire test: `_search` folds it into the guarded `query`, and `_msearch` does not execute it at all (a `knn.filter` matching nothing returns 1 hit there vs 0 on `_search`). The tests fire one body at `_search`, `_msearch`, `_search/template` and `_msearch/template` through the real router and require the verdicts to match; the field table is driven off `GuardedField::ALL`, so a new variant cannot ship without a fixture every entry point is checked against. Reverting only the `_msearch` call site to the whole-body walk fails three of them.
xerj-org
added a commit
to Vinz2168/xerj
that referenced
this pull request
Aug 2, 2026
…sed ternaries Combines PR xerj-org#95 and PR xerj-org#105 (xerj-org#105's head already contained xerj-org#95's commits verbatim, so this is one stack, not two rival changes) and reworks the parts an adversarial review measured as wrong. Rebased onto main now that xerj-org#111 has landed as xerj-org#115; the `check_expr_limits` entry bound this relies on comes from there. `_count` BUCKETS_PATH — TWO FORMS, ONE INVARIANT. `resolve_bucket_script` now resolves `"<sibling>>_count"` (it looked for a sub-agg literally named `_count`, found nothing, and nulled the bucket) and the bare `"_count"` that Timelion's `.opensearch()` datasource emits for its per-bucket count metric. The bare form refers to the CONTAINING bucket's own doc_count, so `run_aggs_in_bucket` stages one under a transient `doc_count` key for the resolver to read, then strips it back out. The invariant, stated once because getting it wrong is what blocked the previous attempt: whatever number a bucket publishes as its `doc_count`, a bare `_count` inside that bucket resolves to THE SAME number. Not "the weighted count" and not "the physical count" — the bucket aggs in this file split into two families and `_count` has to follow each of them rather than pick a side: sum_doc_count(docs) — honours the `_doc_count` metadata field rollup / downsampled indices attach: terms, date_histogram, histogram, filter, filters, date_range, composite. docs.len() — physical rows, `_doc_count` ignored: range, multi_terms, ip_range, ip_prefix, rare_terms, geo_distance, geotile_grid, geohash_grid, adjacency_matrix, time_series, variable_width_histogram, sampler, diversified_sampler, nested, reverse_nested, global. So `run_aggs_in_bucket` stages the count its CALLER is about to publish, passed in as an argument, rather than recomputing one. Recomputing is wrong with either function, and both spellings were measured on a two-doc corpus `[{s:a,n:1,_doc_count:5},{s:a,n:2,_doc_count:3}]` with a `{"buckets_path": "_count"}` sub-agg: staging sum_doc_count -> range doc_count 2 vs `_count` 8 multi_terms 1 vs 5, and the same split for adjacency_matrix, sampler, diversified_sampler, global, reverse_nested, time_series, rare_terms and variable_width_histogram (10 types, not the 2 the report named) staging docs.len() -> terms doc_count 8 vs `_count` 2 Passing the caller's number gives agreement for all 24 bucket types that run sub-aggs, measured one by one — that sweep is now `bare_count_agrees_with_every_bucket_types_doc_count`, which fails on `range` with the recomputing version restored. significant_terms, significant_text and missing never call into sub-aggs at all, so they have no `_count` surface. At the TOP level of an aggs tree there is no enclosing bucket, so `_count` is the physical size of the result set — the number `hits.total.value` carries in the same response. A previous version of this change asserted in a shipped comment, and in its commit message, that `range` builds its doc_count with `sum_doc_count`. It does not: `run_range` uses `bucket_docs.len()`, as does `run_multi_terms`. Both the comment and this message now describe the code that is actually here. THE EXPRESSION EVALUATOR. Three changes, one of which is a deliberate behaviour delta: * `null` literal, so TSVB's `filter_ratio` guard (`params.x != null && ...`) evaluates instead of nulling the bucket. * Ternaries INSIDE PARENTHESES now evaluate. `find_ternary_split` only ever looked at paren-depth 0, and any `?` that survived into `shunting_yard` made it bail, so `(1?1:0)` — and therefore every nested ternary, and `(params.a > 0 ? 1 : 0) + 1` — silently produced a null bucket value for a legal Elasticsearch script. Measured as `None`, verbatim, for `(1?1:0)`, `(1?1:0) + 1`, `1 + (1?1:0)`, `0?1:(0?2:3)`, `(0?1:2) * 10`, `(1>0 ? 1 : 0) + 1` and `1?(1?5:6):0`. Fixed rather than rejected: rejecting would need an error channel `resolve_pipeline_agg_full` does not have (it returns a Value, and every unevaluable script degrades to `{"value": null}`), and a hard error is the wrong answer for a script ES accepts. `fold_paren_ternaries` collapses each parenthesised group containing a `?` to the number it evaluates to before shunting-yard sees it. Groups without a `?` are untouched. That adds a second recursion path, so the entry bound's argument changed with it: a frame consumes either the `?` it split on or one paren level, and `check_expr_limits` caps both at MAX_EXPR_DEPTH, giving 2*MAX_EXPR_DEPTH + 1 frames rather than MAX_EXPR_DEPTH + 1. The doc comment says so now, and `parenthesised_ternaries_stay_inside_the_entry_bound` exercises a chain that alternates the two forms. * `!=` ANSWERS DIFFERENTLY AGAINST NaN. This is the delta an earlier summary denied. `==` / `!=` are derived from one equality test so they can also compare the `null` literal; `!=` is now the negation of `==` instead of its own `(a - b).abs() >= 1e-9`. Every comparison against NaN is false, so the old form answered `NaN != 0` with 0.0 — "they are equal". Measured end to end on `params.a * params.a - params.a * params.a != 0` over one doc with `n = 1e308` (the product overflows to +inf; inf - inf is NaN): 0.0 with the old arms restored on this tree, 1.0 with the current form. `==` is unchanged, 0.0 both ways, and non-NaN operands keep the identical 1e-9 epsilon. 1.0 is what IEEE-754, and therefore Java, and therefore Painless say, so the new answer is kept — but it is a change, it is pinned by `ne_against_a_nan_operand_is_true_not_false` and `nan_operand_ne_reports_inequality_end_to_end`, and it is no longer described as "identical NaN behaviour". TEST CORRECTION. `sibling_underscore_count_matches_a_doc_count_weighted_filter_bucket` was documented as pinning the weighted staging. It does not: with the staging reverted to `docs.len()` it still passes, and only `bare_count_buckets_path_matches_a_doc_count_weighted_bucket` fails. What it actually pins is `"<sibling>>_count"` resolving at all. Its comment now says that, and records the measurement.
xerj-org
added a commit
that referenced
this pull request
Aug 2, 2026
) * fix(api): one script-guard definition for every search entry point PR #115 closed a real bypass — `_msearch`, `_search/template` and `_msearch/template` skip `build_search_request`, so the request-time script guard never ran there — but it closed it with a walk over the WHOLE sub-request body, while `_search` guards six specific fields (`query`, `rescore`, `sort`, `script_fields`, `runtime_mappings`, `aggs`). That made the multi-search API stricter than `_search`. Measured on main through the real router, identical body to both endpoints: * an over-limit `script` under `highlight`, `suggest`, `docvalue_fields` or `collapse.inner_hits`: `_search` 200, `_msearch` 400 * `aggs` ordinary + `aggregations` over-limit: `_search` 200, `_msearch` 400 — and neither endpoint even executes that value, since the typed and the raw resolver both take the FIRST spelling present (`parse_request`: `obj.get("aggs").or_else(|| obj.get("aggregations"))`) `GuardedField` is now the single definition of what the guard walks. Both resolvers — `in_search_body` for the typed `EsSearchBody` (`_search`, scroll, async search) and `in_raw_body` for the raw bodies handed straight to `parse_request` — are exhaustive matches over it, so a field cannot be guarded on one path and skipped on another. The `aggs`/`aggregations` pair is one value in both, resolved the way the executor resolves it, not two keys checked independently. Not a loosening of the guard: an over-limit script in any of the six fields is still rejected on all four entry points, and a hostile `_msearch` item is still rejected on its own while its siblings run. Top-level `knn` stays outside the set with a tripwire test: `_search` folds it into the guarded `query`, and `_msearch` does not execute it at all (a `knn.filter` matching nothing returns 1 hit there vs 0 on `_search`). The tests fire one body at `_search`, `_msearch`, `_search/template` and `_msearch/template` through the real router and require the verdicts to match; the field table is driven off `GuardedField::ALL`, so a new variant cannot ship without a fixture every entry point is checked against. Reverting only the `_msearch` call site to the whole-body walk fails three of them. * fix(api): catch a shrinking guard set, and state the aggs precedence rule exactly Follow-up to the guard-unification commit, closing three things verification found. No change to what any request answers. A COMPILE-TIME WIDTH ASSERTION. Adding a `GuardedField` variant was already a compile error in both exhaustive matches. Removing one was not, and that is the dangerous direction: every test over the guarded set iterates `ALL`, so they prove the guard covers whatever `ALL` currently holds rather than six specific fields. Measured — deleting `Self::Rescore` leaves the entire suite green while an 80,001-byte `rescore` script goes 400 -> 200 on all four entry points, which is exactly the bypass #111 closed. `const _: () = assert!(ALL.len() == 6)` now makes that a build failure; verified by mutation, it fails with "evaluation panicked: assertion failed: GuardedField::ALL.len() == 6". THE AGGS PRECEDENCE RULE WAS AMBIGUOUS. Three places said "first spelling present wins", which reads as document order. `serde_json` preserves key order workspace-wide, so document order is observable and it is NOT the rule: with `{"aggregations": <over-limit>, "aggs": <ordinary>}` all four endpoints answer 200 and execute the `aggs` value even though it is written second. The rule is key-name precedence — `aggs` wins whenever both are present — and the docs now say that. THE BLAST RADIUS WAS UNDER-NAMED. The behaviour-delta list named four unguarded fields plus the aggs pair. The actual change is the whole class: on `_msearch`, `_search/template` and `_msearch/template`, an over-limit script under ANY key other than the six guarded ones goes 400 -> 200. Measured on eight further top-level keys beyond those declared: post_filter, _source, fields, ext, stored_fields, pit, indices_boost, and a top-level inner_hits object. None is a script sink, so nothing executes and the direction is the ES-compatible one, but a reviewer reading the old list would have expected four fields. Two bodies now disagree between the typed and raw paths where main happened to agree: a `knn.filter` carrying a script, and `{"aggs": null, "aggregations": <over-limit>}`. Main's agreement was accidental — the whole-body walk 400'd everything — and no script executes on the raw paths in either case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #111
Two changes with different urgency, stated separately because the issue conflated them and an earlier draft of this branch claimed a process kill that shipped XERJ never had.
Live defect:
/_msearchskipped the script guard/_msearchbuilt its sub-requests straight offparse_request, skippingbuild_search_requestand therefore the request-time script validation the single-search path applies. That is a general bypass of the guard, not something specific to any one script shape.Confirmed against the real handler: a 240,531-byte
_msearchbody whose items carry scripts thatfind_script_limit_violationrejects came back 200 on every item, where_searchreturns 400 for the same scripts._search/templateand_msearch/templateskipped the same guard on their rendered bodies.All three now run it.
_msearchapplies it per sub-search item, failing only the offending item with a 400 in ES's per-item error shape and letting the rest of the batch run. Response-array alignment is preserved.Pre-emptive bound: the evaluator is not recursive today, and is about to be
The issue describes unbounded recursion in
eval_tokens. That function does not exist onmain.eval_script_expristokenize_script→shunting_yard→evaluate_rpn, all three iterative, and?/:are not even tokens (Tokhas onlyNum/Op/LParen/RParen). Measured on an optimized build, the issue's own 80,001-byte nested-ternary fixture returnsNonein 4.5 µs. No released version was process-killable through this path.The recursion arrives with PR #95, which adds
eval_tokensandfind_ternary_splitand re-enters the evaluator once per?. With #95 applied and this guard bypassed, the same source overflows the worker stack and, because the release profile setspanic = "abort", aborts the process. So the bound lands first and #95 lands onto it.check_expr_limitsnow runs on entry toeval_script_expr, before a single token is produced, rejecting a source past 64 KiB, pastMAX_EXPR_DEPTH(64) paren nesting, or carrying more thanMAX_EXPR_DEPTHternaries. That matchessql::MAX_SQL_DEPTHandxerj-query'sMAX_QUERY_DEPTH. Ternaries are counted rather than depth-tracked because a right-associative chain (a?b:c?d:e) descends one level per?while never opening a paren.Blast radius
eval_script_exprhas exactly two production callers,bucket_selectorandresolve_bucket_script; both now go through the entry bound. On the current tree the new limits are unreachable via HTTP: abucket_scriptof N nested parens already 400s at N≥50 from the pre-existing painless request guard, the 64 KiB cap duplicates a 400 that already existed, and the ternary cap is inert because?is not tokenizable yet. Net user-visible change on this tree: none that could be constructed.