Skip to content

ESQL: Prevents mixing TS and STANDARD indexes with TS and views - #153366

Merged
ncordon merged 17 commits into
elastic:mainfrom
ncordon:ts-views-verifier-bug
Jul 24, 2026
Merged

ncordon merged 17 commits into
elastic:mainfrom
ncordon:ts-views-verifier-bug

Conversation

@ncordon

@ncordon ncordon commented Jul 9, 2026 •

Copy link
Copy Markdown
Member

Closes #149619
Closes #153030
Closes #145445

When mixing a time series index with a view that uses a standard index in the same query, plus an aggregation that specifically needs a time series index, we currently fail the query.

Example

country_addresses is a view defined as FROM addresses | STATS count=COUNT() BY country. k8s is a timeseries index.

Then this query:

TS k8s, country_addresses
| STATS bytes = last_over_time(event) BY time_bucket = bucket(@timestamp, 1 day)
| SORT time_bucket
;

would fail at planning time with a missing reference similar to this:

java.lang.IllegalStateException: Found 1 problem
line 2:3: Plan [TimeSeriesAggregate[[_tsid{m}#4340023, @timestamp{r}#4339682],
[LASTOVERTIME(k8s.node.name{f}#4339798,true[BOOLEAN],PT0S[TIME_DURATION],@timestamp{f}#4339825)
AS LASTOVERTIME_$1#4340024, @timestamp{r}#4339682 AS @timestamp#4339682],
BUCKET(@timestamp{f}#4339825,P1D[DATE_PERIOD]),BUCKET(@timestamp{f}#4339825,P1D[DATE_PERIOD]),
@timestamp{f}#4339825,TS_COMMAND]] optimized incorrectly due to missing references [@timestamp{f}#4339825]
    at org.elasticsearch.xpack.esql.optimizer.PostOptimizationPhasePlanVerifier.verify(PostOptimizationPhasePlanVerifier.java:53)
    at org.elasticsearch.xpack.esql.optimizer.LogicalPlanOptimizer.optimize(LogicalPlanOptimizer.java:140)
    at org.elasticsearch.xpack.esql.session.EsqlSession.optimizedPlan(EsqlSession.java:2040)

Solution

There are a couple of possible solutions:

  • Either prevent a view to be used inside a TS command. This could break existing queries, but it'd be consistent with what we do with subqueries. i.e. TS index1, (FROM index2) fails with Subqueries are not supported in TS command
  • Or check whether we are mixing time series indices with normal ones in TS, which is what we did in this PR.

TODO

Also there's a degenerate case we probably want to think how to solve. Doing TS (FROM index) would get substituted at planning time as FROM index because if there's only one subquery in a source command, that's what we replace it by at planning time

@ncordon ncordon added >bug >test-mute Use for PR that only mute tests Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL v9.6.0 labels Jul 9, 2026
@ncordon
ncordon force-pushed the ts-views-verifier-bug branch from b43f767 to 7e2dfb8 Compare July 13, 2026 11:29
@ncordon ncordon removed the >test-mute Use for PR that only mute tests label Jul 13, 2026
@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Hi @ncordon, I've created a changelog YAML for you.

@ncordon ncordon added auto-backport Automatically create backport pull requests when merged v9.4.0 v9.5.1 labels Jul 13, 2026
@github-actions

github-actions Bot commented Jul 13, 2026 •

Copy link
Copy Markdown
Contributor

🔍 Preview links for changed docs

⏳ Building and deploying preview... View progress

This comment will be updated with preview links when the build is complete.

@github-actions

Copy link
Copy Markdown
Contributor

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

@ncordon
ncordon marked this pull request as ready for review July 13, 2026 15:07
@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes an ES|QL planning/verification failure (and potential late execution-time failures) caused by using the TS command over sources that resolve to a mix of time-series and non-time-series indices (including via views). It adds analysis-time verification so mixed index modes are rejected with a clear error before time-series metadata injection/optimization occurs.

Changes:

  • Add TimeSeriesAggregate verification that all upstream EsRelation sources are backed exclusively by IndexMode.TIME_SERIES indices, producing a targeted VerificationException message otherwise.
  • Add EsRelation#isFullyTimeSeries() to distinguish the query’s command label (indexMode()) from the actual resolved concrete index modes (indexNameWithModes()).
  • Update unit tests and golden-test index-resolution setup to correctly model index modes (reading dataset index.mode from settings where available) and assert the new rejection behavior.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.

Show a summary per file
File Description
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/TimeSeriesAggregate.java Adds verification to reject TS aggregations when any feeding relation matches non-time-series indices; includes traversal mirroring TS metadata injection scope.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/EsRelation.java Adds isFullyTimeSeries() based on resolved per-index modes.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java Introduces a capability constant documenting the new TS mixed-index-mode rejection behavior.
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerTests.java Adds coverage for mixed-mode TS wildcard rejection and a uniform-time-series success boundary case; adjusts field-caps fixture to isolate TS role conflicts.
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java Adds a regression test ensuring mixed-mode wildcard TS patterns fail early with the new message.
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/GoldenTestCase.java Updates golden-test index mode resolution to read each dataset’s actual index.mode from its settings.
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/view/InMemoryViewServiceTests.java Adds a boundary test confirming FROM can reference a TS-bodied view without being affected by TS-only constraints.
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/VerifierTests.java Ensures the tsdb test analyzer uses IndexMode.TIME_SERIES for TS-related verification.
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/promql/PromqlVerifierTests.java Ensures PromQL verification uses a TIME_SERIES-mode index setup.
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerUnmappedTests.java Adjusts PromQL/unmapped-fields tests to use TIME_SERIES index mode for tsdb fixtures.
docs/changelog/153366.yaml Adds changelog entry for the bug fix.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Hi @ncordon, I've updated the changelog YAML for you.

@ncordon ncordon added the >test-mute Use for PR that only mute tests label Jul 14, 2026
@ncordon
ncordon marked this pull request as draft July 14, 2026 16:53
@ncordon
ncordon force-pushed the ts-views-verifier-bug branch from c3e0be3 to 0ee055f Compare July 21, 2026 22:44
@ncordon ncordon removed the >test-mute Use for PR that only mute tests label Jul 21, 2026
@ncordon
ncordon marked this pull request as ready for review July 21, 2026 22:45
@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Hi @ncordon, I've created a changelog YAML for you.

@fang-xing-esql fang-xing-esql left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this issue @ncordon ! I think it is a good idea not allowing view used under TS command for now.

I added some comments to the change to LogicalPlanBuilder.

assertEquals("sample_data", castSampleRelation.indexPattern());
}

// Regression tests for the bug where a TS relation nested inside a FROM subquery caused the outer

@fang-xing-esql fang-xing-esql Jul 21, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we convert these positive tests to golden tests (AnalyzerSubqueryGoldenTests)?

And if we think it makes sense to throw a VerificationException if UnionAll/Subquery presents below a TimeSeriesAggregate, it will be great to add some negative tests in AnalyzerSubqueryTests, the new query patterns have STATS with TimeSeriesAggregateFunction in the main query after a mixed of TS, FROM and ROW subqueries.

I we decide to demote TimeSeriesAggregate to a regular Aggregate in LogicalPlanBuilder, then it will be great to have some positive tests in subquery_with_ts_source.csv-spec, they have STATS with TimeSeriesAggregateFunction in the main query after a mixed of TS, FROM and ROW subqueries.

@ncordon ncordon Jul 22, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we convert these positive tests to golden tests (AnalyzerSubqueryGoldenTests)?

Yeah agreed, I thought golden tests didn't have support for views (because views run in several phases), but they do for the analyzer phase 👍

The reason I ended up demoting TimeSeriesAggregate to Aggregate by the way is that just forbidding to use views inside TS wouldn't have fixed all cases we discussed, for example mixing normal indices with time series ones in FROM subqueries:

FROM (TS k8s), (FROM employees)
| STATS bytes = last_over_time(event) BY time_bucket = bucket(@timestamp, 1 day)
| SORT time_bucket

* {@code TS}, so time-series aggregate planning must not be triggered by a relation that is
* isolated inside an independent subquery.
*/
private static boolean hasOuterTimeSeries(LogicalPlan plan) {

@fang-xing-esql fang-xing-esql Jul 22, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we return false(earlier) when seeing a UnionAll as well?

        if (plan instanceof UnionAll || plan instanceof Subquery) {
            return false;
        }

I did some more experiments, these three queries below(I added them in AnalyzerSubqueryTests for experiment purposes), they throw VerificationException.

Q1
public void testTimeSeriesAggregateFunctionOutsideTsCommand() {
        analyzer().addK8s().error("""
            FROM k8s
            | STATS x = last_over_time(event) BY time_bucket = bucket(@timestamp, 1 day)
            """, equalTo("""
            Found 1 problem
            line 2:13: time_series aggregate[last_over_time(event)] can only be used with the TS command"""));
}

Q2
public void testRateAggregateFunctionOutsideTsCommand() {
        analyzer().addK8s().error("""
            FROM k8s
            | STATS max_cost = max(rate(network.total_cost)) BY cluster
            """, equalTo("""
            Found 1 problem
            line 2:24: time_series aggregate[rate(network.total_cost)] can only be used with the TS command"""));
}

Q3
public void testTimeSeriesAggregateFunctionOutsideTsCommandAfterSubquery() {
        analyzer().addK8s().error("""
            FROM (FROM k8s), (FROM k8s)
            | STATS x = last_over_time(event) BY time_bucket = bucket(@timestamp, 1 day)
            """, equalTo("""
            Found 1 problem
            line 2:13: time_series aggregate[last_over_time(event)] can only be used with the TS command"""));
}

And the VerificationException comes from Aggregate.checkTimeSeriesAggregates

    protected void checkTimeSeriesAggregates(Failures failures) {
        Holder<Boolean> isTimeSeriesIndexMode = new Holder<>(false);
        child().forEachDown(p -> { ===> it does not check UnionAll
            if (p instanceof EsRelation er && er.indexMode().isTsdb()) {
                isTimeSeriesIndexMode.set(true);
            }
        });
        if (isTimeSeriesIndexMode.get()) {
            return;
        }
        forEachExpression(
            TimeSeriesAggregateFunction.class,
            r -> failures.add(fail(r, "time_series aggregate[{}] can only be used with the TS command", r.sourceText()))
        );
    }

However queries like below completes successfully

Q4
from (TS k8s), (FROM sample_data)
| STATS x = last_over_time(event) BY time_bucket = bucket(@timestamp, 1 day)

Their(Q1/2/3) behaviors make me think we may consider throwing a VerificationException when trying to apply TimeSeriesAggregateFunction in a regular Aggregate when its under a FROM context and its children have UnionAll/Subquery instead, like Q4. If we want to make subquery(like Q4 or in general) work in similar way as Q1/2/3, there are a couple of options I can think of, as parser does not usually throw VerificationException, we might have to do it at a later phase, like in Analyzer.

Option 1:
Keep the change to LogicalPlanBuilder, and make Aggregate.checkTimeSeriesAggregates throw VerificationException when finding UnionAll in its children(skip the right hand side of an AbstractSubqueryJoin), it should only look at the LHS of AbstractSubqueryJoin when looking for TimeSeries indices as well.

Tentative change(prototyped by LLM) to Aggregate:

    protected void checkTimeSeriesAggregates(Failures failures) {
        if (hasTimeSeriesSource(child())) {
            return;
        }
        Holder<Boolean> hasTimeSeriesAgg = new Holder<>(false);
        forEachExpression(TimeSeriesAggregateFunction.class, r -> hasTimeSeriesAgg.set(true));
        if (hasTimeSeriesAgg.get() == false) {
            return;
        }
        if (child().anyMatch(p -> p instanceof UnionAll)) {
            failures.add(
                fail(
                    this,
                    "time-series aggregation [{}] cannot be applied over a union of data sources; "
                        + "apply the time-series aggregation inside each subquery instead",
                    sourceText()
                )
            );
        } else {
            forEachExpression(
                TimeSeriesAggregateFunction.class,
                r -> failures.add(fail(r, "time_series aggregate[{}] can only be used with the TS command", r.sourceText()))
            );
        }
    }

    /**
     * Returns {@code true} if {@code plan} (or any non-{@link UnionAll} descendant, excluding the right-hand side
     * of an {@link AbstractSubqueryJoin}) holds an {@link EsRelation} in time-series index mode.
     * <p>
     * Traversal stops at {@link UnionAll} boundaries so that a {@code TS} source nested inside a {@code FROM}
     * subquery (e.g. {@code FROM (TS k8s), (FROM sample_data)}) does not allow time-series aggregate functions
     * in the outer {@code STATS}: the {@code TS} relation is isolated inside an independent subquery and does
     * not grant TS semantics to this aggregate.
     */
    private static boolean hasTimeSeriesSource(LogicalPlan plan) {
        if (plan instanceof EsRelation er && er.indexMode().isTsdb()) {
            return true;
        }
        if (plan instanceof UnionAll) {
            return false;
        }
        if (plan instanceof AbstractSubqueryJoin join) {
            // Only the left (main data) side can supply a time-series source; right is a lookup/subquery.
            return hasTimeSeriesSource(join.left());
        }
        for (LogicalPlan child : plan.children()) {
            if (hasTimeSeriesSource(child)) {
                return true;
            }
        }
        return false;
    }

Option 2:

An alternative way is leaving LogicalPlanBuilder unchanged, and moving this validation into TimeSeriesAggregate.verify.

Tentative change(prototyped by LLM) to TimeSeriesAggregate:

    /**
     * Returns true if {@code plan} or any node reachable via the main data pipeline contains a {@link UnionAll}.
     * The right side of an {@link AbstractSubqueryJoin} (the independent subquery) is excluded because a
     * {@link UnionAll} there only affects the join lookup, not the time-series data stream.
     */
    private static boolean hasUnionAllInDataPath(LogicalPlan plan) {
        if (plan instanceof UnionAll) {
            return true;
        }
        if (plan instanceof AbstractSubqueryJoin join) {
            return hasUnionAllInDataPath(join.left());
        }
        for (LogicalPlan child : plan.children()) {
            if (hasUnionAllInDataPath(child)) {
                return true;
            }
        }
        return false;
    }

    public void verify(Failures failures) {
        if (hasUnionAllInDataPath(child())) {
            failures.add(
                fail(
                    this,
                    "time-series aggregation [{}] cannot be applied over a union of data sources; "
                        + "apply the time-series aggregation inside each subquery instead",
                    sourceText()
                )
            );
        }
    ...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am fine with throwing a VerificationException.

There's a case I am torn about though:

FROM (TS k8s)
| STATS bytes = last_over_time(event) BY time_bucket = bucket(@timestamp, 1 day)
| SORT time_bucket

What do we do about this? Right now the rules is that since this is a single subquery we would promote that subquery into the main one at parsing time and this would end up:

TS k8s
| STATS bytes = last_over_time(event) BY time_bucket = bucket(@timestamp, 1 day)
| SORT time_bucket

If we remove the information this was a subquery right at the parser, there's no way we can throw for it afterwards (or even demote it) 😿

@fang-xing-esql fang-xing-esql Jul 22, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's a good catch. if there is only one subquery, it will be promoted to the main query by LogicalPlanBuilder, FROM (TS k8s) is transformed to TS k8s, before the subsequent processing commands in the main query get parsed.

We can leave this case(FROM command only has one subquery, no other index pattern, and there is TimeSeriesAggregateFunction in the main query) unchanged I think, and have a golden test for it in AnalyzerSubqueryGoldenTests to capture its current behavior, if there isn't one yet.

VerificationException is thrown when there are multiple subqueries or mixed subqueries with index patterns, when a UnionAll is seen(be careful of AbstractSubqueryJoin, we should check its left child only) below a TimeSeriesAggregateFunction, which is where to find a _tsid is unclear.

@ncordon
ncordon force-pushed the ts-views-verifier-bug branch from 0fd8bc6 to f6ae679 Compare July 23, 2026 08:51
@ncordon
ncordon force-pushed the ts-views-verifier-bug branch from f6ae679 to 0ebd1e5 Compare July 23, 2026 08:57
@ncordon
ncordon requested a review from fang-xing-esql July 23, 2026 11:14
@ncordon

ncordon commented Jul 23, 2026

Copy link
Copy Markdown
Member Author

I think it should be ready for re-review now @fang-xing-esql

@fang-xing-esql fang-xing-esql left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for fixing this @ncordon !

@ncordon
ncordon merged commit e89d1b0 into elastic:main Jul 24, 2026
43 checks passed
@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

💔 Backport failed

Status Branch Result
❌ 9.4 Commit could not be cherrypicked due to conflicts
❌ 9.5 Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 153366

@ncordon

ncordon commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

💚 All backports created successfully

Status Branch Result
✅ 9.5
✅ 9.4

Questions ?

Please refer to the Backport tool documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Analytics/ES|QL AKA ESQL auto-backport Automatically create backport pull requests when merged >bug Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v9.4.0 v9.5.1 v9.6.0

Projects

None yet

4 participants