Repository navigation
ESQL: Prevents mixing TS and STANDARD indexes with TS and views - #153366
Conversation
b43f767 to
7e2dfb8
Compare
|
Hi @ncordon, I've created a changelog YAML for you. |
🔍 Preview links for changed docs⏳ Building and deploying preview... View progress This comment will be updated with preview links when the build is complete. |
ℹ️ Important: Docs version tagging👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version. We use applies_to tags to mark version-specific features and changes. Expand for a quick overviewWhen to use applies_to tags:✅ At the page level to indicate which products/deployments the content applies to (mandatory) What NOT to do:❌ Don't remove or replace information that applies to an older version 🤔 Need help?
|
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
There was a problem hiding this comment.
Pull request overview
This PR fixes an ES|QL planning/verification failure (and potential late execution-time failures) caused by using the TS command over sources that resolve to a mix of time-series and non-time-series indices (including via views). It adds analysis-time verification so mixed index modes are rejected with a clear error before time-series metadata injection/optimization occurs.
Changes:
- Add
TimeSeriesAggregateverification that all upstreamEsRelationsources are backed exclusively byIndexMode.TIME_SERIESindices, producing a targetedVerificationExceptionmessage otherwise. - Add
EsRelation#isFullyTimeSeries()to distinguish the query’s command label (indexMode()) from the actual resolved concrete index modes (indexNameWithModes()). - Update unit tests and golden-test index-resolution setup to correctly model index modes (reading dataset
index.modefrom settings where available) and assert the new rejection behavior.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/TimeSeriesAggregate.java | Adds verification to reject TS aggregations when any feeding relation matches non-time-series indices; includes traversal mirroring TS metadata injection scope. |
| x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/EsRelation.java | Adds isFullyTimeSeries() based on resolved per-index modes. |
| x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java | Introduces a capability constant documenting the new TS mixed-index-mode rejection behavior. |
| x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerTests.java | Adds coverage for mixed-mode TS wildcard rejection and a uniform-time-series success boundary case; adjusts field-caps fixture to isolate TS role conflicts. |
| x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java | Adds a regression test ensuring mixed-mode wildcard TS patterns fail early with the new message. |
| x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/GoldenTestCase.java | Updates golden-test index mode resolution to read each dataset’s actual index.mode from its settings. |
| x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/view/InMemoryViewServiceTests.java | Adds a boundary test confirming FROM can reference a TS-bodied view without being affected by TS-only constraints. |
| x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/VerifierTests.java | Ensures the tsdb test analyzer uses IndexMode.TIME_SERIES for TS-related verification. |
| x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/promql/PromqlVerifierTests.java | Ensures PromQL verification uses a TIME_SERIES-mode index setup. |
| x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerUnmappedTests.java | Adjusts PromQL/unmapped-fields tests to use TIME_SERIES index mode for tsdb fixtures. |
| docs/changelog/153366.yaml | Adds changelog entry for the bug fix. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Hi @ncordon, I've updated the changelog YAML for you. |
c3e0be3 to
0ee055f
Compare
|
Hi @ncordon, I've created a changelog YAML for you. |
fang-xing-esql
left a comment
There was a problem hiding this comment.
Thanks for fixing this issue @ncordon ! I think it is a good idea not allowing view used under TS command for now.
I added some comments to the change to LogicalPlanBuilder.
| assertEquals("sample_data", castSampleRelation.indexPattern()); | ||
| } | ||
|
|
||
| // Regression tests for the bug where a TS relation nested inside a FROM subquery caused the outer |
There was a problem hiding this comment.
Shall we convert these positive tests to golden tests (AnalyzerSubqueryGoldenTests)?
And if we think it makes sense to throw a VerificationException if UnionAll/Subquery presents below a TimeSeriesAggregate, it will be great to add some negative tests in AnalyzerSubqueryTests, the new query patterns have STATS with TimeSeriesAggregateFunction in the main query after a mixed of TS, FROM and ROW subqueries.
I we decide to demote TimeSeriesAggregate to a regular Aggregate in LogicalPlanBuilder, then it will be great to have some positive tests in subquery_with_ts_source.csv-spec, they have STATS with TimeSeriesAggregateFunction in the main query after a mixed of TS, FROM and ROW subqueries.
There was a problem hiding this comment.
Shall we convert these positive tests to golden tests (AnalyzerSubqueryGoldenTests)?
Yeah agreed, I thought golden tests didn't have support for views (because views run in several phases), but they do for the analyzer phase 👍
The reason I ended up demoting TimeSeriesAggregate to Aggregate by the way is that just forbidding to use views inside TS wouldn't have fixed all cases we discussed, for example mixing normal indices with time series ones in FROM subqueries:
FROM (TS k8s), (FROM employees)
| STATS bytes = last_over_time(event) BY time_bucket = bucket(@timestamp, 1 day)
| SORT time_bucket
| * {@code TS}, so time-series aggregate planning must not be triggered by a relation that is | ||
| * isolated inside an independent subquery. | ||
| */ | ||
| private static boolean hasOuterTimeSeries(LogicalPlan plan) { |
There was a problem hiding this comment.
Shall we return false(earlier) when seeing a UnionAll as well?
if (plan instanceof UnionAll || plan instanceof Subquery) {
return false;
}
I did some more experiments, these three queries below(I added them in AnalyzerSubqueryTests for experiment purposes), they throw VerificationException.
Q1
public void testTimeSeriesAggregateFunctionOutsideTsCommand() {
analyzer().addK8s().error("""
FROM k8s
| STATS x = last_over_time(event) BY time_bucket = bucket(@timestamp, 1 day)
""", equalTo("""
Found 1 problem
line 2:13: time_series aggregate[last_over_time(event)] can only be used with the TS command"""));
}
Q2
public void testRateAggregateFunctionOutsideTsCommand() {
analyzer().addK8s().error("""
FROM k8s
| STATS max_cost = max(rate(network.total_cost)) BY cluster
""", equalTo("""
Found 1 problem
line 2:24: time_series aggregate[rate(network.total_cost)] can only be used with the TS command"""));
}
Q3
public void testTimeSeriesAggregateFunctionOutsideTsCommandAfterSubquery() {
analyzer().addK8s().error("""
FROM (FROM k8s), (FROM k8s)
| STATS x = last_over_time(event) BY time_bucket = bucket(@timestamp, 1 day)
""", equalTo("""
Found 1 problem
line 2:13: time_series aggregate[last_over_time(event)] can only be used with the TS command"""));
}
And the VerificationException comes from Aggregate.checkTimeSeriesAggregates
protected void checkTimeSeriesAggregates(Failures failures) {
Holder<Boolean> isTimeSeriesIndexMode = new Holder<>(false);
child().forEachDown(p -> { ===> it does not check UnionAll
if (p instanceof EsRelation er && er.indexMode().isTsdb()) {
isTimeSeriesIndexMode.set(true);
}
});
if (isTimeSeriesIndexMode.get()) {
return;
}
forEachExpression(
TimeSeriesAggregateFunction.class,
r -> failures.add(fail(r, "time_series aggregate[{}] can only be used with the TS command", r.sourceText()))
);
}
However queries like below completes successfully
Q4
from (TS k8s), (FROM sample_data)
| STATS x = last_over_time(event) BY time_bucket = bucket(@timestamp, 1 day)
Their(Q1/2/3) behaviors make me think we may consider throwing a VerificationException when trying to apply TimeSeriesAggregateFunction in a regular Aggregate when its under a FROM context and its children have UnionAll/Subquery instead, like Q4. If we want to make subquery(like Q4 or in general) work in similar way as Q1/2/3, there are a couple of options I can think of, as parser does not usually throw VerificationException, we might have to do it at a later phase, like in Analyzer.
Option 1:
Keep the change to LogicalPlanBuilder, and make Aggregate.checkTimeSeriesAggregates throw VerificationException when finding UnionAll in its children(skip the right hand side of an AbstractSubqueryJoin), it should only look at the LHS of AbstractSubqueryJoin when looking for TimeSeries indices as well.
Tentative change(prototyped by LLM) to Aggregate:
protected void checkTimeSeriesAggregates(Failures failures) {
if (hasTimeSeriesSource(child())) {
return;
}
Holder<Boolean> hasTimeSeriesAgg = new Holder<>(false);
forEachExpression(TimeSeriesAggregateFunction.class, r -> hasTimeSeriesAgg.set(true));
if (hasTimeSeriesAgg.get() == false) {
return;
}
if (child().anyMatch(p -> p instanceof UnionAll)) {
failures.add(
fail(
this,
"time-series aggregation [{}] cannot be applied over a union of data sources; "
+ "apply the time-series aggregation inside each subquery instead",
sourceText()
)
);
} else {
forEachExpression(
TimeSeriesAggregateFunction.class,
r -> failures.add(fail(r, "time_series aggregate[{}] can only be used with the TS command", r.sourceText()))
);
}
}
/**
* Returns {@code true} if {@code plan} (or any non-{@link UnionAll} descendant, excluding the right-hand side
* of an {@link AbstractSubqueryJoin}) holds an {@link EsRelation} in time-series index mode.
* <p>
* Traversal stops at {@link UnionAll} boundaries so that a {@code TS} source nested inside a {@code FROM}
* subquery (e.g. {@code FROM (TS k8s), (FROM sample_data)}) does not allow time-series aggregate functions
* in the outer {@code STATS}: the {@code TS} relation is isolated inside an independent subquery and does
* not grant TS semantics to this aggregate.
*/
private static boolean hasTimeSeriesSource(LogicalPlan plan) {
if (plan instanceof EsRelation er && er.indexMode().isTsdb()) {
return true;
}
if (plan instanceof UnionAll) {
return false;
}
if (plan instanceof AbstractSubqueryJoin join) {
// Only the left (main data) side can supply a time-series source; right is a lookup/subquery.
return hasTimeSeriesSource(join.left());
}
for (LogicalPlan child : plan.children()) {
if (hasTimeSeriesSource(child)) {
return true;
}
}
return false;
}
Option 2:
An alternative way is leaving LogicalPlanBuilder unchanged, and moving this validation into TimeSeriesAggregate.verify.
Tentative change(prototyped by LLM) to TimeSeriesAggregate:
/**
* Returns true if {@code plan} or any node reachable via the main data pipeline contains a {@link UnionAll}.
* The right side of an {@link AbstractSubqueryJoin} (the independent subquery) is excluded because a
* {@link UnionAll} there only affects the join lookup, not the time-series data stream.
*/
private static boolean hasUnionAllInDataPath(LogicalPlan plan) {
if (plan instanceof UnionAll) {
return true;
}
if (plan instanceof AbstractSubqueryJoin join) {
return hasUnionAllInDataPath(join.left());
}
for (LogicalPlan child : plan.children()) {
if (hasUnionAllInDataPath(child)) {
return true;
}
}
return false;
}
public void verify(Failures failures) {
if (hasUnionAllInDataPath(child())) {
failures.add(
fail(
this,
"time-series aggregation [{}] cannot be applied over a union of data sources; "
+ "apply the time-series aggregation inside each subquery instead",
sourceText()
)
);
}
...
There was a problem hiding this comment.
I am fine with throwing a VerificationException.
There's a case I am torn about though:
FROM (TS k8s)
| STATS bytes = last_over_time(event) BY time_bucket = bucket(@timestamp, 1 day)
| SORT time_bucket
What do we do about this? Right now the rules is that since this is a single subquery we would promote that subquery into the main one at parsing time and this would end up:
TS k8s
| STATS bytes = last_over_time(event) BY time_bucket = bucket(@timestamp, 1 day)
| SORT time_bucket
If we remove the information this was a subquery right at the parser, there's no way we can throw for it afterwards (or even demote it) 😿
There was a problem hiding this comment.
Yeah, that's a good catch. if there is only one subquery, it will be promoted to the main query by LogicalPlanBuilder, FROM (TS k8s) is transformed to TS k8s, before the subsequent processing commands in the main query get parsed.
We can leave this case(FROM command only has one subquery, no other index pattern, and there is TimeSeriesAggregateFunction in the main query) unchanged I think, and have a golden test for it in AnalyzerSubqueryGoldenTests to capture its current behavior, if there isn't one yet.
VerificationException is thrown when there are multiple subqueries or mixed subqueries with index patterns, when a UnionAll is seen(be careful of AbstractSubqueryJoin, we should check its left child only) below a TimeSeriesAggregateFunction, which is where to find a _tsid is unclear.
0fd8bc6 to
f6ae679
Compare
f6ae679 to
0ebd1e5
Compare
|
I think it should be ready for re-review now @fang-xing-esql |
fang-xing-esql
left a comment
There was a problem hiding this comment.
LGTM, thanks for fixing this @ncordon !
💔 Backport failed
You can use sqren/backport to manually backport by running |
💚 All backports created successfully
Questions ?Please refer to the Backport tool documentation |
Closes #149619
Closes #153030
Closes #145445
When mixing a time series index with a view that uses a standard index in the same query, plus an aggregation that specifically needs a time series index, we currently fail the query.
Example
country_addressesis a view defined asFROM addresses | STATS count=COUNT() BY country.k8sis a timeseries index.Then this query:
would fail at planning time with a missing reference similar to this:
Solution
There are a couple of possible solutions:
TScommand. This could break existing queries, but it'd be consistent with what we do with subqueries. i.e.TS index1, (FROM index2)fails withSubqueries are not supported in TS commandTS, which is what we did in this PR.TODO
Also there's a degenerate case we probably want to think how to solve. Doing
TS (FROM index)would get substituted at planning time asFROM indexbecause if there's only one subquery in a source command, that's what we replace it by at planning time