ESQL: Fix Driver creating status with a live list of operators by ivancea · Pull Request #132260 · elastic/elasticsearch

ivancea · 2025-07-31T10:57:08Z

DriverStatus is an immutable record created by the Driver. However, its components are not inherently immutable.

This PR fixes a live collection used by the Driver, that was being directly put into the status, leading to ConcurrentModificationExceptions when reading it (Through the Task list API, for example).

Some example errors:

java.util.ConcurrentModificationException
	at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1096)
	at java.base/java.util.ArrayList$Itr.next(ArrayList.java:1050)
	at org.elasticsearch.compute.operator.DriverStatus.toXContent(DriverStatus.java:134)
	at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:993)
	at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.field(XContentBuilder.java:978)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.tasks.TaskInfo.toXContent(TaskInfo.java:113)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:63)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:67)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.ListTasksResponse.lambda$groupedByParent$10(ListTasksResponse.java:183)

And:

java.util.ConcurrentModificationException
	at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1096)
	at java.base/java.util.ArrayList$Itr.next(ArrayList.java:1050)
	at org.elasticsearch.compute.operator.DriverStatus.documentsFound(DriverStatus.java:157)
	at org.elasticsearch.compute.operator.DriverStatus.toXContent(DriverStatus.java:129)
	at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:993)
	at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.field(XContentBuilder.java:978)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.tasks.TaskInfo.toXContent(TaskInfo.java:113)
	at org.elasticsearch.server@9.2.0org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:63)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:67)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.ListTasksResponse.lambda$groupedByParent$10(ListTasksResponse.java:183)

Also, this looks like the source of this issue, with another similar case:
Fixes #131564

elasticsearchmachine · 2025-07-31T10:57:34Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-07-31T10:57:34Z

Hi @ivancea, I've created a changelog YAML for you.

nik9000 · 2025-07-31T14:35:49Z

                prev.iterations() + extraIterations,
                status,
-                statusOfCompletedOperators,
+                new ArrayList<>(statusOfCompletedOperators),


I'd probably do List.copyOf here, but this is fine.

I think it's worth a comment that we always update the statusOfCompletedOperators list in this thread, but we read it in many threads.

Added the suggestion, and a test.

About the statusOfCompletedOperators, after this change, it's never read outside of the updateStatus()

elasticsearchmachine · 2025-08-01T14:36:18Z

💔 Backport failed

Status	Branch	Result
✅	9.1
❌	8.19	Commit could not be cherrypicked due to conflicts
❌	9.0	Commit could not be cherrypicked due to conflicts
❌	8.17	Commit could not be cherrypicked due to conflicts
❌	8.18	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 132260

…ic#132260) `DriverStatus` is an immutable record created by the Driver. However, its components are not inherently immutable. This PR fixes a live collection used by the Driver, that was being directly put into the status, leading to `ConcurrentModificationException`s when reading it (Through the Task list API, for example). Some example errors: ``` java.util.ConcurrentModificationException at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1096) at java.base/java.util.ArrayList$Itr.next(ArrayList.java:1050) at org.elasticsearch.compute.operator.DriverStatus.toXContent(DriverStatus.java:134) at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:993) at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.field(XContentBuilder.java:978) at org.elasticsearch.server@9.2.0/org.elasticsearch.tasks.TaskInfo.toXContent(TaskInfo.java:113) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:63) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:67) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.ListTasksResponse.lambda$groupedByParent$10(ListTasksResponse.java:183) ``` And: ``` java.util.ConcurrentModificationException at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1096) at java.base/java.util.ArrayList$Itr.next(ArrayList.java:1050) at org.elasticsearch.compute.operator.DriverStatus.documentsFound(DriverStatus.java:157) at org.elasticsearch.compute.operator.DriverStatus.toXContent(DriverStatus.java:129) at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:993) at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.field(XContentBuilder.java:978) at org.elasticsearch.server@9.2.0/org.elasticsearch.tasks.TaskInfo.toXContent(TaskInfo.java:113) at org.elasticsearch.server@9.2.0org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:63) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:67) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.ListTasksResponse.lambda$groupedByParent$10(ListTasksResponse.java:183) ``` Also, this looks like the source of this issue, with another similar case: Fixes elastic#131564

nik9000 · 2025-08-01T14:38:52Z

Status Branch Result
✅ 9.1
❌ 8.19 Commit could not be cherrypicked due to conflicts
❌ 9.0 Commit could not be cherrypicked due to conflicts
❌ 8.17 Commit could not be cherrypicked due to conflicts
❌ 8.18 Commit could not be cherrypicked due to conflicts

Enjoy!

…ic#132260) `DriverStatus` is an immutable record created by the Driver. However, its components are not inherently immutable. This PR fixes a live collection used by the Driver, that was being directly put into the status, leading to `ConcurrentModificationException`s when reading it (Through the Task list API, for example). Some example errors: ``` java.util.ConcurrentModificationException at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1096) at java.base/java.util.ArrayList$Itr.next(ArrayList.java:1050) at org.elasticsearch.compute.operator.DriverStatus.toXContent(DriverStatus.java:134) at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:993) at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.field(XContentBuilder.java:978) at org.elasticsearch.server@9.2.0/org.elasticsearch.tasks.TaskInfo.toXContent(TaskInfo.java:113) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:63) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:67) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.ListTasksResponse.lambda$groupedByParent$10(ListTasksResponse.java:183) ``` And: ``` java.util.ConcurrentModificationException at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1096) at java.base/java.util.ArrayList$Itr.next(ArrayList.java:1050) at org.elasticsearch.compute.operator.DriverStatus.documentsFound(DriverStatus.java:157) at org.elasticsearch.compute.operator.DriverStatus.toXContent(DriverStatus.java:129) at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:993) at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.field(XContentBuilder.java:978) at org.elasticsearch.server@9.2.0/org.elasticsearch.tasks.TaskInfo.toXContent(TaskInfo.java:113) at org.elasticsearch.server@9.2.0org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:63) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:67) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.ListTasksResponse.lambda$groupedByParent$10(ListTasksResponse.java:183) ``` Also, this looks like the source of this issue, with another similar case: Fixes elastic#131564

…) (#132333) `DriverStatus` is an immutable record created by the Driver. However, its components are not inherently immutable. This PR fixes a live collection used by the Driver, that was being directly put into the status, leading to `ConcurrentModificationException`s when reading it (Through the Task list API, for example). Some example errors: ``` java.util.ConcurrentModificationException at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1096) at java.base/java.util.ArrayList$Itr.next(ArrayList.java:1050) at org.elasticsearch.compute.operator.DriverStatus.toXContent(DriverStatus.java:134) at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:993) at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.field(XContentBuilder.java:978) at org.elasticsearch.server@9.2.0/org.elasticsearch.tasks.TaskInfo.toXContent(TaskInfo.java:113) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:63) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:67) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.ListTasksResponse.lambda$groupedByParent$10(ListTasksResponse.java:183) ``` And: ``` java.util.ConcurrentModificationException at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1096) at java.base/java.util.ArrayList$Itr.next(ArrayList.java:1050) at org.elasticsearch.compute.operator.DriverStatus.documentsFound(DriverStatus.java:157) at org.elasticsearch.compute.operator.DriverStatus.toXContent(DriverStatus.java:129) at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:993) at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.field(XContentBuilder.java:978) at org.elasticsearch.server@9.2.0/org.elasticsearch.tasks.TaskInfo.toXContent(TaskInfo.java:113) at org.elasticsearch.server@9.2.0org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:63) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:67) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.ListTasksResponse.lambda$groupedByParent$10(ListTasksResponse.java:183) ``` Also, this looks like the source of this issue, with another similar case: Fixes #131564

…132260) (#132341) Manual 9.0 backport of #132260

…) (#132339) Manual 8.18 backport of #132260

…) (#132338) Manual 8.17 backport of #132260

…) (#132340) Manual 8.19 backport of #132260

…60) (#132340) Manual 8.19 backport of elastic/elasticsearch#132260

ESQL: FIx Driver creating status with a live list of operators

fbf7606

ivancea requested review from idegtiarenko, nik9000 and smalyshev July 31, 2025 10:57

ivancea added >bug Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) auto-backport Automatically create backport pull requests when merged :Analytics/ES|QL AKA ESQL v9.2.0 v9.1.1 v8.19.1 v9.0.5 v8.17.10 v8.18.5 labels Jul 31, 2025

Update docs/changelog/132260.yaml

9a464cc

smalyshev approved these changes Jul 31, 2025

View reviewed changes

nik9000 changed the title ~~ESQL: FIx Driver creating status with a live list of operators~~ ESQL: Fix Driver creating status with a live list of operators Jul 31, 2025

nik9000 approved these changes Jul 31, 2025

View reviewed changes

ivancea and others added 2 commits July 31, 2025 17:05

Use List.copyOf() and add test

961164c

Merge branch 'main' into fix-esql-driver-status-concurrency

8cb50f8

ivancea enabled auto-merge (squash) July 31, 2025 15:08

[CI] Auto commit changes from spotless

0110c99

nik9000 approved these changes Jul 31, 2025

View reviewed changes

ivancea added 2 commits July 31, 2025 19:14

Merge branch 'main' into fix-esql-driver-status-concurrency

7f2a769

Merge branch 'main' into fix-esql-driver-status-concurrency

a5153d2

ivancea merged commit 71eed0a into elastic:main Aug 1, 2025

ivancea deleted the fix-esql-driver-status-concurrency branch August 1, 2025 14:34

ivancea mentioned this pull request Aug 1, 2025

[9.1] ESQL: Fix Driver creating status with a live list of operators (#132260) #132333

Merged

elasticsearchmachine added the backport pending label Aug 1, 2025

ivancea added a commit that referenced this pull request Aug 4, 2025

[9.0] ESQL: Fix Driver creating status with a live list of operators (#…

4844f06

…132260) (#132341) Manual 9.0 backport of #132260

ivancea added a commit that referenced this pull request Aug 4, 2025

ESQL: Fix Driver creating status with a live list of operators (#132260…

57ff7f7

…) (#132339) Manual 8.18 backport of #132260

ivancea added a commit that referenced this pull request Aug 4, 2025

ESQL: Fix Driver creating status with a live list of operators (#132260…

89b4ddd

…) (#132338) Manual 8.17 backport of #132260

ivancea added a commit that referenced this pull request Aug 4, 2025

ESQL: Fix Driver creating status with a live list of operators (#132260…

261da78

…) (#132340) Manual 8.19 backport of #132260

ivancea removed the backport pending label Aug 4, 2025

verawu pushed a commit to verawu/elasticsearch that referenced this pull request Jul 21, 2026

ESQL: Fix Driver creating status with a live list of operators (#1322…

cae9f09

…60) (#132340) Manual 8.19 backport of elastic/elasticsearch#132260

Navigation Menu

Repository navigation

ESQL: Fix Driver creating status with a live list of operators - #132260

ESQL: Fix Driver creating status with a live list of operators#132260
ivancea merged 7 commits into
elastic:mainfrom
ivancea:fix-esql-driver-status-concurrency

ivancea commented Jul 31, 2025

Uh oh!

elasticsearchmachine commented Jul 31, 2025

Uh oh!

elasticsearchmachine commented Jul 31, 2025

Uh oh!

nik9000 Jul 31, 2025

Uh oh!

ivancea Jul 31, 2025

Uh oh!

elasticsearchmachine commented Aug 1, 2025

Uh oh!

nik9000 commented Aug 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ivancea commented Jul 31, 2025

Uh oh!

elasticsearchmachine commented Jul 31, 2025

Uh oh!

elasticsearchmachine commented Jul 31, 2025

Uh oh!

nik9000 Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

ivancea Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Aug 1, 2025

💔 Backport failed

Uh oh!

nik9000 commented Aug 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants