[Transform] Honor cluster health wait timeout when ensuring system index shards are active by rnb-tron · Pull Request #149462 · elastic/elasticsearch

rnb-tron · 2026-05-20T08:26:29Z

Summary

Fix TransformInternalIndex#waitForLatestVersionedIndexShardsActive to
treat a ClusterHealth timeout as a failure instead of silently
completing the listener with success.

Closes #149400

Background

When a transform is created via PUT _transform/{id}, the action calls
TransformInternalIndex#createLatestVersionedIndexIfRequired. If the
versioned internal index already exists but its shards are not yet
active, the code falls back to waitForLatestVersionedIndexShardsActive,
which issues a ClusterHealth request that waits up to 30 seconds for
at least one active shard.

The previous implementation handled the response like this:

ActionListener<ClusterHealthResponse> innerListener = ActionListener.wrap(
    r -> listener.onResponse(null),   // always success!
    listener::onFailure
);

Any non-exceptional response — including timeouts where
ClusterHealthResponse#isTimedOut() returns true — was forwarded
straight to onResponse(null). As a result the caller assumed the
system index was ready and proceeded to write the transform config,
which would later surface as confusing downstream failures.

What changed

In TransformInternalIndex#waitForLatestVersionedIndexShardsActive:

Inspect response.isTimedOut() before completing the listener.
On timeout, fail the listener with
ElasticsearchStatusException(SERVICE_UNAVAILABLE) carrying a
dedicated message constant so the REST layer returns an actionable
503 and the client can retry.
Refactor to the project's idiomatic listener.delegateFailureAndWrap
style (consistent with IndexBasedTransformConfigManager and many
other call sites in the module).

In TransformMessages:

Add a focused constant
TRANSFORM_WAIT_FOR_INDEX_SHARDS_ACTIVE_TIMEOUT = "Timed out waiting for transform system index shards to become active".

Tests

Added TransformInternalIndexTests#testCreateLatestVersionedIndexIfRequired_GivenClusterHealthTimeout
which:

Sets up a cluster state where the versioned index exists but shards
are not active, forcing the code into the waitFor… branch.
Stubs the cluster admin client to return a ClusterHealthResponse
with setTimedOut(true).
Asserts that the listener is failed with
ElasticsearchStatusException, that
status() == RestStatus.SERVICE_UNAVAILABLE, and that the message
matches TransformMessages.TRANSFORM_WAIT_FOR_INDEX_SHARDS_ACTIVE_TIMEOUT.
Verifies the cluster health client is invoked exactly once.

./gradlew :x-pack:plugin:transform:test \
  --tests "org.elasticsearch.xpack.transform.persistence.TransformInternalIndexTests"

Behavior change

Scenario	Before	After
Shards become active within 30s	Success	Success (unchanged)
Inner request fails	`onFailure` propagated	`onFailure` propagated (unchanged)
Cluster health request times out	Silently `onResponse(null)`	`onFailure(503 SERVICE_UNAVAILABLE)`

This is a user-visible behavior change: callers that previously
appeared to succeed (and then failed later in a confusing way) will now
receive a clear 503 they can retry. No transport version or
serialization changes are required.

`waitForLatestVersionedIndexShardsActive` issues a `ClusterHealth` request that waits up to 30s for at least one active shard of the transform internal index. The listener previously dispatched any non-exceptional response straight to `onResponse(null)`, including responses where `isTimedOut() == true`. As a result a PUT transform call could proceed to write its config even when the system index shards were not actually ready, surfacing later as obscure failures. Treat `isTimedOut() == true` as a failure and complete the listener with an `ElasticsearchStatusException(SERVICE_UNAVAILABLE)` so the client gets an actionable 503 and can retry. Closes elastic#149400

elasticsearchmachine · 2026-05-20T14:33:00Z

Pinging @elastic/ml-core (Team:ML)

prwhelan

Looks good - thank you for the contribution

prwhelan · 2026-06-05T17:22:57Z

In queue, will merge Monday June 15th

elasticsearchmachine · 2026-06-15T15:26:31Z

💚 Backport successful

Status	Branch	Result
✅	8.19
✅	9.3
✅	9.4

…dex shards are active (#149462) (#151226) `waitForLatestVersionedIndexShardsActive` issues a `ClusterHealth` request that waits up to 30s for at least one active shard of the transform internal index. The listener previously dispatched any non-exceptional response straight to `onResponse(null)`, including responses where `isTimedOut() == true`. As a result a PUT transform call could proceed to write its config even when the system index shards were not actually ready, surfacing later as obscure failures. Treat `isTimedOut() == true` as a failure and complete the listener with an `ElasticsearchStatusException(SERVICE_UNAVAILABLE)` so the client gets an actionable 503 and can retry. Closes #149400 Co-authored-by: Tron Wang <59386838+rnb-tron@users.noreply.github.com>

…dex shards are active (#149462) (#151228) `waitForLatestVersionedIndexShardsActive` issues a `ClusterHealth` request that waits up to 30s for at least one active shard of the transform internal index. The listener previously dispatched any non-exceptional response straight to `onResponse(null)`, including responses where `isTimedOut() == true`. As a result a PUT transform call could proceed to write its config even when the system index shards were not actually ready, surfacing later as obscure failures. Treat `isTimedOut() == true` as a failure and complete the listener with an `ElasticsearchStatusException(SERVICE_UNAVAILABLE)` so the client gets an actionable 503 and can retry. Closes #149400 Co-authored-by: Tron Wang <59386838+rnb-tron@users.noreply.github.com>

…dex shards are active (#149462) (#151227) `waitForLatestVersionedIndexShardsActive` issues a `ClusterHealth` request that waits up to 30s for at least one active shard of the transform internal index. The listener previously dispatched any non-exceptional response straight to `onResponse(null)`, including responses where `isTimedOut() == true`. As a result a PUT transform call could proceed to write its config even when the system index shards were not actually ready, surfacing later as obscure failures. Treat `isTimedOut() == true` as a failure and complete the listener with an `ElasticsearchStatusException(SERVICE_UNAVAILABLE)` so the client gets an actionable 503 and can retry. Closes #149400 Co-authored-by: Tron Wang <59386838+rnb-tron@users.noreply.github.com>

…dex shards are active (elastic#149462) `waitForLatestVersionedIndexShardsActive` issues a `ClusterHealth` request that waits up to 30s for at least one active shard of the transform internal index. The listener previously dispatched any non-exceptional response straight to `onResponse(null)`, including responses where `isTimedOut() == true`. As a result a PUT transform call could proceed to write its config even when the system index shards were not actually ready, surfacing later as obscure failures. Treat `isTimedOut() == true` as a failure and complete the listener with an `ElasticsearchStatusException(SERVICE_UNAVAILABLE)` so the client gets an actionable 503 and can retry. Closes elastic#149400 Co-authored-by: Pat Whelan <pat.whelan@elastic.co>

elasticsearchmachine added needs:triage Requires assignment of a team area label v9.5.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels May 20, 2026

rnb-tron added 2 commits May 20, 2026 16:28

Rename changelog to match PR number elastic#149462

54b6410

Update pr field in changelog to 149462

df12319

kingherc added the :ml/Transform Transform label May 20, 2026

elasticsearchmachine added Team:ML Meta label for the ML team and removed needs:triage Requires assignment of a team area label labels May 20, 2026

prwhelan self-assigned this May 21, 2026

prwhelan added >bug auto-backport Automatically create backport pull requests when merged branch:8.19 branch:9.3 branch:9.4 labels May 21, 2026

elasticsearchmachine added v8.19.17 v9.3.6 v9.4.3 and removed branch:8.19 branch:9.3 branch:9.4 labels May 27, 2026

prwhelan added 4 commits May 27, 2026 08:32

Merge branch 'main' into fix/149400-cluster-health-timeout

b8af51c

Merge branch 'main' into fix/149400-cluster-health-timeout

33b6a04

Merge branch 'main' into fix/149400-cluster-health-timeout

7b00574

Merge branch 'main' into fix/149400-cluster-health-timeout

9005e41

prwhelan approved these changes Jun 3, 2026

View reviewed changes

prwhelan added 2 commits June 3, 2026 09:25

Merge branch 'main' into fix/149400-cluster-health-timeout

7115487

Merge branch 'main' into fix/149400-cluster-health-timeout

4482088

prwhelan enabled auto-merge (squash) June 4, 2026 19:15

Merge branch 'main' into fix/149400-cluster-health-timeout

f0bd13c

prwhelan merged commit 7591609 into elastic:main Jun 15, 2026
34 checks passed

Navigation Menu

Repository navigation

[Transform] Honor cluster health wait timeout when ensuring system index shards are active - #149462

[Transform] Honor cluster health wait timeout when ensuring system index shards are active#149462
prwhelan merged 10 commits into
elastic:mainfrom
rnb-tron:fix/149400-cluster-health-timeout

rnb-tron commented May 20, 2026

Uh oh!

elasticsearchmachine commented May 20, 2026

Uh oh!

prwhelan left a comment

Uh oh!

prwhelan commented Jun 5, 2026

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

rnb-tron commented May 20, 2026

Summary

Background

What changed

Tests

Behavior change

Uh oh!

elasticsearchmachine commented May 20, 2026

Uh oh!

prwhelan left a comment

Choose a reason for hiding this comment

Uh oh!

prwhelan commented Jun 5, 2026

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 15, 2026

💚 Backport successful

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants