[ML] Surface CCS skipped-cluster stats on datafeed extraction failure by edsavage · Pull Request #157567 · elastic/elasticsearch

edsavage · 2026-08-24T23:41:13Z

Summary

When a datafeed search against a remote cluster throws ResourceNotFoundException (one or more
remote clusters were skipped), the extraction-failure code path in DatafeedJob.run() re-throws
before reaching crossClusterSearchStats.update(). As a result skipped_clusters is always 0
in the running datafeed stats API response even when every search round skips remote clusters.

This PR wires cluster-state metadata from the point of failure back to the stats update:

DataExtractor.getLinkedClusterStates() — new default interface method returning List.of(). Each extractor overrides it to expose the cluster states it has observed so far.
ScrollDataExtractor — updates lastLinkedClusterStates inside the checkForSkippedClusters catch block before re-throwing, so state is not lost when the scroll response is released.
AbstractAggregationDataExtractor / CompositeAggregationDataExtractor — the static executeSearchRequest() wraps ResourceNotFoundException in a new package-private SkippedClustersException carrying the extracted cluster states. The instance search() method catches this, updates lastLinkedClusterStates, then re-throws.
ChunkedDataExtractor — catches ResourceNotFoundException from the temporary summary extractor in setUpChunkedSearch(), merges its skipped cluster states into lastLinkedClusterStates before re-throwing, so states are not lost when the summary extractor is discarded. Also added a test for this path in ChunkedDataExtractorTests.
DatafeedJob.run() — calls dataExtractor.getLinkedClusterStates() in the extraction catch block and passes non-empty results to crossClusterSearchStats.update() before re-throwing.

Test plan

Existing CrossClusterSearchStatsTests and DatafeedJobTests continue to pass
Run full ML unit test suite: ./gradlew :x-pack:plugin:ml:test
Manual verification: configure a datafeed against an unavailable remote cluster and confirm skipped_clusters is non-zero in GET _ml/datafeeds/<id>/_stats

When a datafeed search throws ResourceNotFoundException (skipped remote clusters), DatafeedJob's extraction catch block never reached the crossClusterSearchStats.update() call, leaving skipped_clusters at zero in the running datafeed stats API response. Fix the pipeline so that cluster-state metadata observed before the exception is propagated out: * Each extractor stores cluster states in lastLinkedClusterStates and exposes them via a new DataExtractor.getLinkedClusterStates() default method. * ScrollDataExtractor updates lastLinkedClusterStates in the checkForSkippedClusters catch block before re-throwing. * AbstractAggregationDataExtractor and CompositeAggregationDataExtractor wrap the static executeSearchRequest() ResourceNotFoundException in a new package-private SkippedClustersException that carries the states through the static-method boundary, then update lastLinkedClusterStates when catching it in search(). * ChunkedDataExtractor.getLinkedClusterStates() delegates to the inner extractor so failures inside a chunk window propagate outward. * DatafeedJob.run() calls dataExtractor.getLinkedClusterStates() in the extraction catch block and passes non-empty results to crossClusterSearchStats.update() before re-throwing.

elasticsearchmachine · 2026-08-24T23:41:39Z

Hi @edsavage, I've created a changelog YAML for you.

github-actions · 2026-08-24T23:42:32Z

🔍 Preview links for changed docs

⏳ Building and deploying preview... View progress

This comment will be updated with preview links when the build is complete.

github-actions · 2026-08-24T23:44:18Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

Copilot

Pull request overview

This PR fixes ML datafeed running stats reporting skipped_clusters: 0 when CCS searches fail due to skipped remote clusters, by propagating per-cluster metadata from extractor failure paths back to CrossClusterSearchStats.

Changes:

Add DataExtractor#getLinkedClusterStates() (default empty) and implement it in scroll/aggregation/chunked extractors to expose the most recently observed CCS cluster states (including on failures).
Capture linked cluster states before releasing SearchResponse when ResourceNotFoundException is thrown due to skipped clusters (scroll + aggregation paths).
Update DatafeedJob to update crossClusterSearchStats in the extraction failure catch block using extractor-provided partial states; add changelog entry.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/scroll/ScrollDataExtractor.java	Captures linked cluster states on skipped-cluster failure before releasing the scroll response; exposes cached states via `getLinkedClusterStates()`.
x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/DataExtractor.java	Adds default `getLinkedClusterStates()` to allow failure-path CCS state propagation.
x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/chunked/ChunkedDataExtractor.java	Delegates `getLinkedClusterStates()` to the active inner extractor so chunked extraction can surface failure-path states.
x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/aggregation/SkippedClustersException.java	New internal exception used to carry linked cluster states extracted from a response that is about to be released.
x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/aggregation/CompositeAggregationDataExtractor.java	Captures linked cluster states when the new skipped-cluster wrapper exception is thrown; exposes cached states.
x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/aggregation/AbstractAggregationDataExtractor.java	Wraps skipped-cluster `ResourceNotFoundException` to retain linked cluster states prior to response release; exposes cached states.
x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/DatafeedJob.java	Updates CCS stats even when extraction fails by pulling partial linked cluster states from the extractor in the exception path.
docs/changelog/157567.yaml	Adds ML changelog entry for the bug fix.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

- Re-throw original ResourceNotFoundException from SkippedClustersException catch in AbstractAggregationDataExtractor and CompositeAggregationDataExtractor so the exception type and HTTP status code seen by callers and unwrapCause() are unchanged. - Persist the richer linked-cluster state back to lastLinkedClusterStates in ChunkedDataExtractor.getLinkedClusterStates() so subsequent calls return the merged snapshot rather than recomputing from a potentially cleared inner extractor. - Add DatafeedJobTests.testSkippedClustersStatsUpdatedOnExtractionFailure covering the failure path where the extractor throws while returning non-empty cluster states.

elasticsearchmachine · 2026-08-25T01:17:36Z

Pinging @elastic/ml-core (Team:ML)

elasticsearchmachine · 2026-08-27T21:57:10Z

💔 Backport failed

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 157567

…ilure (#157933) Backport of #157567 to 9.5. Conflict resolution: DatafeedJobTests.java on 9.5 was missing the two new imports (ElasticsearchSecurityException, ResourceNotFoundException) and the three new test methods introduced on main alongside the fix. Both were cleanly added.

…ilure (#157934) Backport of #157567 to 9.4

…elastic#157567) When a datafeed search against a remote cluster throws ResourceNotFoundException (one or more remote clusters were skipped), the extraction-failure code path in DatafeedJob.run() re-throws before reaching crossClusterSearchStats.update(). As a result skipped_clusters is always 0 in the running datafeed stats API response even when every search round skips remote clusters. This PR wires cluster-state metadata from the point of failure back to the stats update

edsavage added >bug :ml Machine learning Team:ML Meta label for the ML team auto-backport Automatically create backport pull requests when merged v9.6.0 v8.19.21 v9.4.6 v9.5.3 labels Aug 24, 2026

Update docs/changelog/157567.yaml

b142610

edsavage requested a lite review from Copilot August 24, 2026 23:57

Copilot started reviewing on behalf of edsavage August 24, 2026 23:57 View session

Copilot AI reviewed Aug 25, 2026

View reviewed changes

edsavage marked this pull request as ready for review August 25, 2026 01:17

Merge branch 'main' into ml-datafeed-ccs-stats-on-extraction-failure

88d9b9a

edsavage added 2 commits August 26, 2026 09:30

[ML] Propagate skipped-cluster states from ChunkedDataExtractor summary

2b93abc

[ML] Address review comments on CCS stats extraction failure PR

f5d939f

edsavage requested review from prwhelan and valeriy42 August 25, 2026 22:52

edsavage added 2 commits August 26, 2026 13:58

Merge branch 'main' into ml-datafeed-ccs-stats-on-extraction-failure

ea360ff

Merge branch 'main' into ml-datafeed-ccs-stats-on-extraction-failure

fe7b92c

prwhelan approved these changes Aug 27, 2026

View reviewed changes

edsavage merged commit 3ce0b34 into elastic:main Aug 27, 2026
37 checks passed

elasticsearchmachine added the backport pending label Aug 27, 2026

This was referenced Aug 27, 2026

[9.5][ML] Surface CCS skipped-cluster stats on datafeed extraction failure #157933

Merged

[9.4][ML] Surface CCS skipped-cluster stats on datafeed extraction failure #157934

Merged

edsavage removed >bug backport pending :ml Machine learning Team:ML Meta label for the ML team auto-backport Automatically create backport pull requests when merged v9.6.0 v8.19.21 v9.4.6 v9.5.3 labels Aug 27, 2026

edsavage added a commit that referenced this pull request Aug 28, 2026

[9.4][ML] Surface CCS skipped-cluster stats on datafeed extraction fa…

d417817

…ilure (#157934) Backport of #157567 to 9.4

edsavage added >bug :ml Machine learning v9.6.0 v8.19.21 v9.4.6 v9.5.3 and removed v8.19.21 labels Aug 28, 2026

edsavage mentioned this pull request Sep 2, 2026

[ML] Auto-stop datafeed after K consecutive extraction failures #158426

Open

edsavage mentioned this pull request Sep 17, 2026

[ML] Auto-stop datafeed after consecutive extraction failures #159554

Merged

5 tasks done

Navigation Menu

Repository navigation

[ML] Surface CCS skipped-cluster stats on datafeed extraction failure - #157567

[ML] Surface CCS skipped-cluster stats on datafeed extraction failure#157567
edsavage merged 8 commits into
elastic:mainfrom
edsavage:ml-datafeed-ccs-stats-on-extraction-failure

edsavage commented Aug 24, 2026 •

edited

Loading

Uh oh!

elasticsearchmachine commented Aug 24, 2026

Uh oh!

github-actions Bot commented Aug 24, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 24, 2026

When to use applies_to tags:

What NOT to do:

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Aug 25, 2026

Uh oh!

Uh oh!

elasticsearchmachine commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

edsavage commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

elasticsearchmachine commented Aug 24, 2026

Uh oh!

github-actions Bot commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

github-actions Bot commented Aug 24, 2026

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Aug 25, 2026

Uh oh!

Uh oh!

elasticsearchmachine commented Aug 27, 2026

💔 Backport failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

edsavage commented Aug 24, 2026 •

edited

Loading

github-actions Bot commented Aug 24, 2026 •

edited

Loading