Repository navigation
Migrate watcher and ccr ftr to scout - #271663
Merged
dmlemeshko merged 18 commits intoJun 3, 2026
Merged
Conversation
…cout' into migrate-watcher-and-ccr-ftr-to-scout
Contributor
Author
|
/flaky scoutConfig:x-pack/platform/plugins/private/cross_cluster_replication/test/scout/ui/playwright.config.ts:20 |
Contributor
Flaky Test Runner✅ Build triggered - kibana-flaky-test-suite-runner#12495
|
Contributor
Author
|
/flaky scoutConfig:x-pack/platform/plugins/private/cross_cluster_replication/test/scout/ui/playwright.config.ts:20 |
Contributor
Flaky Test Runner✅ Build triggered - kibana-flaky-test-suite-runner#12496
|
Contributor
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#12495 |
Contributor
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#12496[✅] x-pack/platform/plugins/private/cross_cluster_replication/test/scout/ui/playwright.config.ts (--arch stateful --domain classic): 20/20 tests passed. |
dmlemeshko
marked this pull request as ready for review
May 28, 2026 15:26
steliosmavro
approved these changes
May 28, 2026
dmlemeshko
enabled auto-merge (squash)
May 28, 2026 16:20
kowalczyk-krzysztof
approved these changes
May 29, 2026
kapral18
suggested changes
May 29, 2026
kapral18
left a comment
Contributor
There was a problem hiding this comment.
Clean migration overall — the new specs are higher quality than what they replace (real minimal roles, a stronger API assertion, hook-based ES cleanup). Two parity items before merge:
- CCR a11y coverage — the deleted CCR a11y suite (8 snapshots) has no Scout replacement, so the "full coverage preserved, no FTR scenarios were dropped" claim isn't accurate yet. Let's restore it.
feature_controls— thekibana_admincase uses a customcluster: []role instead of the built-inkibana_admin, so it tests a different subject than the FTR did. Use the built-in role to keep it like-for-like.
Details inline. Everything else is a nit.
dmlemeshko
disabled auto-merge
May 29, 2026 10:39
…/scout/ui/tests/home_page.spec.ts Co-authored-by: Karen Grigoryan <karen@karenjs.com>
dmlemeshko
added a commit
that referenced
this pull request
Jun 2, 2026
## Summary Unblocks #271663 Many FTR tests rely on Kibana build-in roles and it makes sense to add its support to Scout. It should simplify `FTR -> Scout` migration: ### UI tests auth flow: ``` browserAuth.loginWithBuiltinRole('kibana_admin') │ └─ samlAuth.setBuiltinRole('kibana_admin') ← new helper added │ ├─ esClient.security.getRole({ name: 'kibana_admin' }) │ └─ response: │ { │ kibana_admin: { │ cluster: [], │ indices: [], │ applications: [{ application: 'kibana-.kibana', privileges: ['all'], resources: ['*'] }], │ run_as: [], │ metadata: { _reserved: true }, ← strip │ transient_metadata: { enabled: true } ← strip │ description: '...' ← strip │ } │ } │ └─ setCustomRole(descriptor) ├─ hash check (reuses cached role within same worker) ├─ createElasticsearchCustomRole(esClient, 'custom_role_worker_1', descriptor) │ └─ PUT /_security/role/custom_role_worker_1 { cluster, indices, applications, run_as } └─ customRoleHash = hash(descriptor) └─ loginAs(samlAuth.customRoleName) ← unchanged └─ [local or cloud — same path as loginWithCustomRole] ``` ### API tests auth flow: ``` requestAuth.getApiKeyForBuiltinRole('kibana_admin') │ └─ samlAuth.fetchBuiltinRoleDescriptor('kibana_admin') ← new helper added, no role created in ES │ ├─ esClient.security.getRole({ name: 'kibana_admin' }) │ └─ response: │ { │ kibana_admin: { │ cluster: [], │ indices: [], │ applications: [{ application: 'kibana-.kibana', privileges: ['all'], resources: ['*'] }], │ run_as: [], │ metadata: { _reserved: true }, ← dropped (allow-list) │ transient_metadata: { enabled: true } ← dropped (allow-list) │ description: '...' ← dropped (allow-list) │ } │ } │ └─ returns allow-listed descriptor (cluster, indices, applications, run_as only) │ └─ createApiKeyWithAdminCredentials( roleName = 'kibana_admin', descriptors = { kibana_admin: descriptor } ← inline, no ES role involved ) │ ├─ samlAuth.session.getApiCredentialsForRole('admin') ← admin cookie header │ └─ POST /internal/security/api_key body: { name: 'myTestApiKey-0-kibana_admin-worker-1', role_descriptors: { kibana_admin: { cluster: [], indices: [], applications: [{ application: 'kibana-.kibana', privileges: ['all'], resources: ['*'] }], run_as: [] } } } └─ { id, name, api_key, encoded } └─ { apiKey, apiKeyHeader: { Authorization: 'ApiKey <encoded>' } } ``` Note: since `build-in` roles are available only in stateful, the error will be thrown when you call it in tests running serverless env --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Cesare de Cal <cesare.decal@elastic.co>
dmlemeshko
added a commit
that referenced
this pull request
Jun 18, 2026
# Backport This will backport the following commits from `main` to `9.3`: - [Migrate watcher and ccr ftr to scout (#271663)](#271663) <!--- Backport version: 11.0.2 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Dzmitry Lemechko","email":"dzmitry.lemechko@elastic.co"},"sourceCommit":{"committedDate":"2026-06-03T16:32:56Z","message":"Migrate watcher and ccr ftr to scout (#271663)\n\n## Summary\n\n### FTR configs removed\n\nConfig / file deleted | Plugin\n-- | --\nx-pack/platform/test/functional/apps/watcher/config.ts + index.ts |\nWatcher\nx-pack/platform/test/api_integration/apis/watcher/ (whole dir) | Watcher\nx-pack/platform/test/accessibility/apps/group3/watcher.ts | Watcher\nx-pack/platform/test/functional/page_objects/watcher_page.ts | Watcher\n\nx-pack/platform/test/functional/apps/cross_cluster_replication/config.ts + index.ts\n| CCR\n\nx-pack/platform/test/functional/apps/cross_cluster_replication/feature_controls/\n| CCR\n\nx-pack/platform/test/accessibility/apps/group3/cross_cluster_replication.ts\n| CCR\n\n### FTR → Scout scenario map\n\nParity verdict: full coverage preserved. No FTR scenarios were dropped.\n\nFTR suite | FTR scenario | Scout spec | Scout scenario\n-- | -- | -- | --\nfunctional/apps/watcher | Create a JSON watch | watch_crud.spec.ts |\ncreate JSON watch step\nfunctional/apps/watcher | Reject duplicate watch ID | watch_crud.spec.ts\n| reject duplicate watch ID with error callout step\nfunctional/apps/watcher | Delete watch via bulk selection |\nwatch_crud.spec.ts | delete watch via bulk selection step\naccessibility/watcher | A11y snapshot — empty list | watch_crud.spec.ts\n| page.checkA11y in watcher list page loads step\naccessibility/watcher | A11y snapshot — watch list | watch_crud.spec.ts\n| page.checkA11y in create JSON watch step\naccessibility/watcher | A11y snapshot — duplicate error |\nwatch_crud.spec.ts | page.checkA11y in reject duplicate step\naccessibility/watcher | A11y snapshot — delete modal |\nwatch_crud.spec.ts | page.checkA11y in delete watch step\napi_integration/watcher |\nGET /api/watcher/indices/index_patterns returns data views |\nindex_patterns.spec.ts | returns a list that includes the created data\nview title\nfunctional/apps/cross_cluster_replication | App renders with correct\ntitle + button | home_page.spec.ts | loads the app and displays the\ncreate follower index button\nfunctional/apps/cross_cluster_replication/feature_controls |\nkibana_admin — CCR link hidden | feature_controls.spec.ts | kibana_admin\ncan access Stack Management but not CCR\nfunctional/apps/cross_cluster_replication/feature_controls | ccr_user —\nCCR link visible | feature_controls.spec.ts | ccr_user with dashboard\nread can access CCR in Stack Management\n\n</div></div></div>\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Karen Grigoryan <karen@karenjs.com>","sha":"977c061c93c1de8385853040903a61227d56e811","branchLabelMapping":{"^v9.5.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:all-open","v9.5.0","v9.4.3"],"title":"Migrate watcher and ccr ftr to scout","number":271663,"url":"https://github.com/elastic/kibana/pull/271663","mergeCommit":{"message":"Migrate watcher and ccr ftr to scout (#271663)\n\n## Summary\n\n### FTR configs removed\n\nConfig / file deleted | Plugin\n-- | --\nx-pack/platform/test/functional/apps/watcher/config.ts + index.ts |\nWatcher\nx-pack/platform/test/api_integration/apis/watcher/ (whole dir) | Watcher\nx-pack/platform/test/accessibility/apps/group3/watcher.ts | Watcher\nx-pack/platform/test/functional/page_objects/watcher_page.ts | Watcher\n\nx-pack/platform/test/functional/apps/cross_cluster_replication/config.ts + index.ts\n| CCR\n\nx-pack/platform/test/functional/apps/cross_cluster_replication/feature_controls/\n| CCR\n\nx-pack/platform/test/accessibility/apps/group3/cross_cluster_replication.ts\n| CCR\n\n### FTR → Scout scenario map\n\nParity verdict: full coverage preserved. No FTR scenarios were dropped.\n\nFTR suite | FTR scenario | Scout spec | Scout scenario\n-- | -- | -- | --\nfunctional/apps/watcher | Create a JSON watch | watch_crud.spec.ts |\ncreate JSON watch step\nfunctional/apps/watcher | Reject duplicate watch ID | watch_crud.spec.ts\n| reject duplicate watch ID with error callout step\nfunctional/apps/watcher | Delete watch via bulk selection |\nwatch_crud.spec.ts | delete watch via bulk selection step\naccessibility/watcher | A11y snapshot — empty list | watch_crud.spec.ts\n| page.checkA11y in watcher list page loads step\naccessibility/watcher | A11y snapshot — watch list | watch_crud.spec.ts\n| page.checkA11y in create JSON watch step\naccessibility/watcher | A11y snapshot — duplicate error |\nwatch_crud.spec.ts | page.checkA11y in reject duplicate step\naccessibility/watcher | A11y snapshot — delete modal |\nwatch_crud.spec.ts | page.checkA11y in delete watch step\napi_integration/watcher |\nGET /api/watcher/indices/index_patterns returns data views |\nindex_patterns.spec.ts | returns a list that includes the created data\nview title\nfunctional/apps/cross_cluster_replication | App renders with correct\ntitle + button | home_page.spec.ts | loads the app and displays the\ncreate follower index button\nfunctional/apps/cross_cluster_replication/feature_controls |\nkibana_admin — CCR link hidden | feature_controls.spec.ts | kibana_admin\ncan access Stack Management but not CCR\nfunctional/apps/cross_cluster_replication/feature_controls | ccr_user —\nCCR link visible | feature_controls.spec.ts | ccr_user with dashboard\nread can access CCR in Stack Management\n\n</div></div></div>\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Karen Grigoryan <karen@karenjs.com>","sha":"977c061c93c1de8385853040903a61227d56e811"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.5.0","branchLabelMappingKey":"^v9.5.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/271663","number":271663,"mergeCommit":{"message":"Migrate watcher and ccr ftr to scout (#271663)\n\n## Summary\n\n### FTR configs removed\n\nConfig / file deleted | Plugin\n-- | --\nx-pack/platform/test/functional/apps/watcher/config.ts + index.ts |\nWatcher\nx-pack/platform/test/api_integration/apis/watcher/ (whole dir) | Watcher\nx-pack/platform/test/accessibility/apps/group3/watcher.ts | Watcher\nx-pack/platform/test/functional/page_objects/watcher_page.ts | Watcher\n\nx-pack/platform/test/functional/apps/cross_cluster_replication/config.ts + index.ts\n| CCR\n\nx-pack/platform/test/functional/apps/cross_cluster_replication/feature_controls/\n| CCR\n\nx-pack/platform/test/accessibility/apps/group3/cross_cluster_replication.ts\n| CCR\n\n### FTR → Scout scenario map\n\nParity verdict: full coverage preserved. No FTR scenarios were dropped.\n\nFTR suite | FTR scenario | Scout spec | Scout scenario\n-- | -- | -- | --\nfunctional/apps/watcher | Create a JSON watch | watch_crud.spec.ts |\ncreate JSON watch step\nfunctional/apps/watcher | Reject duplicate watch ID | watch_crud.spec.ts\n| reject duplicate watch ID with error callout step\nfunctional/apps/watcher | Delete watch via bulk selection |\nwatch_crud.spec.ts | delete watch via bulk selection step\naccessibility/watcher | A11y snapshot — empty list | watch_crud.spec.ts\n| page.checkA11y in watcher list page loads step\naccessibility/watcher | A11y snapshot — watch list | watch_crud.spec.ts\n| page.checkA11y in create JSON watch step\naccessibility/watcher | A11y snapshot — duplicate error |\nwatch_crud.spec.ts | page.checkA11y in reject duplicate step\naccessibility/watcher | A11y snapshot — delete modal |\nwatch_crud.spec.ts | page.checkA11y in delete watch step\napi_integration/watcher |\nGET /api/watcher/indices/index_patterns returns data views |\nindex_patterns.spec.ts | returns a list that includes the created data\nview title\nfunctional/apps/cross_cluster_replication | App renders with correct\ntitle + button | home_page.spec.ts | loads the app and displays the\ncreate follower index button\nfunctional/apps/cross_cluster_replication/feature_controls |\nkibana_admin — CCR link hidden | feature_controls.spec.ts | kibana_admin\ncan access Stack Management but not CCR\nfunctional/apps/cross_cluster_replication/feature_controls | ccr_user —\nCCR link visible | feature_controls.spec.ts | ccr_user with dashboard\nread can access CCR in Stack Management\n\n</div></div></div>\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Karen Grigoryan <karen@karenjs.com>","sha":"977c061c93c1de8385853040903a61227d56e811"}},{"branch":"9.4","label":"v9.4.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/272818","number":272818,"state":"MERGED","mergeCommit":{"sha":"61e163101c71598ab75973950b44691bc9a6a607","message":"[9.4] Migrate watcher and ccr ftr to scout (#271663) (#272818)\n\n# Backport\n\nThis will backport the following commits from `main` to `9.4`:\n- [Migrate watcher and ccr ftr to scout\n(#271663)](https://github.com/elastic/kibana/pull/271663)\n\n\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sorenlouv/backport)\n\n\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>"}}]}] BACKPORT--> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This was referenced Jul 28, 2026
Closed
patrykkopycinski
pushed a commit
to patrykkopycinski/kibana
that referenced
this pull request
Aug 5, 2026
…c#271992) ## Summary Unblocks elastic#271663 Many FTR tests rely on Kibana build-in roles and it makes sense to add its support to Scout. It should simplify `FTR -> Scout` migration: ### UI tests auth flow: ``` browserAuth.loginWithBuiltinRole('kibana_admin') │ └─ samlAuth.setBuiltinRole('kibana_admin') ← new helper added │ ├─ esClient.security.getRole({ name: 'kibana_admin' }) │ └─ response: │ { │ kibana_admin: { │ cluster: [], │ indices: [], │ applications: [{ application: 'kibana-.kibana', privileges: ['all'], resources: ['*'] }], │ run_as: [], │ metadata: { _reserved: true }, ← strip │ transient_metadata: { enabled: true } ← strip │ description: '...' ← strip │ } │ } │ └─ setCustomRole(descriptor) ├─ hash check (reuses cached role within same worker) ├─ createElasticsearchCustomRole(esClient, 'custom_role_worker_1', descriptor) │ └─ PUT /_security/role/custom_role_worker_1 { cluster, indices, applications, run_as } └─ customRoleHash = hash(descriptor) └─ loginAs(samlAuth.customRoleName) ← unchanged └─ [local or cloud — same path as loginWithCustomRole] ``` ### API tests auth flow: ``` requestAuth.getApiKeyForBuiltinRole('kibana_admin') │ └─ samlAuth.fetchBuiltinRoleDescriptor('kibana_admin') ← new helper added, no role created in ES │ ├─ esClient.security.getRole({ name: 'kibana_admin' }) │ └─ response: │ { │ kibana_admin: { │ cluster: [], │ indices: [], │ applications: [{ application: 'kibana-.kibana', privileges: ['all'], resources: ['*'] }], │ run_as: [], │ metadata: { _reserved: true }, ← dropped (allow-list) │ transient_metadata: { enabled: true } ← dropped (allow-list) │ description: '...' ← dropped (allow-list) │ } │ } │ └─ returns allow-listed descriptor (cluster, indices, applications, run_as only) │ └─ createApiKeyWithAdminCredentials( roleName = 'kibana_admin', descriptors = { kibana_admin: descriptor } ← inline, no ES role involved ) │ ├─ samlAuth.session.getApiCredentialsForRole('admin') ← admin cookie header │ └─ POST /internal/security/api_key body: { name: 'myTestApiKey-0-kibana_admin-worker-1', role_descriptors: { kibana_admin: { cluster: [], indices: [], applications: [{ application: 'kibana-.kibana', privileges: ['all'], resources: ['*'] }], run_as: [] } } } └─ { id, name, api_key, encoded } └─ { apiKey, apiKeyHeader: { Authorization: 'ApiKey <encoded>' } } ``` Note: since `build-in` roles are available only in stateful, the error will be thrown when you call it in tests running serverless env --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Cesare de Cal <cesare.decal@elastic.co>
patrykkopycinski
pushed a commit
to patrykkopycinski/kibana
that referenced
this pull request
Aug 5, 2026
## Summary ### FTR configs removed Config / file deleted | Plugin -- | -- x-pack/platform/test/functional/apps/watcher/config.ts + index.ts | Watcher x-pack/platform/test/api_integration/apis/watcher/ (whole dir) | Watcher x-pack/platform/test/accessibility/apps/group3/watcher.ts | Watcher x-pack/platform/test/functional/page_objects/watcher_page.ts | Watcher x-pack/platform/test/functional/apps/cross_cluster_replication/config.ts + index.ts | CCR x-pack/platform/test/functional/apps/cross_cluster_replication/feature_controls/ | CCR x-pack/platform/test/accessibility/apps/group3/cross_cluster_replication.ts | CCR ### FTR → Scout scenario map Parity verdict: full coverage preserved. No FTR scenarios were dropped. FTR suite | FTR scenario | Scout spec | Scout scenario -- | -- | -- | -- functional/apps/watcher | Create a JSON watch | watch_crud.spec.ts | create JSON watch step functional/apps/watcher | Reject duplicate watch ID | watch_crud.spec.ts | reject duplicate watch ID with error callout step functional/apps/watcher | Delete watch via bulk selection | watch_crud.spec.ts | delete watch via bulk selection step accessibility/watcher | A11y snapshot — empty list | watch_crud.spec.ts | page.checkA11y in watcher list page loads step accessibility/watcher | A11y snapshot — watch list | watch_crud.spec.ts | page.checkA11y in create JSON watch step accessibility/watcher | A11y snapshot — duplicate error | watch_crud.spec.ts | page.checkA11y in reject duplicate step accessibility/watcher | A11y snapshot — delete modal | watch_crud.spec.ts | page.checkA11y in delete watch step api_integration/watcher | GET /api/watcher/indices/index_patterns returns data views | index_patterns.spec.ts | returns a list that includes the created data view title functional/apps/cross_cluster_replication | App renders with correct title + button | home_page.spec.ts | loads the app and displays the create follower index button functional/apps/cross_cluster_replication/feature_controls | kibana_admin — CCR link hidden | feature_controls.spec.ts | kibana_admin can access Stack Management but not CCR functional/apps/cross_cluster_replication/feature_controls | ccr_user — CCR link visible | feature_controls.spec.ts | ccr_user with dashboard read can access CCR in Stack Management </div></div></div> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Karen Grigoryan <karen@karenjs.com>
SoniaSanzV
added a commit
that referenced
this pull request
Aug 6, 2026
Closes #281688 ## Summary - Migrates the remaining `cross_cluster_replication` API integration FTR suite (auto-follow patterns + follower indices) to a new Scout API suite under `x-pack/platform/plugins/private/cross_cluster_replication/test/scout/api/`; the CCR UI and a11y coverage was already migrated in #270902 / #271663. - Preserves the FTR's single-cluster setup: a self-referential remote cluster (the test node pointing at its own transport address) is registered via `esClient.cluster.putSettings`, resolving the transport `publish_address` dynamically with `nodes.info` instead of the FTR's hardcoded `TEST_ES_TRANSPORT_PORT`/`localhost:9300`. - The FTR's advanced-settings-defaults test silently `return`ed when the follower was transiently `paused`; the migrated spec polls for `active` and asserts the hard-coded `FOLLOWER_INDEX_ADVANCED_SETTINGS` against Elasticsearch for real. - Deletes the FTR suite and its `loadTestFile` entry, drops the now-unused `@kbn/cross-cluster-replication-plugin` reference from `x-pack/platform/test/tsconfig.json`, and removes two stale CCR CODEOWNERS entries left by the earlier UI migration. - Runs stateful classic only (`tags.stateful.classic`), matching the CCR UI Scout suite and the FTR's `skipCloud`: the self-referential `localhost` remote is not reachable on Cloud. ### Test plan - Scout flaky test runner ran with `/flaky scoutConfig:x-pack/platform/plugins/private/cross_cluster_replication/test/scout/api/playwright.config.ts:30` [kibana-flaky-test-suite-runner#13438](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/13438) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FTR configs removed
FTR → Scout scenario map
Parity verdict: full coverage preserved. No FTR scenarios were dropped.