Migrations v2 ignore fleet agent events by rudolf · Pull Request #96690 · elastic/kibana

rudolf · 2021-04-09T09:08:35Z

Summary

Ignore fleet-agent-events and tsvb-validation-telemetry saved objects from both v1 and v2 migrations.

Release notes

Users who tried the fleet agent beta in 7.10 might have hundreds of thousands of fleet-agent-events in their .kibana index. Due to the large volume of documents, upgrades to 7.12 would frequently fail for these users. In 7.12.1 we drop all fleet-agent-events during the upgrade migration to address this problem. The fleet-agent-events documents are still left in the older .kibana_N index from previous versions, once the upgrade has succeeded we recommend deleting these indices to recover the disk space, see https://www.elastic.co/guide/en/kibana/current/upgrade-migrations.html#upgrade-migrations-old-indices

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2021-04-09T13:21:33Z

Pinging @elastic/kibana-core (Team:Core)

rudolf · 2021-04-09T13:24:10Z

    });

-    it('should create or update the fleet_enroll user if called multiple times with forceRecreate flag', async () => {
+    it.skip('should create or update the fleet_enroll user if called multiple times with forceRecreate flag', async () => {


@elastic/fleet @nchaulet somehow this test is failing when I remove fleet-agent-events. The same test also fails on master if I enable v1 migrations for this test with the following:

--- a/x-pack/test/fleet_api_integration/config.ts +++ b/x-pack/test/fleet_api_integration/config.ts @@ -65,6 +65,7 @@ export default async function ({ readConfigFile }: FtrConfigProviderContext) { kbnTestServer: { ...xPackAPITestsConfig.get('kbnTestServer'), serverArgs: [ + '--migrations.enableV2=false', ...xPackAPITestsConfig.get('kbnTestServer.serverArgs'), ...(registryPort ? [`--xpack.fleet.registryUrl=http://localhost:${registryPort}`] : []), ],

We're targeting 7.12.1 for this fix, so please take a look and let me know if it's possible to fix this test or if we should skip it.

I think it's okay to skip it, we are going to remove that API in 7.13.

elasticmachine · 2021-04-09T13:24:19Z

Pinging @elastic/fleet (Team:Fleet)

afharo

🚀 LGTM! Just added a NIT about term vs. terms. According to https://discuss.elastic.co/t/term-vs-terms-filter/15533, the only difference in performance is caching. Since, ideally, we should be running this once, I guess it makes no difference.

rudolf · 2021-04-12T14:06:37Z

@elasticmachine merge upstream

kibanamachine · 2021-04-12T16:09:57Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💔 Build #118515 failed 2eeac2a
💔 Build #118504 failed 259308c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* migrationsv2: ignore fleet agent events and tsvb telemetry * migrationsv1: ignore tsvb-validation-telemetry * Skip fleet test that depends on fleet-agent-events * Fix typescript errors Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-04-12T16:22:40Z

💔 Backport failed

Status	Branch	Result
❌	7.12	Commit could not be cherrypicked due to conflicts
✅	7.x

Successful backport PRs will be merged automatically after passing CI.

To backport manually run:
node scripts/backport --pr 96690

* migrationsv2: ignore fleet agent events and tsvb telemetry * migrationsv1: ignore tsvb-validation-telemetry * Skip fleet test that depends on fleet-agent-events * Fix typescript errors Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Rudolf Meijering <skaapgif@gmail.com>

* migrationsv2: ignore fleet agent events and tsvb telemetry * migrationsv1: ignore tsvb-validation-telemetry * Skip fleet test that depends on fleet-agent-events * Fix typescript errors Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # src/core/server/saved_objects/migrationsv2/integration_tests/actions.test.ts

…sted sessions (#96938) (#96892) * Migrations v2 ignore fleet agent events (#96690) * migrationsv2: ignore fleet agent events and tsvb telemetry * migrationsv1: ignore tsvb-validation-telemetry * Skip fleet test that depends on fleet-agent-events * Fix typescript errors Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # src/core/server/saved_objects/migrationsv2/integration_tests/actions.test.ts * Correctly fix merge conflicts * Skip failing fleet-agent-events tests * Skip security solution endpoint telemetry tests depending on fleet-agent-events so's * Exclude non-persisted sessions from SO migration (#96938) (cherry picked from commit bfd5b7b) * use `any` for estypes.QueryContainer Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Pierre Gayvallet <pierre.gayvallet@gmail.com>

tonymeehan · 2021-04-28T16:36:09Z

👍 🥇 💯

* migrationsv2: ignore fleet agent events and tsvb telemetry * migrationsv1: ignore tsvb-validation-telemetry * Skip fleet test that depends on fleet-agent-events * Fix typescript errors Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

rudolf added 2 commits April 9, 2021 01:25

migrationsv2: ignore fleet agent events and tsvb telemetry

8861c86

migrationsv1: ignore tsvb-validation-telemetry

259308c

rudolf added Team:Core Platform Core services: plugins, logging, config, saved objects, http, ES client, i18n, etc t// project:ResilientSavedObjectMigrations Reduce Kibana upgrade failures by making saved object migrations more resilient v7.12.1 v7.13.0 v8.0.0 labels Apr 9, 2021

rudolf added 2 commits April 9, 2021 15:19

Skip fleet test that depends on fleet-agent-events

1ac27e7

Fix typescript errors

2eeac2a

rudolf marked this pull request as ready for review April 9, 2021 13:21

rudolf requested a review from a team as a code owner April 9, 2021 13:21

rudolf commented Apr 9, 2021

View reviewed changes

botelastic Bot added the Team:Fleet - DEPRECATED Use Team:streams-ui label Apr 9, 2021

rudolf added the release_note:fix label Apr 9, 2021

afharo approved these changes Apr 9, 2021

View reviewed changes

Comment thread src/core/server/saved_objects/migrations/core/elastic_index.ts

Comment thread src/core/server/saved_objects/migrationsv2/actions/index.ts

Merge branch 'master' into migrations-v2-ignore-fleet-agent-events

2048186

rudolf added the auto-backport Deprecated - use backport:version if exact versions are needed label Apr 12, 2021

rudolf merged commit f544d8d into elastic:master Apr 12, 2021

rudolf deleted the migrations-v2-ignore-fleet-agent-events branch April 12, 2021 16:19

kibanamachine mentioned this pull request Apr 12, 2021

[7.x] Migrations v2 ignore fleet agent events (#96690) #96823

Merged

rudolf mentioned this pull request Apr 12, 2021

[7.12] Migrations v2 ignore fleet agent events (#96690) and non-persisted sessions (#96938) #96892

Merged

Navigation Menu

Repository navigation

Migrations v2 ignore fleet agent events - #96690

Migrations v2 ignore fleet agent events#96690
rudolf merged 5 commits into
elastic:masterfrom
rudolf:migrations-v2-ignore-fleet-agent-events

rudolf commented Apr 9, 2021 •

edited

Loading

Uh oh!

elasticmachine commented Apr 9, 2021

Uh oh!

rudolf Apr 9, 2021

Uh oh!

nchaulet Apr 12, 2021

Uh oh!

elasticmachine commented Apr 9, 2021

Uh oh!

afharo left a comment

Uh oh!

Uh oh!

Uh oh!

rudolf commented Apr 12, 2021

Uh oh!

kibanamachine commented Apr 12, 2021

Uh oh!

kibanamachine commented Apr 12, 2021

Uh oh!

tonymeehan commented Apr 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

rudolf commented Apr 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Release notes

Checklist

For maintainers

Uh oh!

elasticmachine commented Apr 9, 2021

Uh oh!

rudolf Apr 9, 2021

Choose a reason for hiding this comment

Uh oh!

nchaulet Apr 12, 2021

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Apr 9, 2021

Uh oh!

afharo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rudolf commented Apr 12, 2021

Uh oh!

kibanamachine commented Apr 12, 2021

💚 Build Succeeded

Metrics [docs]

History

Uh oh!

kibanamachine commented Apr 12, 2021

💔 Backport failed

Uh oh!

tonymeehan commented Apr 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

rudolf commented Apr 9, 2021 •

edited

Loading