[ML] Adds new Data comparison view by qn895 · Pull Request #161365 · elastic/kibana

qn895 · 2023-07-06T15:01:51Z

Summary

Follow up of the original proof-of-concept PR which started as a onweek project. This PR introduces the new Data Comparison view in ML. This view lies under Data Visualizer in the navigation, and showcases the biggest difference in each field for two different time ranges of data.

Screen.Recording.2023-07-06.at.09.43.24.mov

Part of this PR also includes some refactoring to consolidate code within ML/AIOps/Data Visualizer plugin. Changes include:

A new @kbn/ml-in-memory-table package which contains useTableState
Moved DocumentCountChart from plugins/aiops to @kbn/ml-aiops-components
Moved getDefaultQuery and type SearchQueryLanguage to @kbn/ml-query-utils
Modified DocumentCountChart to take additional props to showcase colors for the annotation and different labels for the brush

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

This was checked for breaking API changes and was labeled appropriately

…-fix'

…fix'

…e from '@kbn/ml-query-utils'

peteharverson · 2023-07-11T16:18:56Z

Adding a comment for feedback as I test:

I assume this only works for data views with timestamps? In which case, we should add a similar callout to that used on the AIOps pages:

If we keep this inside the Data visualizer nav grouping, we should consider changing the routing for the breadcrumbs so that the 'Data visualizer' header in the left nav no longer acts as a link to the select data view / file page. Otherwise if I click on the 'Data visualizer' breadcrumb it takes me out of the Data comparison workflow and into the data visualizer selector page.

Using packetbeat-7.10.0-2021-03-10-no-sunburst* - run an analysis with client.bytes > 10. After completion edit the filter to client.bytes > 100 and when it reruns the page crashes with this error:

There is a - after the progress bar %, which works for e.g. Explain Log Rate spikes as we append text on the stage the analysis is in, but we should remove it here, unless you can think of text to add after the -

I find that for larger data sets that it can take a while for the progress to move past 0%, is there some text we could add to indicate something is happening at this initial stage?

qn895 · 2023-07-11T19:33:08Z

  barColorOverride,
+  barStyleAccessor,
  barHighlightColorOverride,
+  deviationBrush = {},


Note: Passing some additional optional overrides here

peteharverson · 2023-07-26T10:27:15Z

+
+  const forceRefresh = useCallback(() => setLastRefresh(Date.now()), [setLastRefresh]);
+
+  const randomSampler = useMemo(


Have you run into this error when using the random sampler. Here the error occurs even at 50% sampling rate. Is there anything we can do to prevent this, or add a suggestion to the error callout to try increasing / turning off random sampling:

Should be fixed with this change 05245ea (#161365)

qn895 · 2023-07-26T20:48:21Z

@elasticmachine merge upstream

peteharverson

Tested latest changes and LGTM

walterra

Latest changes LGTM. Just added a small note that another plain string could be replaced by a constant. Would also be good to take note to follow up at some point with code consolidation with the parts that are now somewhat duplicated from the aiops plugin.

walterra · 2023-07-28T09:41:31Z

+  basePath: string
+): MlRoute => ({
+  id: 'data_view_data_comparison',
+  path: createPath('data_comparison_index_select'),


This could use ML_PAGES.DATA_COMPARISON_INDEX_SELECT.

qn895 · 2023-07-31T14:08:25Z

@elasticmachine merge upstream

kibana-ci · 2023-07-31T14:52:01Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`aiops`	435	439	+4
`dataVisualizer`	348	538	+190
`ml`	1764	1770	+6
`transform`	363	366	+3
total			+203

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`dataVisualizer`	24	25	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`aiops`	535.2KB	539.6KB	+4.4KB
`dataVisualizer`	376.0KB	605.0KB	⚠️ +229.0KB
`ml`	3.4MB	3.4MB	+2.5KB
`transform`	404.6KB	404.7KB	+83.0B
total			+236.0KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`@kbn/aiops-components`	0	1	+1
`dataVisualizer`	0	1	+1
total			+2

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`dataVisualizer`	21.7KB	23.1KB	+1.4KB
`ml`	73.9KB	74.0KB	+157.0B
total			+1.5KB

Unknown metric groups

API count

id	before	after	diff
`@kbn/aiops-components`	6	30	+24
`@kbn/ml-in-memory-table`	-	11	+11
`@kbn/ml-query-utils`	11	14	+3
`@kbn/ml-random-sampler-utils`	5	30	+25
`dataVisualizer`	28	31	+3
total			+66

async chunk count

id	before	after	diff
`dataVisualizer`	5	8	+3

ESLint disabled line counts

id	before	after	diff
`@kbn/aiops-components`	0	2	+2
`aiops`	23	21	-2
`dataVisualizer`	32	45	+13
total			+13

Total ESLint disabled count

id	before	after	diff
`@kbn/aiops-components`	0	2	+2
`aiops`	23	21	-2
`dataVisualizer`	32	45	+13
total			+13

History

💚 Build #145675 succeeded 8c8d116
💚 Build #145233 succeeded 3021ebf
💚 Build #145051 succeeded 16c7ebd
💚 Build #145044 succeeded 0a4ea12
💔 Build #145008 failed 0622573

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @qn895

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

## Summary Related PR: #161365 Related issue: https://github.com/elastic/platform-docs-team/issues/153 This PR drafts documentation for the new data comparison feature under the Data Visualizer in Kibana.

## Summary Related PR: elastic#161365 Related issue: https://github.com/elastic/platform-docs-team/issues/153 This PR drafts documentation for the new data comparison feature under the Data Visualizer in Kibana. (cherry picked from commit e911038)

…164722) # Backport This will backport the following commits from `main` to `8.10`: - [[DOCS] Adds documentation for data comparison view (#164297)](#164297)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: István Zoltán Szabó <szabosteve@gmail.com>

qn895 added 2 commits July 5, 2023 18:18

Initialize Data comparison view (Refactor & fix conflicts from poc PR)

567f088

Clean up

a3dc124

qn895 self-assigned this Jul 6, 2023

qn895 added :ml v8.10.0 labels Jul 6, 2023

qn895 changed the title ~~Initialize Data comparison view (Refactor & fix conflicts from poc PR)~~ [ML] Add new Data comparison view Jul 6, 2023

qn895 force-pushed the data-drift-part-1 branch from 28fcf9c to 0f10040 Compare July 6, 2023 15:17

Renaming

f991036

qn895 force-pushed the data-drift-part-1 branch from 0f10040 to f991036 Compare July 6, 2023 15:19

kibanamachine and others added 3 commits July 6, 2023 15:58

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

9ea789b

…-fix'

Add tests

0ec0ed3

[ML] Refactor to data visualizer & its own packages

61c5da3

qn895 force-pushed the data-drift-part-1 branch from 58e6a65 to 61c5da3 Compare July 11, 2023 04:58

[ML] Fix git missing some moved files

5afb8a8

qn895 added the ci:cloud-deploy Create or update a Cloud deployment label Jul 11, 2023

kibanamachine and others added 6 commits July 11, 2023 14:35

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

5bf40bc

…fix'

[CI] Auto-commit changed files from 'node scripts/generate codeowners'

f33968b

Fix types

60ecf83

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

108969a

…fix'

Update imports for SearchQueryLanguage and SEARCH_QUERY_LANGUAGE to b…

7dbc4f8

…e from '@kbn/ml-query-utils'

Update imports for useTableState to be from '@kbn/ml-in-memory-table'

25156e9

peteharverson reviewed Jul 11, 2023

View reviewed changes

Comment thread x-pack/plugins/data_visualizer/public/application/data_comparison/data_comparison_view.tsx Outdated

qn895 added 5 commits July 11, 2023 13:00

Add callout message for non time-series data view

a129654

Fix aggegations undefined

cca0ffe

Allow to modify badge width

b13fa68

Fix progress message

c3d43fa

Added progress message

1ad5425

qn895 requested a review from walterra July 11, 2023 19:31

qn895 commented Jul 11, 2023

View reviewed changes

[ML] Add comments to aiops-component

c35bd60

peteharverson reviewed Jul 26, 2023

View reviewed changes

qn895 force-pushed the data-drift-part-1 branch from 05245ea to 5d078fd Compare July 26, 2023 14:51

[ML] Fix issue with no agg, comment, remove any

bf34f73

qn895 force-pushed the data-drift-part-1 branch from 5d078fd to bf34f73 Compare July 26, 2023 15:13

qn895 added 4 commits July 26, 2023 10:54

Revert api doc changes

f4a76d5

Rename typo ComparisonHistogram

f5ed74f

Add field type icon

0622573

Fix in memory comments

0a4ea12

Merge branch 'main' into data-drift-part-1

16c7ebd

peteharverson approved these changes Jul 27, 2023

View reviewed changes

qn895 added 2 commits July 27, 2023 09:35

Remove todo

bb91294

Merge remote-tracking branch 'upstream/main' into data-drift-part-1

3021ebf

walterra approved these changes Jul 28, 2023

View reviewed changes

qn895 added 2 commits July 28, 2023 11:26

Merge upstream/main into branch

7c9f899

Fix SearchQueryLanguage

8c8d116

Merge branch 'main' into data-drift-part-1

159241e

qn895 enabled auto-merge (squash) July 31, 2023 14:08

qn895 merged commit 0728003 into elastic:main Jul 31, 2023

kibanamachine added the backport:skip This PR does not require backporting label Jul 31, 2023

ThomThomson pushed a commit to ThomThomson/kibana that referenced this pull request Aug 1, 2023

[ML] Add new Data comparison view (elastic#161365)

fab1097

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

szabosteve mentioned this pull request Aug 21, 2023

[DOCS] Adds documentation for data comparison view #164297

Merged

1 task done

peteharverson changed the title ~~[ML] Add new Data comparison view~~ [ML] Adds new Data comparison view Aug 22, 2023

Navigation Menu

Repository navigation

[ML] Adds new Data comparison view - #161365

[ML] Adds new Data comparison view#161365
qn895 merged 73 commits into
elastic:mainfrom
qn895:data-drift-part-1

qn895 commented Jul 6, 2023 •

edited

Loading

Uh oh!

Uh oh!

peteharverson commented Jul 11, 2023 •

edited

Loading

Uh oh!

qn895 Jul 11, 2023

Uh oh!

peteharverson Jul 26, 2023

Uh oh!

qn895 Jul 26, 2023

Uh oh!

qn895 commented Jul 26, 2023

Uh oh!

peteharverson left a comment

Uh oh!

walterra left a comment

Uh oh!

walterra Jul 28, 2023

Uh oh!

qn895 commented Jul 31, 2023

Uh oh!

kibana-ci commented Jul 31, 2023 •

edited

Loading

API count

async chunk count

ESLint disabled line counts

Total ESLint disabled count

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants


		const forceRefresh = useCallback(() => setLastRefresh(Date.now()), [setLastRefresh]);

		const randomSampler = useMemo(

Conversation

qn895 commented Jul 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Risk Matrix

For maintainers

Uh oh!

Uh oh!

peteharverson commented Jul 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qn895 Jul 11, 2023

Choose a reason for hiding this comment

Uh oh!

peteharverson Jul 26, 2023

Choose a reason for hiding this comment

Uh oh!

qn895 Jul 26, 2023

Choose a reason for hiding this comment

Uh oh!

qn895 commented Jul 26, 2023

Uh oh!

peteharverson left a comment

Choose a reason for hiding this comment

Uh oh!

walterra left a comment

Choose a reason for hiding this comment

Uh oh!

walterra Jul 28, 2023

Choose a reason for hiding this comment

Uh oh!

qn895 commented Jul 31, 2023

Uh oh!

kibana-ci commented Jul 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Public APIs missing exports

Page load bundle

API count

async chunk count

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

qn895 commented Jul 6, 2023 •

edited

Loading

peteharverson commented Jul 11, 2023 •

edited

Loading

kibana-ci commented Jul 31, 2023 •

edited

Loading