[timelion] allow sum, subtract, multiply, divide functions to accept seriesList with multiple series by nreese · Pull Request #14891 · elastic/kibana

nreese · 2017-11-10T02:00:14Z

Allow something like:

.es(
	index='*', timefield='field', metric='sum:int1', split='user:10'
).subtract(.es(
	index='*', timefield='field', metric='sum:int2', split='user:10'
))

Another example .es(index=logstash-*,split=machine.os.raw:5).add(.es(index=logstash-*,split=machine.os.raw:5))

SeriesList are applied label-wise to each other resulting in one series per split.

thomasneirynck

This is a great improvement, makes a lot of sense intuitively, 👍

The behavior, when the labels in the split-series are not equal, seems a little undefined for a couple of use-cases.

Consider two scenarios (all use logstash example data):

labels are identical but from different series

consider these two split-series , which are identical:

We cannot substract them:

Fails with:

 Error: series could not be found for label q:* > geo.dest:CN > sum(bytes)
    at /home/thomas/repos/kibana/src/core_plugins/timelion/server/lib/reduce.js:34:17
    at next (native)
    at step (/home/thomas/repos/kibana/src/core_plugins/timelion/server/lib/reduce.js:13:191)
    at /home/thomas/repos/kibana/src/core_plugins/timelion/server/lib/reduce.js:13:437
    at /home/thomas/repos/kibana/src/core_plugins/timelion/server/lib/reduce.js:13:99
    at seriesList.list.forEach (/home/thomas/repos/kibana/src/core_plugins/timelion/server/lib/reduce.js:32:7)
    at Array.forEach (native)
    at /home/thomas/repos/kibana/src/core_plugins/timelion/server/lib/reduce.js:32:23
    at next (native)
    at step (/home/thomas/repos/kibana/src/core_plugins/timelion/server/lib/reduce.js:13:191)
    at /home/thomas/repos/kibana/src/core_plugins/timelion/server/lib/reduce.js:13:361

labels are not identical

these two series are different:

but subtracting them (which you probably don't want) doesn't yield any useful info

Fails with:

server    log   [21:29:26.869] [warning][process] Error: series could not be found for label q:* > machine.os.raw:win 7 > sum(bytes)
    at /home/thomas/repos/kibana/src/core_plugins/timelion/server/lib/reduce.js:34:17
    at next (native)
    at step (/home/thomas/repos/kibana/src/core_plugins/timelion/server/lib/reduce.js:13:191)
    at /home/thomas/repos/kibana/src/core_plugins/timelion/server/lib/reduce.js:13:437
    at /home/thomas/repos/kibana/src/core_plugins/timelion/server/lib/reduce.js:13:99
    at seriesList.list.forEach (/home/thomas/repos/kibana/src/core_plugins/timelion/server/lib/reduce.js:32:7)
    at Array.forEach (native)
    at /home/thomas/repos/kibana/src/core_plugins/timelion/server/lib/reduce.js:32:23
    at next (native)
    at step (/home/thomas/repos/kibana/src/core_plugins/timelion/server/lib/reduce.js:13:191)
    at /home/thomas/repos/kibana/src/core_plugins/timelion/server/lib/reduce.js:13:361

Maybe we should only apply this:

when the length is equal
and the labels are equal (not the full series-name, but only the label)
and the label order is identical

If those cases are not met, we should surface the error. What do you think?

nreese · 2017-11-17T21:48:07Z

Why the third case (label order is identical)?

nreese · 2017-11-17T23:54:34Z

@thomasneirynck Looks like comparing the labels is a bad idea. The labels contain the metric name and query string so they don't match if you are looking at different metrics or different queries. The split label will have to be added to the seriesList so reduce can compare that label without all of the other stuff. I updated how timelion turns aggregation results into seriesList and added a split field. Then reduce does its label comparison on that field. Now queries like the one below work.

.es(index=logstash-*,split=machine.os.raw:5,q="bytes: [5000 TO 20000]")
.add(.es(index=logstash-*,split=machine.os.raw:5))

nreese · 2017-11-27T02:16:59Z

@thomasneirynck @ppisljar This PR is ready for another look.

ppisljar

LGTM, i think it gives nice capabilities, and when you type something that wouldn't work you get the notification warning.

by the way, thomas in the examples above: why would you be doing the split on two different fields, whats the use case?

thomasneirynck · 2017-12-01T18:54:54Z

@ppisljar I think it might be useful to run a calculation on two different series when the concept in the two fields are the same. say e.g. to a delta comparison between sum of two fields, split over a couple of terms. Those terms may be the same concept, e.g. zipcode, country-code, username, ...

ppisljar · 2017-12-05T06:26:09Z

@thomasneirynck i agree, but i guess you should always split on a same field ... so first serie would be sum of bytes (split on top 5 countries) and second series might be the count (again on the same split). Dont see the usecase where you would want to first split on geo.src and in then on geo.dest ... it won't give you same top5 countries, so it won't make any sense to divide bytes from US by count from china ?

… and query strings

…splitKey field

thomasneirynck · 2017-12-05T14:22:08Z

@ppisljar makes sense.that example seems not a real-world use-case after all. i thought ordering could also be determined by the terms themselves.

…seriesList with multiple series (elastic#14891) * allow seriesList with multiple series * remove junk file * fix promise bug in precision.js * remove another junk file * throw error outside of async function so it is properly handled * compare split label and not entire series label which includes metric and query strings * reduce seriesList by label field when both seriesList do not contain splitKey field * use clearer variable names * move pairwiseReduce to its own function * fix es.js test

…seriesList with multiple series (#14891) (#15413) * allow seriesList with multiple series * remove junk file * fix promise bug in precision.js * remove another junk file * throw error outside of async function so it is properly handled * compare split label and not entire series label which includes metric and query strings * reduce seriesList by label field when both seriesList do not contain splitKey field * use clearer variable names * move pairwiseReduce to its own function * fix es.js test

rbosneag · 2018-01-05T15:12:53Z

What is the first release this PR made it into? I looked everywhere and couldn't find anything like it.
Sorry for being off topic...

nreese · 2018-01-05T15:23:24Z

It will be in the next released version - 6.2

rbosneag · 2018-01-05T15:27:16Z

Thanks!

👍

…seriesList with multiple series (elastic#14891) * allow seriesList with multiple series * remove junk file * fix promise bug in precision.js * remove another junk file * throw error outside of async function so it is properly handled * compare split label and not entire series label which includes metric and query strings * reduce seriesList by label field when both seriesList do not contain splitKey field * use clearer variable names * move pairwiseReduce to its own function * fix es.js test

nreese added Feature:Timelion Timelion app and visualization release_note:enhancement v6.1.0 v7.0.0 labels Nov 10, 2017

nreese requested review from ppisljar and thomasneirynck November 10, 2017 02:00

thomasneirynck reviewed Nov 16, 2017

View reviewed changes

nreese force-pushed the multiple_series_in_series_list branch from 1fdd20d to c21fed0 Compare November 17, 2017 21:51

thomasneirynck self-requested a review November 17, 2017 23:44

nreese added v6.2.0 and removed v6.1.0 labels Nov 21, 2017

nreese force-pushed the multiple_series_in_series_list branch from ff7f068 to 5bef751 Compare November 27, 2017 01:17

ppisljar approved these changes Dec 1, 2017

View reviewed changes

thomasneirynck approved these changes Dec 1, 2017

View reviewed changes

nreese added 10 commits December 5, 2017 05:44

allow seriesList with multiple series

b3a4c43

remove junk file

0cb56d8

fix promise bug in precision.js

9e05856

remove another junk file

f079931

throw error outside of async function so it is properly handled

983a61e

compare split label and not entire series label which includes metric…

2111825

… and query strings

reduce seriesList by label field when both seriesList do not contain …

2769359

…splitKey field

use clearer variable names

14a00c1

move pairwiseReduce to its own function

160b7e8

fix es.js test

b1888df

nreese force-pushed the multiple_series_in_series_list branch from 865641d to b1888df Compare December 5, 2017 12:45

nreese merged commit d1550c1 into elastic:master Dec 5, 2017

nreese mentioned this pull request Dec 5, 2017

[6.x] [timelion] allow sum, subtract, multiply, divide functions to accept seriesList with multiple series (#14891) #15413

Merged

Navigation Menu

Repository navigation

[timelion] allow sum, subtract, multiply, divide functions to accept seriesList with multiple series - #14891

[timelion] allow sum, subtract, multiply, divide functions to accept seriesList with multiple series#14891
nreese merged 10 commits into
elastic:masterfrom
nreese:multiple_series_in_series_list

nreese commented Nov 10, 2017 •

edited

Loading

Uh oh!

thomasneirynck left a comment •

edited

Loading

Uh oh!

nreese commented Nov 17, 2017 •

edited

Loading

Uh oh!

nreese commented Nov 17, 2017 •

edited

Loading

Uh oh!

nreese commented Nov 27, 2017

Uh oh!

ppisljar left a comment

Uh oh!

thomasneirynck commented Dec 1, 2017 •

edited

Loading

Uh oh!

ppisljar commented Dec 5, 2017

Uh oh!

thomasneirynck commented Dec 5, 2017

Uh oh!

rbosneag commented Jan 5, 2018

Uh oh!

nreese commented Jan 5, 2018

Uh oh!

rbosneag commented Jan 5, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

nreese commented Nov 10, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thomasneirynck left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nreese commented Nov 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nreese commented Nov 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nreese commented Nov 27, 2017

Uh oh!

ppisljar left a comment

Choose a reason for hiding this comment

Uh oh!

thomasneirynck commented Dec 1, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ppisljar commented Dec 5, 2017

Uh oh!

thomasneirynck commented Dec 5, 2017

Uh oh!

rbosneag commented Jan 5, 2018

Uh oh!

nreese commented Jan 5, 2018

Uh oh!

rbosneag commented Jan 5, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nreese commented Nov 10, 2017 •

edited

Loading

thomasneirynck left a comment •

edited

Loading

nreese commented Nov 17, 2017 •

edited

Loading

nreese commented Nov 17, 2017 •

edited

Loading

thomasneirynck commented Dec 1, 2017 •

edited

Loading