Repository navigation
Average _count per interval (eg minute) metric #4646
Description
Activity
You can do this with a metric visualisation and the average calculation on the field.
Reacted by Vladislav@markwalkom on what field should I calculate the average on? I am not sure if you understood my question. I would like a metric that would show documents indexed per minute.
My index mapping:
{ "url": "string", "method": "string", "body": "string" }So if I selected
Last 15 minutesin the upper right corner and there were a total of 100 documents indexed over the past 15 minutes, the metric should show6.6 (100 / 15)I understand, and as I said there is an average metric you can use to provide this.
As to what field you want to calculate that on, that is up to you.Here's an example of it in a visualisation;
Then that average is calculated on the timespan you pick.
Reacted by Vladislavhttps://www.elastic.co/guide/en/kibana/current/metric-chart.html also has information.
But that is not what I need. I need a Count metric, divided by the amount of minutes in a selected period. That would give me the amount of documents indexed per minute. It is not an average of any field.
@spalger, @rashidkpc, @palecur or @lukasolson, please comment if you understand the issue I am trying to describe. Thanks.
Reacted by Kostya Kostyushko, tf42src, RJ Garcia and VladislavYou want the average _count per minute for the selected time period.
I understand what you're trying to accomplish, currently there is no way to accomplish this. This probably falls under "metric math", but you also want it pegged to the time picker, which is a bit more challenging.
Reacted by Vladislav- changed the title
[-]Average _count per interval metric[/-][+]Average _count per interval (eg minute) metric[/+]on Aug 12, 2015 Now that I've renamed the title, this probably isn't that hard to implement if it was implemented on the _count metric alone.
Really you'd want a "scale to X" on any _count metric where it takes the number and divide, or multiplies it such that it reflects the _count per X.
👍 I would find this very useful, on top of queries this can offer a lot of flexibility
👍
My use case (in brief: "me too"):
I already have a Kibana 4 metric visualization that displays a count aggregation: that is, the number of Elasticsearch documents in the current time range (in the specified index pattern).
In my case, each Elasticsearch document represents a transaction. So the count represents the number of transactions in the time range. That's useful (thanks!). So far, so good.
Now, independent of the time range, I want metric visualizations that display "average count per second" and "average count per day": "average count per [arbitrary interval]" would be nice.
So that I can compare, for example, the average number of transactions per second (TPS) in the last 30 days with the average TPS yesterday (or any other arbitrary time ranges, of same or different duration); same dashboard, same visualization, just changing the time range.
Reacted by Ferran Muñoz, Kostya Kostyushko, ZhangRui and Danilo TeodoroSimilar use case, different visualization: I wanted to create a line chart of transactions per second (TPS), which, as described in my previous comment, is a count of documents (in the specified index pattern) per second.
Aware that angels would be standing back, lighting cigarettes, and placing bets on my likelihood of survival, I created a visualization with the following details:
with a time range of... 30 days. Yes, understandably: kaboom. My Firefox browser hung, then crashed.
With a much smaller time range - a manageable number of buckets - the chart displays with no problem. But I want to chart TPS across arbitrarily wide time ranges, which is where an "average count per second" would be useful. I understand that, at a time range of a few seconds or less, TPS becomes less useful, but I'm okay with that, as long as this combination of metric and time range doesn't crash my browser (or Kibana).
That was actually my second attempt. On my first attempt - with the same time range of 30 days - I specified an interval of "Second" rather "Auto", and omitted the JSON (which I copied from #4459 ). On that attempt, Kibana displayed an information icon with the tooltip "This interval creates too many buckets...", and adjusted the interval; hence my second attempt that "manually" constrained the interval to 1s.
Reacted by 复读机 and ZhangRuiES2.0 pipeline aggregations should be able to help but they are no yet supported on Kibana I think.
67 remaining items
this should be somehow possible:
so add a date histogram, and for interval select minute. thats it.
if you zoom out to a long time range (lets say one year) you wont see a bar for every minute, but it will still represent per minute value. So lets say that you get one bar per day, it will still show you your per minute count (averaged)
Is there a question about if the community wants this feature or not? Seems like there are loads of upvotes and +1. I spent an hour trying to make this work until I finally came across this feature request. I surprised it wasn't already part of it given Splunk does it rather handily.
Is there any clear solution for creating graph/chart for Total Transactions/sec (In aggregated way for all transactions), similar to graph in JMeter, LoadRunner or other performance testing tools? Please don't mix it with chart which shows the count for a particular instant.
I could not implement if there any solution above.
@bharat129 you're probably looking for Timelion
Is there a fix for this issue yet? I want to graph bandwidth as well.
It's pretty easy to do by taking the sum and dividing it by 60 with a one minute interval but that isn't usable long term because it will create too many buckets.
Timelion's scale.interval doesn't create the correct graph when zoomed up.
It is a bit frustrating to see that after people asking for a (imho basic) feature like this for 2 years there still doesn't appear to be a real solution.
Logstash comes with a netflow collector, you'd expect Kibana being able to do something useful with that data ;)
Looks like this thread really covers two separate enhancement requests.
- An update to the
Metric Visualizationthat displays the metric value as a ratio (metric value divided by the time picker duration and specified interval). For example, the Metric Visualization is displaying the Sum Metric Aggregation on the field "bytes". Under Options, the user has configured an interval of "1 minute". The time picker duration is 15 minutes. The displayed metric value issum/15and the default label is "(sum of bytes) per minute". This sounds like the original intent of @n3ziniuka5's request and @GrahamHannington use case. - An update to the
Date Histogrambucket aggregation that allows for setting a metric interval that uses a scripted metric to turn the metric value for each bucket into a ratio (metric value divided by the bucket size and interval). This covers the functionality specified by @DeeeFOX and @pagenbag.
- An update to the
@nreese, I think #1 can already be done using the "Average Bucket" aggregation. You can pick a sub bucket aggregation of "Date Histogram" with an interval of "1 minute" and then a sub metric aggregation of "Sum" and field "bytes".
It does like there is a bug with that, though. It isn't handling the scaling correctly when the time range is too big.
#2 also might be possible with having an "Average Bucket" metric aggregation and then a normal "Date Histogram" bucket. So the metric would be an average for the smaller timespan but graphed using the larger timespan.
I don't think the second example proposed by @nreese really works as intended, due to scaling of the time period withing the date historgram on the Y-axis.
Shouldn't this be done in Kibana, rather than in ES itself?
I'm going to call X-interval the date histogram interval on the X-axis, and Y-interval the date histogram interval in the Y-axis subagg.
The proposed behaviour does something like this: for each X-interval, get all Y-interval values from ES, then average them to present a single number (average of all per-Y-interval values) for the X-interval. If there would be too many buckets created for the X-interval (ie X-interval * Y-interval is too large), autoscale Y-interval to reduce the number of buckets.
Whereas the desired behaviour would be: get a single number for each X-interval, then - without performing another date histogram agg on the X-interval buckets, divide the figure by Y-interval.
For example, as in the second gif above: X-interval is 1m, Y-interval is 1s, full period is 15m.
Not so good: 15 buckets for the X-interval is fine, but 60 * 15 = 900 is too many, so Y-interval is scaled to 5s. The resulting numbers show "average count per 5 seconds", which is calculated as something like "average of per-5s-average" for each X bucket.
Better: divide the total for the X-interval by the Y-interval, producing a desired-accuracy average without relying on ES further nor allowing the too many buckets issue to arise.
Reacted by Jim Maskelony+1
I'd like to be able to show avg. number of requests per hour of the day, and then if possible compare year-over-year (or also do month-by-month comparisons).
I have an index in which every document represents a request. I want a metric that would show requests per minute to measure how busy the site is.
Currently I have a count metric with query
@timestamp: [now-1m TO now]which shows the amount of requests received during the last minute.However, I want to use Kibana's built in duration(top-right corner). So if I select
Last 15 minutes, it should get a count of documents indexed during the past 15 minutes and divide that by 15. Similarly, if I selectLast 1 hour, it should divide it by 60.Is it possible to accomplish that with the current stable version of Kibana? If not, do you think it would be a good feature to add?