Repository navigation
Monitoring tab shows wrong CPU utilization #9377
Description
Activity
I tried running metricbeat with monitoring turned on and compared the CPU utilization chart with the output of
top. The numbers seem to be in the same ballpark for me.@simitt Could you include the steps you took to cause this on your end? Perhaps that'll help me reproduce it on my end. Thanks!
I used 6.5.2 for testing and basically didn't do anything special. For generating load on the beat itself, I set
period: 10msinmodules.d/system.ymland inmodules.d/golang.yml.On my local machine with 4 CPUs this results in:
htop (values between 95 and 160%)
monitoring:
I also used it in a cloud gcp setup, with a small machine of 1CPU. It doesn't seem to not normalize the amount of used CPU to the available CPU for my box.
htop
monitoring:
That's perfect, @simitt. I'm able to reproduce it locally now and can investigate further. Thanks!
Reacted by simittI'm not sure what's going on with the calculation for this metric in the UI code (yet) but I started by looking at the raw documents in
.monitoring-beats-6-*to try and narrow down where the problem might be.The metric is being calculated from the
beats_stats.metrics.beat.cpu.total.valuefield, which is a counter metric and whose units arems.When the time picker in the UI is set to "Last 1 hour", the bucket size of each X-axis bucket gets automatically set to
30s. So I setup adate_histogramaggregation with anintervalof30s. Since we're dealing with a counter metric, I asked for themaxof the counter metric field within each date histogram bucket. Also, for the same reason, I asked for a derivative (difference) between adjacent date histogram bucket values for the metric, so we get to see the CPU utilization (in ms) just for each 30s time period. Finally, for ease of inspection, I restricted the time range to the last couple of minutes. Here's the final query with its results:Query
POST .monitoring-beats-6-*/_search?filter_path=aggregations { "size": 0, "query": { "range": { "timestamp": { "gte": "now-2m/m", "lte": "now" } } }, "aggs": { "timeseries": { "date_histogram": { "field": "timestamp", "interval": "30s" }, "aggs": { "total_cpu_ms": { "max": { "field": "beats_stats.metrics.beat.cpu.total.value" } }, "rate": { "derivative": { "buckets_path": "total_cpu_ms" } } } } } }
Results
{ "aggregations" : { "timeseries" : { "buckets" : [ { "key_as_string" : "2018-12-11T18:36:00.000Z", "key" : 1544553360000, "doc_count" : 4, "total_cpu_ms" : { "value" : 6787848.0 } }, { "key_as_string" : "2018-12-11T18:36:30.000Z", "key" : 1544553390000, "doc_count" : 3, "total_cpu_ms" : { "value" : 6857067.0 }, "rate" : { "value" : 69219.0 } }, { "key_as_string" : "2018-12-11T18:37:00.000Z", "key" : 1544553420000, "doc_count" : 4, "total_cpu_ms" : { "value" : 6925499.0 }, "rate" : { "value" : 68432.0 } }, { "key_as_string" : "2018-12-11T18:37:30.000Z", "key" : 1544553450000, "doc_count" : 3, "total_cpu_ms" : { "value" : 6994194.0 }, "rate" : { "value" : 68695.0 } } ] } } }If you take the
rate.valuefrom any date histogram bucket and divide it by the bucket size (in the same unit,ms), we get the % of CPU utilized during that date histogram bucket (i.e. 30-second period). For example, if you take the very last bucket in the above results, you get the % CPU utilization for that bucket =68695.0 / (30 * 1000) = 2.28 = 228%. At this point, when I flipped over to myhtopoutput, I see something very close to it:In conclusion, it appears that the data in the
.monitoring-beats-6-*docs is good; we are just not performing the right calculation in the UI code.FYI @chrisronline, since this appears to be a UI bug. Let me know if you want a separate issue in the Kibana repo but there's information in this issue that you might want to look at as well.
Thank you for the detailed analyzation! We do have the same problem for the CPU utilization in the APM monitoring. Let me know if I should create a separate issue for that (but I assume it is the same bug).
It's the same bug, this issue should suffice for both.
@chrisronline I think this issue can be closed now, as it was resolved via elastic/kibana#27416?
Yup. Closing
Hey @chrisronline, in your comment above, you said:
It seems the math is just wrong here: elastic/kibana:x-pack/plugins/monitoring/server/lib/metrics/beats/classes.js@master#L155 and elastic/kibana:x-pack/plugins/monitoring/server/lib/metrics/apm/classes.js@master#L72
Specifically, you linked to beats and apm code.
However, looking at the PR that resolved this issue, it appears only the beats code was fixed. Is there a corresponding PR for the APM code as well?
@ycombinator There is not. Good catch. Filed elastic/kibana#33025
Excerpt from my comment above:
I also used it in a cloud gcp setup, with a small machine of 1CPU. It doesn't seem to not normalize the amount of used CPU to the available CPU for my box.
@ycombinator I assumed the CPU usage is supposed to be shown relative to the CPU ticks assigned to the container and not relative to the system CPU. I don't think that's the case though, could you please confirm.
Closing as the fix landed in elastic/kibana#37230
Testing with
metricbeat 6.5.1I see a very low CPU utilization, although the system is running on ~80% on all cores.