Telegraf issues with Windows multi instance processes #1827

willemdh · 2016-09-29T14:02:42Z

Bug report

I noticed in Grafana that some Windows process % Processor Time was incorrect on one server for one process. The process is called UserMonitor showed as using 100 % CPU for almost all the time.

Doublechecking the perfmon on the Windows Server showed that this process was actually only using +- 8 % CPU.

The config in Grafana:

And a query qhowing the UserMonitor process data coming in as 100%:

Relevant telegraf.conf:

  [[inputs.win_perf_counters.object]]
    ObjectName = "Process"
    Counters = ["% Processor Time","Pool Nonpaged Bytes","Pool Paged Bytes","Working Set - Private"]
    Instances = ["*"]
    Measurement = "win_processes"
    IncludeTotal=true

System info:

Telegraf 1.0.0
Windows Server 2012 R2

The text was updated successfully, but these errors were encountered:

willemdh · 2016-10-11T10:02:40Z

I'm seeing more abnormal Windows process results. One of my colleagues pointed out for example that Telegraf.exe was using more then 10 % CPU on some servers in Grafana. I doublechecked this on the server and it was only using 1 % CPU.
Is someone able to reproduce these issues?

francoishill · 2017-01-04T16:03:52Z

When there are multiple of the same processes it seems to only pick one of them, my config is also as follows:

  [[inputs.win_perf_counters.object]]
    # Process metrics, in this case for IIS only
    ObjectName = "Process"
    Counters = ["% Processor Time","Handle Count","Private Bytes","Thread Count","Virtual Bytes","Working Set"]
    # Instances = ["gitkraken"]
    Instances = ["*"]
    Measurement = "win_proc"
    IncludeTotal=true

willemdh · 2017-01-11T12:22:18Z

Posted this problem on Google forums, but no answer. https://groups.google.com/forum/#!topic/influxdb/k2njOfflGmo

It seems there is no way atm to monitor processes which have multiple instances. Although perfMon is able to show data for each process instance, Telegraf fails to separate processes with the same name..

Changing the title of this issue for more clarity.

sparrc · 2017-01-11T12:42:10Z

That does appear to be the issue. PRs would be appreciated if anyone could find a solution for tagging the process with the Windows instance #. In unix we would call this a PID (Process ID), not sure what it's referred to in the Windows world.

francoishill · 2017-01-11T13:56:39Z

@sparrc, in Windows it is also called PID.

discoduck2x · 2017-01-19T14:25:50Z

+1 on getting this solved !

bullshit · 2017-01-31T08:51:10Z

Hi, i managed to debug it on windows and it turned out that the filledBuf var after UTF16PtrToString(c.SzName) contains only a view characters. For example, i configured my Instance to be chrome#1 and c.Szname was ch and thats the reason why it never will be added,
because with metric.instance == s the name must match exactly.
For now i don't get it if the Problem is in the second PdhGetFormattedCounterArrayDouble call or if the UTF16PtrToString and with the syscalls and pointer casting i think i won't get any deeper information on it.

bullshit · 2017-03-20T15:03:51Z

Hi, could anybody test my changes. i really would appreciate your help #2352 https://github.com/bullshit/telegraf/tree/fix-1827-win_perf_counters

lucadistefano · 2017-03-23T20:03:41Z

@bullshit , tried the fix and seems to work:
this is the output of

[[inputs.win_perf_counters]]
	PrintValid=true

	[[inputs.win_perf_counters.object]]
		ObjectName = "Process"
		Counters = ["Handle Count"]
		Instances = ["spoolsv","svchost","svchost#1","svchost#2"]
		Measurement = "win_proc1"

C:\Program Files\Telegraf>ptelegraf.exe --debug --config test-instancedash.conf
2017-03-23T19:46:51Z D! Attempting connection to output: influxdb
2017-03-23T19:46:51Z D! Successfully connected to output: influxdb
2017-03-23T19:46:51Z I! Starting Telegraf (version dev-92-g616b66f)
2017-03-23T19:46:51Z I! Loaded outputs: influxdb
2017-03-23T19:46:51Z I! Loaded inputs: inputs.win_perf_counters
2017-03-23T19:46:51Z I! Tags enabled: host=WIN-MONTEST
2017-03-23T19:46:51Z I! Agent Config: Interval:5s, Quiet:false, Hostname:"WIN-MONTEST", Flush Interval:10s
Valid: \Process(spoolsv)\Handle Count
Valid: \Process(svchost)\Handle Count
Valid: \Process(svchost#1)\Handle Count
Valid: \Process(svchost#2)\Handle Count
win_proc1,instance=spoolsv,objectname=Process,host=WIN-MONTEST Handle_Count=368 1490298415000000000

win_proc1,objectname=Process,host=WIN-MONTEST,instance=svchost Handle_Count=458 1490298415000000000

win_proc1,instance=svchost#1,objectname=Process,host=WIN-MONTEST Handle_Count=359 1490298415000000000

win_proc1,instance=svchost#2,objectname=Process,host=WIN-MONTEST Handle_Count=613 1490298415000000000

win_proc1,instance=spoolsv,objectname=Process,host=WIN-MONTEST Handle_Count=368 1490298420000000000

win_proc1,instance=svchost,objectname=Process,host=WIN-MONTEST Handle_Count=458 1490298420000000000

win_proc1,objectname=Process,host=WIN-MONTEST,instance=svchost#1 Handle_Count=359 1490298420000000000

win_proc1,instance=svchost#2,objectname=Process,host=WIN-MONTEST Handle_Count=613 1490298420000000000

win_proc1,instance=svchost,objectname=Process,host=WIN-MONTEST Handle_Count=458 1490298425000000000

win_proc1,objectname=Process,host=WIN-MONTEST,instance=svchost#1 Handle_Count=359 1490298425000000000

win_proc1,instance=svchost#2,objectname=Process,host=WIN-MONTEST Handle_Count=615 1490298425000000000

win_proc1,instance=spoolsv,objectname=Process,host=WIN-MONTEST Handle_Count=368 1490298425000000000

2017-03-23T19:47:05Z D! Output [influxdb] buffer fullness: 12 / 1000 metrics.
2017-03-23T19:47:05Z D! Output [influxdb] wrote batch of 12 metrics in 5.0058ms

by the way the following issues may be related to this one: #2210, #1546

thanks,
Luca

danielnelson · 2017-07-05T19:43:46Z

Reopening based on #2879 (comment)

barbarajoost · 2018-11-15T10:50:34Z

It seems that the problem still exists. I have this issue on a Server 2008 R2 with Telegraf 1.8.

Windows sees

process
process#1
process#2
process#3

but in Grafana I found only process.

danielnelson · 2018-11-16T19:23:50Z

This issue should be mostly solved by enabling wildcard expansion:

[[inputs.win_perf_counters]]
  UseWildcardsExpansion = true

  [[inputs.win_perf_counters.object]]
    ObjectName = "Process"
    Instances = ["chrome*"]
    Counters = [
      "% Processor Time"
    ]

This option has a side effect of localizing the counter names, which is usually unwanted, but is discussed on a different issue. #4280

sparrc added bug unexpected problem or unintended behavior help wanted Request for community participation, code, contribution platform/windows labels Sep 29, 2016

sparrc added this to the Future Milestone milestone Sep 29, 2016

willemdh changed the title ~~Telegraf Windows process CPU usage sometimes incorrect?~~ Telegraf issues with multi instance processes? Jan 11, 2017

willemdh changed the title ~~Telegraf issues with multi instance processes?~~ Telegraf issues with Windows multi instance processes? Jan 11, 2017

sparrc changed the title ~~Telegraf issues with Windows multi instance processes?~~ Telegraf issues with Windows multi instance processes Jan 11, 2017

bullshit added a commit to bullshit/telegraf that referenced this issue Feb 1, 2017

Fix influxdata#1827 - multi instances in win perf counters

d71ad94

bullshit mentioned this issue Feb 1, 2017

Fix #1827 - multi instances in win_perf_counters #2352

Merged

3 tasks

willemdh mentioned this issue Mar 21, 2017

win_perf_counters ignore instances with '#' in instance name #2557

Closed

danielnelson closed this as completed in #2352 Mar 22, 2017

This was referenced Mar 23, 2017

WinPerf Input Plugin Problems with multiple process instances #2210

Closed

Escaping # in metric instance name in telegraf.conf #1546

Closed

marefr mentioned this issue Jun 2, 2017

Cannot use instance wildcard in win_perf_counters #2879

Closed

danielnelson reopened this Jul 5, 2017

amitkumar912 mentioned this issue Oct 2, 2017

Cannot escaping backslash, spaces, special chars in exec input on Windows #2230

Closed

vlastahajek mentioned this issue May 23, 2018

Fix Cannot use instance wildcard in win_perf_counters #2879 and others #4189

Merged

3 tasks

danielnelson closed this as completed Nov 16, 2018

SimonMcN mentioned this issue Jan 9, 2020

Multiple processes with the same name, add counter as tag in win_perf_counters #6881

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Telegraf issues with Windows multi instance processes #1827

Telegraf issues with Windows multi instance processes #1827

willemdh commented Sep 29, 2016

willemdh commented Oct 11, 2016

francoishill commented Jan 4, 2017

willemdh commented Jan 11, 2017 •

edited

Loading

sparrc commented Jan 11, 2017 •

edited

Loading

francoishill commented Jan 11, 2017 •

edited

Loading

discoduck2x commented Jan 19, 2017

bullshit commented Jan 31, 2017

bullshit commented Mar 20, 2017 •

edited

Loading

lucadistefano commented Mar 23, 2017

danielnelson commented Jul 5, 2017

barbarajoost commented Nov 15, 2018

danielnelson commented Nov 16, 2018

Telegraf issues with Windows multi instance processes #1827

Telegraf issues with Windows multi instance processes #1827

Comments

willemdh commented Sep 29, 2016

Bug report

Relevant telegraf.conf:

System info:

willemdh commented Oct 11, 2016

francoishill commented Jan 4, 2017

willemdh commented Jan 11, 2017 • edited Loading

sparrc commented Jan 11, 2017 • edited Loading

francoishill commented Jan 11, 2017 • edited Loading

discoduck2x commented Jan 19, 2017

bullshit commented Jan 31, 2017

bullshit commented Mar 20, 2017 • edited Loading

lucadistefano commented Mar 23, 2017

danielnelson commented Jul 5, 2017

barbarajoost commented Nov 15, 2018

danielnelson commented Nov 16, 2018

willemdh commented Jan 11, 2017 •

edited

Loading

sparrc commented Jan 11, 2017 •

edited

Loading

francoishill commented Jan 11, 2017 •

edited

Loading

bullshit commented Mar 20, 2017 •

edited

Loading