Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Input: DiskIO] Deleted devices are not deleted from the diskInfoCache #6345

Closed
johnsudaar opened this issue Sep 4, 2019 · 2 comments · Fixed by #6835
Closed

[Input: DiskIO] Deleted devices are not deleted from the diskInfoCache #6345

johnsudaar opened this issue Sep 4, 2019 · 2 comments · Fixed by #6835
Labels
bug unexpected problem or unintended behavior
Milestone

Comments

@johnsudaar
Copy link
Contributor

johnsudaar commented Sep 4, 2019

Relevant telegraf.conf:

[[inputs.diskio]]
  name_templates = ["$ID_FS_LABEL","$DM_VG_NAME/$DM_LV_NAME"]

System info:

$ telegraf --version
Telegraf 1.10.2 (git: HEAD 3303f5c3)
 uname -ar
Linux [REDACTED] 4.4.0-151-generic #178-Ubuntu SMP Tue Jun 11 08:30:22 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Steps to reproduce:

  1. Create a Logival Volume (using LVM) with the name lv-a
  2. Let telegraf gather some points
  3. Delete the LV lv_a
  4. Create a LV with the name lv-b
  5. Let telegraf gather some points

Expected behavior:

The new points should have the tag: vg-1/lv-b

Actual behavior:

The new points will have the tag: vg-1/lv-a

Additional info:

This is because linux reuse DM ID. Let's say that the first LV had the DM ID 39 its path would be /dev/dm-39.

When deleting the LV, this dm ID will be freed. Thus linux can (and will) reuse it when we'll create another LV.

However since telegraf use a cache for the device infos (https://github.com/influxdata/telegraf/blob/master/plugins/inputs/diskio/diskio_linux.go#L33). It will always return the lv-a informations.

A solution would be to track DM deletions (see if something is in the cache but not in the device list) and remove it from the cache.

@danielnelson danielnelson added the bug unexpected problem or unintended behavior label Sep 5, 2019
@danielnelson
Copy link
Contributor

I wonder if we could use file mtime or similar to invalidate the cache? It may even be worth removing the cache altogether, originally it was added because we ran udevadm but now we are just reading a file.

@johnsudaar
Copy link
Contributor Author

I like the idea of using Mtim, i'll submit a PR. Thanks for taking a look at it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug unexpected problem or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants