Skip to content

[k8s Integratiion] Adding container.id as filter of Cluster Overview and removing median filter occurencies - #10893

Merged
gizas merged 3 commits into
mainfrom
fix_dashboards_apple
Aug 28, 2024
Merged

[k8s Integratiion] Adding container.id as filter of Cluster Overview and removing median filter occurencies#10893
gizas merged 3 commits into
mainfrom
fix_dashboards_apple

Conversation

@gizas

@gizas gizas commented Aug 27, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor
  • Bug

Proposed commit message

  • WHAT: Updated the Top Memory and Top CPU visualisations of Cluster Overview Dashboard in order to use the container.id filter
  • WHY: The kubernetes.container.name used in previous visualisations was not unique for all kinds of k8s resources (eg. the kubernetes.container.name is the same for all pods of same daemonset and this was creating false calculations)

Additionally we removed all median functions from or visualisations and replaced them either with Average or Last value
The updated dashboards where the median has been removed are:

  • Cronjobs
  • Deployments
  • Volume
  • Daemonsets

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

  1. Prequisite to have a local k8s cluster. Use kind and run kind create cluster
  2. Install kube-state-metrics in your k8s cluster
  3. Clone this branch
  4. Change to directory integrations/packages/kubernetes
  5. Run elastic-package stack up -d -v --version=8.15.0 in order to spin the ES locally
  6. Run docker network connect elastic-package-stack_default kind-control-plane to connect the networks of the ES and your k8s node
  7. Open the local kibana page and install a k8s integration and create a new Elastic Agent policy
  8. Apply the new manfest to your k8s cluster and once the agents are up and running navigate to your dahboards
  9. Validate the changes in the visualisations

Related issues

Screenshots

From Cluster Overview:

Screenshot 2024-08-27 at 12 31 56 PM

From Cronjobs:

Screenshot 2024-08-27 at 12 32 23 PM

@gizas
gizas requested review from a team as code owners August 27, 2024 09:46
@andrewkroh andrewkroh added Integration:kubernetes Kubernetes Team:Cloudnative-Monitoring Cloud Native Monitoring team [elastic/obs-cloudnative-monitoring] labels Aug 27, 2024
@gizas gizas changed the title adding container.id as filter and removing median as filter [k8s Integratiion] Adding container.id as filter of Cluster Overview and removing median filter occurencies Aug 27, 2024
@andrewkroh andrewkroh added the bugfix Pull request that fixes a bug issue label Aug 27, 2024
@elasticmachine

elasticmachine commented Aug 27, 2024

Copy link
Copy Markdown

🚀 Benchmarks report

Package kubernetes 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
container_logs 142857.14 111111.11 -31746.03 (-22.22%) 💔

To see the full report comment with /test benchmark fullreport

@elastic-sonarqube

elastic-sonarqube Bot commented Aug 27, 2024

Copy link
Copy Markdown

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@elasticmachine

elasticmachine commented Aug 27, 2024

Copy link
Copy Markdown

💚 Build Succeeded

History

@constanca-m

constanca-m commented Aug 28, 2024

Copy link
Copy Markdown
Contributor

Additionally we removed all median functions from or visualisations and replaced them either with Average or Last value

Why is this @gizas?

@gizas

gizas commented Aug 28, 2024

Copy link
Copy Markdown
Contributor Author

Additionally we removed all median functions from or visualisations and replaced them either with Average or Last value

Why is this @gizas?

In all of the instances where median was used was a mistake, either needed to be last value or average and we aligned it with the other fields of the visualisations. Seems that because the first choice of filter is median (when someone creates a visualisation) this was left by mistake.

Additionally, after a relevant slack thread we were pointed out that average provided better results for a customer of ours so we made the decision to prefer average.

constanca-m
constanca-m approved these changes Aug 28, 2024
View reviewed changes

@constanca-m constanca-m left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright. I always thought that median was the best one to use since it does not get affected by outliers as much as average, but I trust your judgement. Could you just link the slack thread? For future reference, it might help.

@gizas
gizas merged commit 930a301 into main Aug 28, 2024
@gizas
gizas deleted the fix_dashboards_apple branch August 28, 2024 14:54
@elasticmachine

elasticmachine commented Aug 28, 2024

Copy link
Copy Markdown

Package kubernetes - 1.66.4 containing this change is available at https://epr.elastic.co/search?package=kubernetes

@andrewkroh andrewkroh added the dashboard Relates to a Kibana dashboard bug, enhancement, or modification. label Aug 30, 2024
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
@gizas
[k8s Integratiion] Adding container.id as filter of Cluster Overview …
d2e24dc 
…and removing median filter occurencies (elastic#10893)

* adding container.id as filter and removing median as filter
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
@gizas
[k8s Integratiion] Adding container.id as filter of Cluster Overview …
5416bc5 
…and removing median filter occurencies (elastic#10893)

* adding container.id as filter and removing median as filter
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@constanca-m constanca-m constanca-m approved these changes
+1 more reviewer
@tetianakravchenko tetianakravchenko tetianakravchenko approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

bugfix Pull request that fixes a bug issue dashboard Relates to a Kibana dashboard bug, enhancement, or modification. Integration:kubernetes Kubernetes Team:Cloudnative-Monitoring Cloud Native Monitoring team [elastic/obs-cloudnative-monitoring]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Kubernetes]Overview dashboard - total cpu and memory usage are incorrect

5 participants

@gizas @elasticmachine @constanca-m @tetianakravchenko @andrewkroh