feat: Add Disk Metrics #9344

R3hankhan123 · 2025-07-15T15:11:51Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR implements the previously missing disk metrics:

container_fs_inodes_free
container_fs_inodes_total
container_fs_limit_bytes
container_fs_usage_bytes

Which issue(s) this PR fixes:

contributes towards #9125

Special notes for your reviewer:

None

Does this PR introduce a user-facing change?

No

Added disk metrics (`container_fs_inodes_free`, `container_fs_inodes_total`, `container_fs_limit_bytes`, `container_fs_usage_bytes`)

openshift-ci · 2025-07-15T15:12:04Z

Hi @R3hankhan123. Thanks for your PR.

I'm waiting for a cri-o member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

internal/lib/stats/disk_metrics.go

bitoku · 2025-07-15T17:09:46Z

Thanks @R3hankhan123 . Can you add e2e tests in test/cri-metrics.bats?

R3hankhan123 · 2025-07-16T05:09:57Z

Thanks @R3hankhan123 . Can you add e2e tests in test/cri-metrics.bats?

Done @bitoku

codecov · 2025-07-16T08:30:45Z

Codecov Report

❌ Patch coverage is 69.16667% with 37 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.52%. Comparing base (6ab8d26) to head (4907339).
⚠️ Report is 58 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9344      +/-   ##
==========================================
- Coverage   67.11%   66.52%   -0.59%     
==========================================
  Files         202      203       +1     
  Lines       27900    27994      +94     
==========================================
- Hits        18725    18623     -102     
- Misses       7603     7789     +186     
- Partials     1572     1582      +10

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

bitoku · 2025-07-16T10:09:51Z

/ok-to-test

R3hankhan123 · 2025-07-16T14:04:04Z

@bitoku can you help me debug the prow tests, i see that container is being exited but can't find the reason why exactly its exiting

bitoku · 2025-07-16T14:09:41Z

@R3hankhan123
See the path of the URL to know where the log is.
For integration tests
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/cri-o_cri-o/9344/pull-ci-cri-o-cri-o-main-ci-fedora-integration/1945473469558820864/artifacts/fedora-integration/cri-o-gather/artifacts/testout.txt

I think other e2e tests come from infra issues, not because of this change.

R3hankhan123 · 2025-07-16T14:34:16Z

@bitoku , looks like port 9090 is being used when trying to run disk metrics but its already in use, should i kill the process that is using the port (ie memory metrics crio) wdyt

internal/lib/stats/disk_metrics.go

bitoku · 2025-07-16T14:42:43Z

@R3hankhan123 You can use free_port.

cri-o/test/conmon.bats

Lines 17 to 20 in 7c127d3

    
           port=$(free_port) 
        
           CONTAINER_ENABLE_METRICS=true \ 
        
           	CONTAINER_METRICS_PORT=$port \ 
        
           	start_crio

internal/oci/runtime_oci_linux.go

bitoku

Just out of curiosity. Do you happen to generate the code with AI?

test/cri-metrics.bats

R3hankhan123 · 2025-08-13T13:41:21Z

Just out of curiosity. Do you happen to generate the code with AI?

Only for the test cases(not good at writing bat files and shell scripts 🥲😅, also its a new thing for me)

openshift-ci · 2025-08-13T14:42:23Z

@R3hankhan123: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/ci-e2e-evented-pleg	`3c601f7`	link	false	`/test ci-e2e-evented-pleg`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

R3hankhan123 · 2025-08-18T07:05:07Z

Hi @bitoku can you retrigger [integration / critest / conmon-rs / runc / amd64 (pull_request)], please and thank you

R3hankhan123 · 2025-08-18T12:33:59Z

Hi @bitoku any more suggestions you have that I need to address, please and thank you

bitoku

/lgtm

bitoku · 2025-08-18T17:06:41Z

internal/lib/stats/descriptors.go

+	containerFsInodesFree = &types.MetricDescriptor{
+		Name:      "container_fs_inodes_free",
+		Help:      "Number of free inodes.",
+		LabelKeys: baseLabelKeys,
+	}


we could add extra labels, but I found general discrepancies in labels between crio and cadvisor. we can ignore it for now.
https://github.com/google/cadvisor/blob/5adb1c3bb38b4c5d50b31f39faf3214a44ae479b/metrics/prometheus.go#L537

bitoku · 2025-08-18T17:07:43Z

@cri-o/cri-o-maintainers PTAL

R3hankhan123 · 2025-08-19T12:36:03Z

Thanks @bitoku for the review, once the PR is approved I will start implementing the DISK IO metrics also☺️

R3hankhan123 · 2025-08-25T14:35:36Z

Hi @cri-o/cri-o-maintainers PTAL when you get a chance 🙏. Thanks!

Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com>

R3hankhan123 · 2025-08-28T06:02:20Z

@bitoku rebasing removed the lgtm tag😅

bitoku · 2025-08-29T13:45:48Z

/lgtm

github-actions · 2025-09-29T00:03:41Z

A friendly reminder that this PR had no activity for 30 days.

bitoku · 2025-09-30T13:39:24Z

/remove-lifecycle stale

R3hankhan123 requested a review from mrunalp as a code owner July 15, 2025 15:11

openshift-ci bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Jul 15, 2025

openshift-ci bot requested review from hasan4791 and littlejawa July 15, 2025 15:12

openshift-ci bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 15, 2025

haircommander reviewed Jul 15, 2025

View reviewed changes

internal/lib/stats/disk_metrics.go Outdated Show resolved Hide resolved

R3hankhan123 marked this pull request as draft July 15, 2025 16:21

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 15, 2025

R3hankhan123 force-pushed the disk-metrics branch from 7ba0ec1 to 9df95df Compare July 15, 2025 16:24

R3hankhan123 marked this pull request as ready for review July 15, 2025 16:24

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 15, 2025

openshift-ci bot requested review from bitoku and klihub July 15, 2025 16:25

R3hankhan123 force-pushed the disk-metrics branch from 9df95df to 2a81e7d Compare July 16, 2025 05:09

R3hankhan123 force-pushed the disk-metrics branch from 2a81e7d to bf0ebff Compare July 16, 2025 08:59

openshift-ci bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 16, 2025

R3hankhan123 force-pushed the disk-metrics branch from bf0ebff to 0bd729d Compare July 16, 2025 13:19

bitoku reviewed Jul 16, 2025

View reviewed changes

internal/lib/stats/disk_metrics.go Outdated Show resolved Hide resolved

R3hankhan123 requested a review from bitoku August 12, 2025 17:54

bitoku reviewed Aug 13, 2025

View reviewed changes

internal/oci/runtime_oci_linux.go Outdated Show resolved Hide resolved

bitoku reviewed Aug 13, 2025

View reviewed changes

test/cri-metrics.bats Outdated Show resolved Hide resolved

test/cri-metrics.bats Outdated Show resolved Hide resolved

R3hankhan123 force-pushed the disk-metrics branch from fd41dec to 2a11a18 Compare August 13, 2025 14:09

R3hankhan123 force-pushed the disk-metrics branch from 2a11a18 to cf93109 Compare August 13, 2025 14:48

R3hankhan123 requested a review from bitoku August 18, 2025 07:03

bitoku reviewed Aug 18, 2025

View reviewed changes

openshift-ci bot assigned bitoku Aug 18, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 18, 2025

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 27, 2025

R3hankhan123 added 3 commits August 28, 2025 11:30

feat: Added Disk Metrics

62ec920

Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com>

Extended ContainerStats to include disk metrics

81002c7

Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com>

Extend Disk Metrics for other filesystems

4907339

Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com>

R3hankhan123 force-pushed the disk-metrics branch from cf93109 to 4907339 Compare August 28, 2025 06:01

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Aug 28, 2025

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 28, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 29, 2025

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 29, 2025

openshift-ci bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 30, 2025

feat: Add Disk Metrics #9344

Are you sure you want to change the base?

feat: Add Disk Metrics #9344

Uh oh!

Conversation

R3hankhan123 commented Jul 15, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci bot commented Jul 15, 2025

Uh oh!

Uh oh!

bitoku commented Jul 15, 2025

Uh oh!

R3hankhan123 commented Jul 16, 2025

Uh oh!

codecov bot commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bitoku commented Jul 16, 2025

Uh oh!

R3hankhan123 commented Jul 16, 2025

Uh oh!

bitoku commented Jul 16, 2025

Uh oh!

R3hankhan123 commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

bitoku commented Jul 16, 2025

Uh oh!

Uh oh!

bitoku left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

R3hankhan123 commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

R3hankhan123 commented Aug 18, 2025

Uh oh!

R3hankhan123 commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bitoku left a comment

Choose a reason for hiding this comment

Uh oh!

bitoku Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

bitoku commented Aug 18, 2025

Uh oh!

R3hankhan123 commented Aug 19, 2025

Uh oh!

R3hankhan123 commented Aug 25, 2025

Uh oh!

R3hankhan123 commented Aug 28, 2025

Uh oh!

bitoku commented Aug 29, 2025

Uh oh!

github-actions bot commented Sep 29, 2025

Uh oh!

bitoku commented Sep 30, 2025

Uh oh!

Uh oh!

R3hankhan123 commented Jul 15, 2025 •

edited by openshift-ci bot

Loading

codecov bot commented Jul 16, 2025 •

edited

Loading

R3hankhan123 commented Jul 16, 2025 •

edited

Loading

bitoku left a comment •

edited

Loading

R3hankhan123 commented Aug 13, 2025 •

edited

Loading

openshift-ci bot commented Aug 13, 2025 •

edited

Loading

R3hankhan123 commented Aug 18, 2025 •

edited

Loading