OCPBUGS-61220: internal/cgmgr: fix CPU weight setting for cgroup v2 #9456

sohankunkerkar · 2025-09-05T20:53:47Z

It resolves Pod InPlace Resize Container test failure where cpu.weight file was empty for Guaranteed QoS pods with restartable init containers. Sometimes, systemd fails to convert CPUShares → CPUWeight for this specific scenario, likely due to how it handles complex pod hierarchies with concurrent containers, but the exact internal mechanism is unclear. This acts more as a safeguard in case systemd doesn’t set the value.

What type of PR is this?

/kind bug
/kind flake

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-61220

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Use systemd property as fallback for cpu weight

openshift-ci-robot · 2025-09-05T20:53:53Z

@sohankunkerkar: This pull request references Jira Issue OCPBUGS-61220, which is invalid:

expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

It resolves Pod InPlace Resize Container test failure where cpu.weight file was empty for Guaranteed QoS pods with restartable init containers. Sometimes, systemd fails to convert CPUShares → CPUWeight for this specific scenario, likely due to how it handles complex pod hierarchies with concurrent containers, but the exact internal mechanism is unclear. This acts more as a safeguard in case systemd doesn’t set the value.

What type of PR is this?

/kind bug
/kind flake

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-61220

Special notes for your reviewer:

Does this PR introduce a user-facing change?
Use systemd property as fallback for cpu weight

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2025-09-05T20:53:54Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2025-09-05T20:53:59Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sohankunkerkar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sohankunkerkar]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

codecov · 2025-09-05T22:24:42Z

Codecov Report

❌ Patch coverage is 85.71429% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.05%. Comparing base (d3b6c3c) to head (2172b77).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9456      +/-   ##
==========================================
+ Coverage   66.96%   67.05%   +0.09%     
==========================================
  Files         202      202              
  Lines       27997    28016      +19     
==========================================
+ Hits        18747    18786      +39     
+ Misses       7673     7656      -17     
+ Partials     1577     1574       -3

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

openshift-ci-robot · 2025-09-05T23:20:26Z

@sohankunkerkar: This pull request references Jira Issue OCPBUGS-61220. The bug has been updated to no longer refer to the pull request using the external bug tracker. All external bug links have been closed. The bug has been moved to the NEW state.

In response to this:

It resolves Pod InPlace Resize Container test failure where cpu.weight file was empty for Guaranteed QoS pods with restartable init containers. Sometimes, systemd fails to convert CPUShares → CPUWeight for this specific scenario, likely due to how it handles complex pod hierarchies with concurrent containers, but the exact internal mechanism is unclear. This acts more as a safeguard in case systemd doesn’t set the value.

What type of PR is this?

/kind bug
/kind flake

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-61220

Special notes for your reviewer:

Does this PR introduce a user-facing change?
Use systemd property as fallback for cpu weight

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-09-07T01:14:59Z

@sohankunkerkar: This pull request references Jira Issue OCPBUGS-61220, which is invalid:

expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

It resolves Pod InPlace Resize Container test failure where cpu.weight file was empty for Guaranteed QoS pods with restartable init containers. Sometimes, systemd fails to convert CPUShares → CPUWeight for this specific scenario, likely due to how it handles complex pod hierarchies with concurrent containers, but the exact internal mechanism is unclear. This acts more as a safeguard in case systemd doesn’t set the value.

What type of PR is this?

/kind bug
/kind flake

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-61220

Special notes for your reviewer:

Does this PR introduce a user-facing change?
Use systemd property as fallback for cpu weight

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

internal/config/cgmgr/systemd_linux.go

haircommander · 2025-09-08T13:53:08Z

internal/config/cgmgr/systemd_linux.go

-				Name:  "CPUShares",
-				Value: dbus.MakeVariant(resources.CPU.Shares),
-			})
+			if node.CgroupIsV2() {


I'm curious why the IPPR test fails when conmon cgroup doesn't have cpu weight set. it's checking the pod cgroup weight and that's the culmination of its child cgroups?

I think the pod-level cgroup aggregates CPU weights from all its children, including conmon. If conmon’s cgroup has no cpu.weight, it defaults to 100, which can make the pod-level weight inconsistent. This will avoid that issue.

Looking at the log, it checked /sys/fs/cgroup/cpu.weight in the target container (c1-init) and it was empty.
Is it a pod level cgroup? It looks like container-level.

yeah sounds like container level. @sohankunkerkar this fix did fix the test for you though?

internal/factory/container/container.go

This resolves Pod InPlace Resize Container test failure where cpu.weight file was empty for Guaranteed QoS pods with restartable init containers. Sometimes, systemd fails to convert CPUShares → CPUWeight for this specific scenario, likely due to how it handles complex pod hierarchies with concurrent containers, but the exact internal mechanism is unclear. This acts more as a safeguard in case systemd doesn’t set the value. Signed-off-by: Sohan Kunkerkar <sohank2602@gmail.com>

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 5, 2025

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 5, 2025

openshift-ci bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Sep 5, 2025

openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Sep 5, 2025

openshift-ci bot added the dco-signoff: yes Indicates the PR's author has DCO signed all their commits. label Sep 5, 2025

openshift-ci bot added kind/bug Categorizes issue or PR as related to a bug. kind/flake Categorizes issue or PR as related to a flaky test. labels Sep 5, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 5, 2025

sohankunkerkar force-pushed the fix-podInPlace branch from 001f5e1 to b926c0d Compare September 5, 2025 22:21

sohankunkerkar closed this Sep 5, 2025

sohankunkerkar reopened this Sep 7, 2025

sohankunkerkar force-pushed the fix-podInPlace branch from b926c0d to 2537bdc Compare September 7, 2025 14:32

haircommander reviewed Sep 8, 2025

View reviewed changes

internal/config/cgmgr/systemd_linux.go Outdated Show resolved Hide resolved

haircommander reviewed Sep 8, 2025

View reviewed changes

sohankunkerkar force-pushed the fix-podInPlace branch from 2537bdc to 9ad7250 Compare September 8, 2025 15:15

sohankunkerkar marked this pull request as ready for review September 8, 2025 16:06

sohankunkerkar requested a review from mrunalp as a code owner September 8, 2025 16:06

openshift-ci bot requested review from bitoku and QiWang19 September 8, 2025 16:06

sohankunkerkar force-pushed the fix-podInPlace branch 2 times, most recently from 2bf48c2 to ed7dc8d Compare September 9, 2025 17:18

haircommander reviewed Sep 9, 2025

View reviewed changes

internal/factory/container/container.go Outdated Show resolved Hide resolved

sohankunkerkar force-pushed the fix-podInPlace branch 2 times, most recently from 24334cc to aba57f1 Compare September 9, 2025 21:14

sohankunkerkar force-pushed the fix-podInPlace branch 5 times, most recently from eb7712a to c424158 Compare September 10, 2025 00:58

sohankunkerkar force-pushed the fix-podInPlace branch from c424158 to 2172b77 Compare September 10, 2025 01:07

sohankunkerkar changed the title ~~[WIP] OCPBUGS-61220: internal/cgmgr: fix CPU weight setting for cgroup v2~~ OCPBUGS-61220: internal/cgmgr: fix CPU weight setting for cgroup v2 Sep 10, 2025

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 10, 2025

OCPBUGS-61220: internal/cgmgr: fix CPU weight setting for cgroup v2 #9456

Are you sure you want to change the base?

OCPBUGS-61220: internal/cgmgr: fix CPU weight setting for cgroup v2 #9456

Uh oh!

Conversation

sohankunkerkar commented Sep 5, 2025

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci-robot commented Sep 5, 2025

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci bot commented Sep 5, 2025

Uh oh!

openshift-ci bot commented Sep 5, 2025

Uh oh!

codecov bot commented Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

openshift-ci-robot commented Sep 5, 2025

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci-robot commented Sep 7, 2025

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

Uh oh!

haircommander Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

sohankunkerkar Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

bitoku Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

haircommander Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Sep 5, 2025 •

edited

Loading