OCPBUGS-55485: Fix deadlock when stopping uninterruptible container #9256

bitoku · 2025-06-13T06:18:36Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

Because c.stopTimeoutChan is consumed only until it starts to SIGKILL, it could be deadlock in this scenario.

StopContainer(1)
1. SetAsStopping
2. StopLoopForContainer
3. killContainer
StopContainer(2)
1. SetAsStopping => false
2. WaitOnStopTimeout
  c.stopLock.Lock()
  c.stopTimeoutChan <- timeout (len(c.stopTimeoutChan) == 1)
  c.stopLock.ULock()
StopContainer(n = 3~11)
1. SetAsStopping => false
2. WaitOnStopTimeout
  c.stopLock.Lock()
  c.stopTimeoutChan <- timeout (len(c.stopTimeoutChan) == n)
  c.stopLock.ULock()
StopContainer(12~)
1. SetAsStopping => false
2. WaitOnStopTimeout
  c.stopLock.Lock()
  c.stopTimeoutChan <- timeout (BLOCKED!!)
After the process was killed (back to 1)
1. StopLoopForContainer done
2. KillExecPIDs
3. c.stopLock.Lock() (DEADLOCK!!)

Which issue(s) this PR fixes:

https://issues.redhat.com/browse/OCPBUGS-55485

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Fix the bug that pod can't be terminated when the process is uninterruptible sleep for a while.

codecov · 2025-06-13T08:44:48Z

Codecov Report

Attention: Patch coverage is 92.85714% with 1 line in your changes missing coverage. Please review.

Project coverage is 66.91%. Comparing base (3dce7d8) to head (1e751b4).
Report is 18 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9256      +/-   ##
==========================================
- Coverage   67.05%   66.91%   -0.14%     
==========================================
  Files         198      198              
  Lines       27176    27189      +13     
==========================================
- Hits        18222    18193      -29     
- Misses       7449     7495      +46     
+ Partials     1505     1501       -4

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

haircommander · 2025-06-13T17:38:18Z

internal/oci/container.go

+	// because it is not controlled by the timeout anymore.
 	stopTimeoutChan       chan int64
 	stopWatchers          []chan struct{}
+	stopKillLoop          bool


nit: maybe stopKillLoopBegun or something? making clear it's part of the 'stop' operation and not an instruction whether to stop the kill loop

Thanks! done

Signed-off-by: Ayato Tokubi <atokubi@redhat.com>

openshift-ci-robot · 2025-06-19T06:06:37Z

@bitoku: This pull request references Jira Issue OCPBUGS-55485, which is invalid:

expected the bug to target the "4.20.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

What type of PR is this?

/kind bug

What this PR does / why we need it:

Because c.stopTimeoutChan is consumed only until it starts to SIGKILL, it could be deadlock in this scenario.

StopContainer(1)

SetAsStopping

StopLoopForContainer

killContainer

StopContainer(2)

SetAsStopping => false

WaitOnStopTimeout
c.stopLock.Lock()
c.stopTimeoutChan <- timeout (len(c.stopTimeoutChan) == 1)
c.stopLock.ULock()

StopContainer(n = 3~11)

SetAsStopping => false

WaitOnStopTimeout
c.stopLock.Lock()
c.stopTimeoutChan <- timeout (len(c.stopTimeoutChan) == n)
c.stopLock.ULock()

StopContainer(12~)

SetAsStopping => false

WaitOnStopTimeout
c.stopLock.Lock()
c.stopTimeoutChan <- timeout (BLOCKED!!)

After the process was killed (back to 1)

StopLoopForContainer done

KillExecPIDs

c.stopLock.Lock() (DEADLOCK!!)

Which issue(s) this PR fixes:

https://issues.redhat.com/browse/OCPBUGS-55485

Special notes for your reviewer:

Does this PR introduce a user-facing change?
Fix the bug that pod can't be terminated when the process is uninterruptible sleep for a while.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

bitoku · 2025-06-19T06:07:13Z

/jira refresh

openshift-ci-robot · 2025-06-19T06:07:18Z

@bitoku: This pull request references Jira Issue OCPBUGS-55485, which is invalid:

expected the bug to target the "4.20.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

bitoku · 2025-06-19T06:08:00Z

/jira refresh

openshift-ci-robot · 2025-06-19T06:08:08Z

@bitoku: This pull request references Jira Issue OCPBUGS-55485, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.20.0) matches configured target version for branch (4.20.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @lyman9966

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2025-06-19T06:08:10Z

@openshift-ci-robot: GitHub didn't allow me to request PR reviews from the following users: lyman9966.

Note that only cri-o members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

@bitoku: This pull request references Jira Issue OCPBUGS-55485, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)

bug target version (4.20.0) matches configured target version for branch (4.20.0)

bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @lyman9966

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

haircommander · 2025-06-20T18:36:39Z

/override ci/prow/ci-e2e-evented-pleg
/approve

LGTM, @cri-o/cri-o-maintainers PTAL

openshift-ci · 2025-06-20T18:36:45Z

@haircommander: Overrode contexts on behalf of haircommander: ci/prow/ci-e2e-evented-pleg

In response to this:

/override ci/prow/ci-e2e-evented-pleg
/approve

LGTM, @cri-o/cri-o-maintainers PTAL

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

sohankunkerkar

/lgtm

openshift-ci · 2025-06-20T21:15:23Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bitoku, haircommander, sohankunkerkar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander,sohankunkerkar]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

bitoku · 2025-06-24T02:00:48Z

/retest

bitoku · 2025-06-24T03:29:18Z

/retest

openshift-ci-robot · 2025-06-24T06:08:54Z

@bitoku: Jira Issue OCPBUGS-55485: All pull requests linked via external trackers have merged:

cri-o/cri-o#9256

Jira Issue OCPBUGS-55485 has been moved to the MODIFIED state.

In response to this:

What type of PR is this?

/kind bug

What this PR does / why we need it:

Because c.stopTimeoutChan is consumed only until it starts to SIGKILL, it could be deadlock in this scenario.

StopContainer(1)

SetAsStopping

StopLoopForContainer

killContainer

StopContainer(2)

SetAsStopping => false

WaitOnStopTimeout
c.stopLock.Lock()
c.stopTimeoutChan <- timeout (len(c.stopTimeoutChan) == 1)
c.stopLock.ULock()

StopContainer(n = 3~11)

SetAsStopping => false

WaitOnStopTimeout
c.stopLock.Lock()
c.stopTimeoutChan <- timeout (len(c.stopTimeoutChan) == n)
c.stopLock.ULock()

StopContainer(12~)

SetAsStopping => false

WaitOnStopTimeout
c.stopLock.Lock()
c.stopTimeoutChan <- timeout (BLOCKED!!)

After the process was killed (back to 1)

StopLoopForContainer done

KillExecPIDs

c.stopLock.Lock() (DEADLOCK!!)

Which issue(s) this PR fixes:

https://issues.redhat.com/browse/OCPBUGS-55485

Special notes for your reviewer:

Does this PR introduce a user-facing change?
Fix the bug that pod can't be terminated when the process is uninterruptible sleep for a while.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

bitoku · 2025-07-04T07:02:33Z

/cherry-pick release-1.33

openshift-cherrypick-robot · 2025-07-04T07:03:32Z

@bitoku: new pull request created: #9320

In response to this:

/cherry-pick release-1.33

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

bitoku · 2025-07-08T11:13:27Z

/cherry-pick release-1.32

bitoku · 2025-07-08T11:13:54Z

/cherry-pick release-1.31

openshift-cherrypick-robot · 2025-07-08T11:14:07Z

@bitoku: #9256 failed to apply on top of branch "release-1.32":

Applying: fix deadlock when the container is in uninterruptible sleep
Using index info to reconstruct a base tree...
M	internal/oci/container.go
M	internal/oci/runtime_oci.go
Falling back to patching base and 3-way merge...
Auto-merging internal/oci/runtime_oci.go
CONFLICT (content): Merge conflict in internal/oci/runtime_oci.go
Auto-merging internal/oci/container.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 fix deadlock when the container is in uninterruptible sleep

In response to this:

/cherry-pick release-1.32

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

bitoku · 2025-07-08T11:14:08Z

/cherry-pick release-1.30

openshift-cherrypick-robot · 2025-07-08T11:14:35Z

@bitoku: #9256 failed to apply on top of branch "release-1.31":

Applying: fix deadlock when the container is in uninterruptible sleep
Using index info to reconstruct a base tree...
M	internal/oci/container.go
M	internal/oci/runtime_oci.go
Falling back to patching base and 3-way merge...
Auto-merging internal/oci/runtime_oci.go
CONFLICT (content): Merge conflict in internal/oci/runtime_oci.go
Auto-merging internal/oci/container.go
CONFLICT (content): Merge conflict in internal/oci/container.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 fix deadlock when the container is in uninterruptible sleep

In response to this:

/cherry-pick release-1.31

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-cherrypick-robot · 2025-07-08T11:14:48Z

@bitoku: #9256 failed to apply on top of branch "release-1.30":

Applying: fix deadlock when the container is in uninterruptible sleep
Using index info to reconstruct a base tree...
M	internal/oci/container.go
M	internal/oci/runtime_oci.go
Falling back to patching base and 3-way merge...
Auto-merging internal/oci/runtime_oci.go
CONFLICT (content): Merge conflict in internal/oci/runtime_oci.go
Auto-merging internal/oci/container.go
CONFLICT (content): Merge conflict in internal/oci/container.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 fix deadlock when the container is in uninterruptible sleep

In response to this:

/cherry-pick release-1.30

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

bitoku requested a review from mrunalp as a code owner June 13, 2025 06:18

openshift-ci bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. kind/bug Categorizes issue or PR as related to a bug. labels Jun 13, 2025

openshift-ci bot requested review from klihub and littlejawa June 13, 2025 06:19

bitoku force-pushed the fix-deadlock branch from c27c906 to 8191850 Compare June 13, 2025 07:01

haircommander reviewed Jun 13, 2025

View reviewed changes

fix deadlock when the container is in uninterruptible sleep

1e751b4

Signed-off-by: Ayato Tokubi <atokubi@redhat.com>

bitoku force-pushed the fix-deadlock branch from 8191850 to 1e751b4 Compare June 18, 2025 01:54

bitoku changed the title ~~Fix deadlock when stopping uninterruptible container~~ OCPBUGS-55485: Fix deadlock when stopping uninterruptible container Jun 19, 2025

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 19, 2025

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 19, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 20, 2025

sohankunkerkar approved these changes Jun 20, 2025

View reviewed changes

openshift-ci bot assigned sohankunkerkar Jun 20, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 20, 2025

openshift-merge-bot bot merged commit 43ed0a1 into cri-o:main Jun 24, 2025
86 of 89 checks passed

bitoku deleted the fix-deadlock branch July 4, 2025 07:02

openshift-cherrypick-robot mentioned this pull request Jul 4, 2025

[release-1.33]: OCPBUGS-55485: Fix deadlock when stopping uninterruptible container #9320

Merged

lance5890 mentioned this pull request Aug 15, 2025

Container stuch in stopping status when pid and conmon-pid are all gone #9406

Closed

github-actions bot mentioned this pull request Sep 14, 2025

Bump crio from v1.30.3 to v1.34.0 kokyhm/kubespray#293

Open

OCPBUGS-55485: Fix deadlock when stopping uninterruptible container #9256

OCPBUGS-55485: Fix deadlock when stopping uninterruptible container #9256

Uh oh!

Conversation

bitoku commented Jun 13, 2025

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

codecov bot commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

haircommander Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

bitoku Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

openshift-ci-robot commented Jun 19, 2025

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

bitoku commented Jun 19, 2025

Uh oh!

openshift-ci-robot commented Jun 19, 2025

Uh oh!

bitoku commented Jun 19, 2025

Uh oh!

openshift-ci-robot commented Jun 19, 2025

Uh oh!

openshift-ci bot commented Jun 19, 2025

Uh oh!

haircommander commented Jun 20, 2025

Uh oh!

openshift-ci bot commented Jun 20, 2025

Uh oh!

sohankunkerkar left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Jun 20, 2025

Uh oh!

bitoku commented Jun 24, 2025

Uh oh!

bitoku commented Jun 24, 2025

Uh oh!

Uh oh!

openshift-ci-robot commented Jun 24, 2025

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

bitoku commented Jul 4, 2025

Uh oh!

openshift-cherrypick-robot commented Jul 4, 2025

Uh oh!

bitoku commented Jul 8, 2025

Uh oh!

bitoku commented Jul 8, 2025

Uh oh!

openshift-cherrypick-robot commented Jul 8, 2025

Uh oh!

bitoku commented Jul 8, 2025

Uh oh!

openshift-cherrypick-robot commented Jul 8, 2025

Uh oh!

openshift-cherrypick-robot commented Jul 8, 2025

Uh oh!

Uh oh!

codecov bot commented Jun 13, 2025 •

edited

Loading