OCPBUGS-58229: server: handle missing network namespace gracefully during networkStop #9301

sohankunkerkar · 2025-06-30T13:36:17Z

After host reboot, network namespaces are destroyed but CRI-O attempts to clean them up during pod sandbox destruction, causing CNI plugin failures and preventing pods from restarting properly. The fix ensures pods can restart normally after host reboots.

What type of PR is this?

/kind bug

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Link to journal logs

Does this PR introduce a user-facing change?

Handle missing network namespace gracefully during networkStop

openshift-ci-robot · 2025-06-30T13:36:23Z

@sohankunkerkar: This pull request references Jira Issue OCPBUGS-58229, which is invalid:

expected the bug to target the "4.20.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

After host reboot, network namespaces are destroyed but CRI-O attempts to clean them up during pod sandbox destruction, causing CNI plugin failures and preventing pods from restarting properly. The fix ensures pods can restart normally after host reboots.

What type of PR is this?

/kind bug

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?
Handle missing network namespace gracefully during networkStop

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

danwinship · 2025-06-30T13:41:04Z

After a reboot there would be no iptables/nftables hostport rules anyway, so I don't think this is the problem. (And I don't see anything about hostports in the linked bug, though I didn't pull the full logs from the Google drive link...)

codecov · 2025-06-30T14:51:51Z

Codecov Report

Attention: Patch coverage is 61.53846% with 15 lines in your changes missing coverage. Please review.

Project coverage is 66.89%. Comparing base (2edb23f) to head (faec3f5).
Report is 19 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9301      +/-   ##
==========================================
- Coverage   66.92%   66.89%   -0.03%     
==========================================
  Files         198      199       +1     
  Lines       27306    27456     +150     
==========================================
+ Hits        18274    18367      +93     
- Misses       7525     7568      +43     
- Partials     1507     1521      +14

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

server/sandbox_network.go

haircommander · 2025-07-01T14:49:43Z

/approve

it's possible you'd be able to make a integration test by killing a pod and unmounting its netns, and restarting crio. may be annoying and not worth the specific case, but worth a shot I think

sohankunkerkar · 2025-07-10T12:45:18Z

/retest

After host reboot, network namespaces are destroyed but CRI-O attempts to clean them up during pod sandbox destruction, causing CNI plugin failures and preventing pods from restarting properly. The fix ensures pods can restart normally after host reboots. Signed-off-by: Sohan Kunkerkar <sohank2602@gmail.com>

…etns Signed-off-by: Sohan Kunkerkar <sohank2602@gmail.com>

Signed-off-by: Sohan Kunkerkar <sohank2602@gmail.com>

Kata VMs use real infra containers that persist in storage, unlike normal containers that use spoofed infra containers. This fundamental architectural difference means the 'Network recovery after reboot with destroyed netns' test scenario doesn't apply to Kata VMs in the same way. Signed-off-by: Sohan Kunkerkar <sohank2602@gmail.com>

sohankunkerkar · 2025-07-10T15:45:04Z

/test e2e-gcp-ovn

saschagrunert

Just a non-blocking nit, otherwise LGTM

saschagrunert · 2025-07-11T07:10:41Z

server/sandbox_network_linux.go

+		return fmt.Errorf("invalid network namespace: %w", err)
+	}
+
+	defer netns.Close()


I'd remove the defer here.

openshift-ci · 2025-07-11T07:11:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: haircommander, saschagrunert, sohankunkerkar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander,saschagrunert,sohankunkerkar]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

saschagrunert · 2025-07-11T07:12:31Z

/override ci/prow/ci-e2e-evented-pleg

openshift-ci · 2025-07-11T07:12:37Z

@saschagrunert: Overrode contexts on behalf of saschagrunert: ci/prow/ci-e2e-evented-pleg

In response to this:

/override ci/prow/ci-e2e-evented-pleg

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci-robot · 2025-07-11T07:15:46Z

@sohankunkerkar: Jira Issue OCPBUGS-58229: All pull requests linked via external trackers have merged:

cri-o/cri-o#9301

Jira Issue OCPBUGS-58229 has been moved to the MODIFIED state.

In response to this:

After host reboot, network namespaces are destroyed but CRI-O attempts to clean them up during pod sandbox destruction, causing CNI plugin failures and preventing pods from restarting properly. The fix ensures pods can restart normally after host reboots.

What type of PR is this?

/kind bug

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Link to journal logs

Does this PR introduce a user-facing change?
Handle missing network namespace gracefully during networkStop

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

sohankunkerkar · 2025-07-11T11:59:19Z

/cherry-pick release-1.33

openshift-cherrypick-robot · 2025-07-11T12:00:21Z

@sohankunkerkar: new pull request created: #9337

In response to this:

/cherry-pick release-1.33

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

aojea · 2025-07-31T10:27:21Z

server/sandbox_network.go

-		retErr := fmt.Errorf("failed to destroy network for pod sandbox %s(%s): %w", sb.Name(), sb.ID(), err)
-
+	// Check if the network namespace file exists and is valid before attempting CNI teardown.
+	// If the file doesn't exist or is invalid, skip CNI teardown and mark network as stopped.


We had IP leaks on containerd because of a similar change that ignored calling the network CNI plugin on some cases, causing the CNI to no release the IP address containerd/containerd#12132

I can not see this is similar or not, but @danwinship you should ensure that in this scenario the plugin gets the CNI DEL to release any resource associated to the Pod

@aojea Thanks for calling that out. I can see the potential issue there. Let me go ahead and fix the issue.

sohankunkerkar requested a review from mrunalp as a code owner June 30, 2025 13:36

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 30, 2025

openshift-ci-robot added the jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. label Jun 30, 2025

openshift-ci bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jun 30, 2025

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 30, 2025

openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Jun 30, 2025

openshift-ci bot added kind/bug Categorizes issue or PR as related to a bug. dco-signoff: no Indicates the PR's author has not DCO signed all their commits. labels Jun 30, 2025

openshift-ci bot requested review from littlejawa and QiWang19 June 30, 2025 13:36

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 30, 2025

sohankunkerkar force-pushed the fix-cni-issue branch from b281307 to 7b24a96 Compare June 30, 2025 13:52

openshift-ci bot added dco-signoff: yes Indicates the PR's author has DCO signed all their commits. and removed dco-signoff: no Indicates the PR's author has not DCO signed all their commits. labels Jun 30, 2025

sohankunkerkar force-pushed the fix-cni-issue branch 2 times, most recently from fa257cc to 56a1b4e Compare June 30, 2025 14:18

sohankunkerkar force-pushed the fix-cni-issue branch from 56a1b4e to 3ef3837 Compare June 30, 2025 16:16

haircommander reviewed Jun 30, 2025

View reviewed changes

server/sandbox_network.go Outdated Show resolved Hide resolved

sohankunkerkar force-pushed the fix-cni-issue branch 3 times, most recently from d0787db to 72cbd03 Compare July 1, 2025 14:42

sohankunkerkar force-pushed the fix-cni-issue branch from 72cbd03 to 95f7c65 Compare July 1, 2025 17:25

sohankunkerkar changed the title ~~[WIP] OCPBUGS-58229: server: handle missing network namespace gracefully during networkStop~~ OCPBUGS-58229: server: handle missing network namespace gracefully during networkStop Jul 1, 2025

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 1, 2025

sohankunkerkar force-pushed the fix-cni-issue branch 2 times, most recently from dbf2aa3 to bd9856e Compare July 1, 2025 18:14

sohankunkerkar force-pushed the fix-cni-issue branch from f18dd4f to b1f23b2 Compare July 10, 2025 12:44

sohankunkerkar force-pushed the fix-cni-issue branch 3 times, most recently from edbd7a7 to c0b6856 Compare July 10, 2025 13:03

sohankunkerkar added 4 commits July 10, 2025 10:21

test: add coverage for network recovery after reboot with destroyed n…

1ae6655

…etns Signed-off-by: Sohan Kunkerkar <sohank2602@gmail.com>

test: use quay.io instead of registry.fedoraproject.org

65395cb

Signed-off-by: Sohan Kunkerkar <sohank2602@gmail.com>

sohankunkerkar force-pushed the fix-cni-issue branch from c0b6856 to faec3f5 Compare July 10, 2025 14:21

saschagrunert approved these changes Jul 11, 2025

View reviewed changes

openshift-ci bot assigned saschagrunert Jul 11, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jul 11, 2025

openshift-merge-bot bot merged commit 9cdf516 into cri-o:main Jul 11, 2025
85 of 89 checks passed

openshift-cherrypick-robot mentioned this pull request Jul 11, 2025

[release-1.33] : server: handle missing network namespace gracefully during networkStop #9337

Merged

haircommander mentioned this pull request Jul 30, 2025

Failed to destroy network for pod from CNI network after reboot kubernetes/kubernetes#133081

Closed

aojea reviewed Jul 31, 2025

View reviewed changes

sohankunkerkar mentioned this pull request Jul 31, 2025

server: ensure CNI teardown prevents IP leaks with missing netns #9372

Merged

This was referenced Aug 8, 2025

[release-1.32] Multiple patches to fix OCPBUGS-48043 #9395

Merged

[release-1.31] Multiple patches to fix OCPBUGS-48043 #9397

Open

This was referenced Aug 28, 2025

[release-1.30] Multiple patches to fix OCPBUGS-48043 #9435

Open

[release-1.29] Multiple patches to fix OCPBUGS-48043 #9436

Open

github-actions bot mentioned this pull request Sep 14, 2025

Bump crio from v1.30.3 to v1.34.0 kokyhm/kubespray#293

Open

OCPBUGS-58229: server: handle missing network namespace gracefully during networkStop #9301

OCPBUGS-58229: server: handle missing network namespace gracefully during networkStop #9301

Uh oh!

Conversation

sohankunkerkar commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci-robot commented Jun 30, 2025

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

danwinship commented Jun 30, 2025

Uh oh!

codecov bot commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

haircommander commented Jul 1, 2025

Uh oh!

sohankunkerkar commented Jul 10, 2025

Uh oh!

sohankunkerkar commented Jul 10, 2025

Uh oh!

saschagrunert left a comment

Choose a reason for hiding this comment

Uh oh!

saschagrunert Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Jul 11, 2025

Uh oh!

saschagrunert commented Jul 11, 2025

Uh oh!

openshift-ci bot commented Jul 11, 2025

Uh oh!

Uh oh!

openshift-ci-robot commented Jul 11, 2025

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

sohankunkerkar commented Jul 11, 2025

Uh oh!

openshift-cherrypick-robot commented Jul 11, 2025

Uh oh!

aojea Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

sohankunkerkar Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sohankunkerkar commented Jun 30, 2025 •

edited

Loading

codecov bot commented Jun 30, 2025 •

edited

Loading