Fix HostPort manager when one backend is unavailable #9222

danwinship · 2025-05-26T15:59:38Z

The new iptables-and-nftables MetaHostPortManager code was falling victim to a classic golang gotcha. If newHostportManagerIPTables returned a nil value (of type *hostportManagerIPTables), then that value would get assigned to mh.managers[utilnet.IPv4].iptables, but that value is a HostPortManager, so the value would be typecast, becoming a non-nil HostPortManager value wrapping the nil *hostportManagerIPTables value, and then later, this code

cri-o/internal/hostport/meta_hostport_manager.go

Lines 143 to 149 in 9d143f9

    
           if managers.iptables != nil { 
        
           	err := managers.iptables.Remove(id, mappingsForFamily) 
        
           	// Ignore iptables errors if we're primarily using nftables 
        
           	if err != nil && managers.nftables == nil { 
        
           		errstrings = append(errstrings, err.Error()) 
        
           	} 
        
           }

would see that it was non-nil and attempt to call a method on it, which would then SEGV because the underlying *hostportManagerIPTables was nil.

There are a few possible ways to fix this... this is one of them...

This wasn't caught by unit tests because the unit tests don't use NewMetaHostportManager (since it's not fully mock-able) and it wasn't caught by e2e tests because the e2e environment has both iptables and nftables installed.

I also pulled in a cri-o-ified version of kubernetes/kubernetes@b031258 to get rid of the superfluous error messages also seen in #9220.

Fixes #9220
/kind bug
/kind cleanup

Does this PR introduce a user-facing change?

Fixes a crash introduced in 1.33.0 when cleaning up a pod that uses HostPorts
on a system that has either just iptables (but not nftables) or just nftables
(but not iptables).

openshift-ci · 2025-05-26T15:59:50Z

Hi @danwinship. Thanks for your PR.

I'm waiting for a cri-o member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2025-05-26T16:07:04Z

Codecov Report

Attention: Patch coverage is 78.94737% with 8 lines in your changes missing coverage. Please review.

Project coverage is 66.52%. Comparing base (4abe996) to head (b3f1394).
Report is 9 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9222      +/-   ##
==========================================
+ Coverage   66.50%   66.52%   +0.01%     
==========================================
  Files         198      198              
  Lines       27164    27176      +12     
==========================================
+ Hits        18065    18078      +13     
+ Misses       7612     7611       -1     
  Partials     1487     1487

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

bitoku · 2025-05-26T16:41:14Z

/ok-to-test

bitoku · 2025-05-27T09:18:08Z

internal/iptables/iptables.go

+	if err != nil {
+		// The only likely error is "no such file or directory", in which case any
+		// further commands will fail the same way, so we don't need to do
+		// anything special here.
+		return runner
+	}


I might be too sensitive, but I want to log something every time we get error, especially when it interacts with something not managed by us.
But debug log is enough, I think.

internal/hostport/hostport_iptables.go

bitoku · 2025-05-27T09:22:47Z

Thank you for the quick fix!

If `iptables --version` failed, iptables.New() would log a warning and assume that the problem was that you had an implausibly ancient version of iptables installed. Change it to instead assume that the problem is that you don't have iptables installed at all (and only log at debug). Signed-off-by: Dan Winship <danwinship@redhat.com>

Add an internal constructor so that the unit tests share the same logic as the real constructor, and fix it to handle nil sub-managers correctly. Signed-off-by: Dan Winship <danwinship@redhat.com>

bitoku · 2025-05-30T09:09:40Z

/retest

openshift-ci · 2025-05-30T10:55:05Z

@danwinship: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/ci-e2e-evented-pleg	`b3f1394`	link	false	`/test ci-e2e-evented-pleg`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

bitoku · 2025-05-30T11:01:09Z

/skip
/lgtm
@cri-o/cri-o-maintainers PTAL

haircommander · 2025-05-30T15:38:43Z

/approve

thank you for the quick fix!

openshift-ci · 2025-05-30T15:38:50Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danwinship, haircommander

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

haircommander · 2025-06-02T13:13:46Z

/cherry-pick release-1.33

openshift-cherrypick-robot · 2025-06-02T13:14:51Z

@haircommander: new pull request created: #9241

In response to this:

/cherry-pick release-1.33

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

danwinship requested a review from mrunalp as a code owner May 26, 2025 15:59

openshift-ci bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 26, 2025

openshift-ci bot requested review from hasan4791 and klihub May 26, 2025 16:00

openshift-ci bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 26, 2025

bitoku reviewed May 27, 2025

View reviewed changes

internal/hostport/hostport_iptables.go Outdated Show resolved Hide resolved

danwinship added 2 commits May 27, 2025 07:53

Redo metaHostportManager construction, fix bug

b3f1394

Add an internal constructor so that the unit tests share the same logic as the real constructor, and fix it to handle nil sub-managers correctly. Signed-off-by: Dan Winship <danwinship@redhat.com>

danwinship force-pushed the hostport-no-iptables branch from cbb5c36 to b3f1394 Compare May 27, 2025 11:55

openshift-ci bot assigned bitoku May 30, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 30, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 30, 2025

openshift-merge-bot bot merged commit b31fbf3 into cri-o:main May 30, 2025
70 of 89 checks passed

danwinship deleted the hostport-no-iptables branch May 31, 2025 14:29

openshift-cherrypick-robot mentioned this pull request Jun 2, 2025

[release-1.33] Fix HostPort manager when one backend is unavailable #9241

Merged

github-actions bot mentioned this pull request Sep 14, 2025

Bump crio from v1.30.3 to v1.34.0 kokyhm/kubespray#293

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix HostPort manager when one backend is unavailable #9222

Fix HostPort manager when one backend is unavailable #9222

Uh oh!

danwinship commented May 26, 2025

Uh oh!

openshift-ci bot commented May 26, 2025

Uh oh!

codecov bot commented May 26, 2025 •

edited

Loading

Uh oh!

bitoku commented May 26, 2025

Uh oh!

bitoku May 27, 2025

Uh oh!

Uh oh!

bitoku commented May 27, 2025

Uh oh!

bitoku commented May 30, 2025

Uh oh!

openshift-ci bot commented May 30, 2025

Uh oh!

bitoku commented May 30, 2025

Uh oh!

haircommander commented May 30, 2025

Uh oh!

openshift-ci bot commented May 30, 2025

Uh oh!

Uh oh!

haircommander commented Jun 2, 2025

Uh oh!

openshift-cherrypick-robot commented Jun 2, 2025

Uh oh!

Uh oh!

	if managers.iptables != nil {
	err := managers.iptables.Remove(id, mappingsForFamily)
	// Ignore iptables errors if we're primarily using nftables
	if err != nil && managers.nftables == nil {
	errstrings = append(errstrings, err.Error())
	}
	}

Fix HostPort manager when one backend is unavailable #9222

Fix HostPort manager when one backend is unavailable #9222

Uh oh!

Conversation

danwinship commented May 26, 2025

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci bot commented May 26, 2025

Uh oh!

codecov bot commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bitoku commented May 26, 2025

Uh oh!

bitoku May 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bitoku commented May 27, 2025

Uh oh!

bitoku commented May 30, 2025

Uh oh!

openshift-ci bot commented May 30, 2025

Uh oh!

bitoku commented May 30, 2025

Uh oh!

haircommander commented May 30, 2025

Uh oh!

openshift-ci bot commented May 30, 2025

Uh oh!

Uh oh!

haircommander commented Jun 2, 2025

Uh oh!

openshift-cherrypick-robot commented Jun 2, 2025

Uh oh!

Uh oh!

codecov bot commented May 26, 2025 •

edited

Loading