-
Notifications
You must be signed in to change notification settings - Fork 41.6k
Description
What happened?
I tried to run a standalone kubelet (without kube-apiserver, so no kubeClient in kubelet), but there is a panic "invalid memory address or nil pointer dereference"
The corresponding logs:
E0211 15:50:57.552748 794906 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 60 [running]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.logPanic({0x3bd69e0, 0x72c4480})
/home/xiejinwei/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x85
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00020ccc0})
/home/xiejinwei/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x75
panic({0x3bd69e0, 0x72c4480})
/home/xiejinwei/.gvm/gos/go1.17/src/runtime/panic.go:1038 +0x215
k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.(*reconciler).updateDevicePath(0xc00020ccc0, 0x24)
/home/xiejinwei/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:590 +0x40
k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.(*reconciler).updateStates(0xc00020ccc0, 0xc00185fc68)
/home/xiejinwei/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:621 +0x3e
k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.(*reconciler).syncStates(0xc00020ccc0)
/home/xiejinwei/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:425 +0x1a5
k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.(*reconciler).sync(0x0)
/home/xiejinwei/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:339 +0x50
k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.(*reconciler).reconciliationLoopFunc.func1()
/home/xiejinwei/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:158 +0xd6
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000ee6f00)
/home/xiejinwei/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x67
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x410c67, {0x4af7180, 0xc00034a960}, 0x1, 0xc00005e4e0)
/home/xiejinwei/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00020ccc0, 0x5f5e100, 0x0, 0x0, 0x442ea5)
/home/xiejinwei/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
/home/xiejinwei/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.(*reconciler).Run(0xc00020ccc0, 0x0)
/home/xiejinwei/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:146 +0x45
created by k8s.io/kubernetes/pkg/kubelet/volumemanager.(*volumeManager).Run
/home/xiejinwei/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/volume_manager.go:292 +0x35f
What did you expect to happen?
I expect standalone kubelet to run without panic.
How can we reproduce it (as minimally and precisely as possible)?
Run a standalone kubelet
Anything else we need to know?
I think this is a problem of kubelet passing kubeClient into VolumeManager without checking if it is nil.
Related code snippet:
-
In
pkg/kubelet/kubelet.go,NewMainKubelet()passed the nilkubeClientintoNewVolumeManager():kubernetes/pkg/kubelet/kubelet.go
Lines 799 to 805 in 559b798
klet.volumeManager = volumemanager.NewVolumeManager( kubeCfg.EnableControllerAttachDetach, nodeName, klet.podManager, klet.podWorkers, klet.kubeClient, klet.volumePluginMgr, -
In
pkg/kubelet/volumemanager/volume_manager.go,NewVolumeManager()passed the nilkubeClientintoreconciler:
kubernetes/pkg/kubelet/volumemanager/volume_manager.go
Lines 218 to 220 in 559b798
vm.reconciler = reconciler.NewReconciler( kubeClient, controllerAttachDetachEnabled, -
In
pkg/kubelet/volumemanager/reconciler/reconciler.go,reconcilerused the nilkubeClientwithout checking:
kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go
Lines 589 to 590 in a753dac
func (rc *reconciler) updateDevicePath(volumesNeedUpdate map[v1.UniqueVolumeName]*reconstructedVolume) { node, fetchErr := rc.kubeClient.CoreV1().Nodes().Get(context.TODO(), string(rc.nodeName), metav1.GetOptions{})
Kubernetes version
I'm only running kubelet, the kubelet version is Kubernetes v1.22.7-rc.0.17+559b798e25c8f8-dirty
Cloud provider
Not using cloud provider
OS version
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focalInstall tools
I installed with source code, the tag is release-1.22
git clone https://github.com/kubernetes/kubernetes.git
git fetch origin release-1.22
git checkout release-1.22
make WHAT=cmd/kubelet
cp ./_output/local/bin/linux/amd64/kubelet /usr/local/bin/kubeletContainer runtime (CRI) and and version (if applicable)
containerd 1.4.12
Related plugins (CNI, CSI, ...) and versions (if applicable)
Metadata
Metadata
Assignees
Labels
Type
Projects
Status