old instance-manager pods are not terminated after upgrading (live) engine #13264

jackchuong · 2026-06-05T09:24:05Z

jackchuong
Jun 5, 2026

Hi all,
In my environment I have 4 VM in k8s cluster build by k3s

kubectl get nodes --show-labels
NAME             STATUS   ROLES                  AGE      VERSION        LABELS
k3s-dc-worker3   Ready    <none>                 2y140d   v1.28.5+k3s1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3s-dc-worker3,kubernetes.io/os=linux,longhorn-node=true,node.kubernetes.io/instance-type=k3s
k3s-dc-master1   Ready    control-plane,master   2y267d   v1.27.5+k3s1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3s-dc-master1,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=true,node-role.kubernetes.io/master=true,node.kubernetes.io/instance-type=k3s
k3s-dc-worker1   Ready    <none>                 2y266d   v1.28.5+k3s1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3s-dc-worker1,kubernetes.io/os=linux,longhorn-node=true,node.kubernetes.io/instance-type=k3s
k3s-dc-worker2   Ready    <none>                 2y197d   v1.28.5+k3s1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k3s-dc-worker2,kubernetes.io/os=linux,longhorn-node=true,node.kubernetes.io/instance-type=k3s

I installed longhorn by helm on k3s-dc-worker1 , k3s-dc-worker2 , k3s-dc-worker3 , mount /mnt/longhorn-sdb on each server

cat longhorn-values.yaml
global:
  nodeSelector:
    longhorn-node: "true"
  timezone: "Asia/Ho_Chi_Minh"

persistence:
  defaultClass: false

ingress:
  enabled: true
  ingressClassName: "nginx"
  host: "longhorn-k8s-local.mydomain.com"
  tls: true
  tlsSecret: "mydomain.com-tls"
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: longhorn-basic-auth
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - Longhorn'

defaultSettings:
  deletingConfirmationFlag: "true"
  v2DataEngine: "true"

I accidentally upgraded longhorn release
helm -n longhorn upgrade longhorn longhorn/longhorn -f longhorn-values.yaml

helm -n longhorn list
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
longhorn        longhorn        3               2026-05-06 10:23:53.459494963 +0700 +07 deployed        longhorn-1.11.1 v1.11.1

A new engine image appears

kubectl -n longhorn get engineimage
NAME          INCOMPATIBLE   STATE      IMAGE                                          REFCOUNT   BUILDDATE   AGE
ei-75a03ec3   false          deployed   docker.io/longhornio/longhorn-engine:v1.11.1   100        83d         30d
ei-3154f3aa   false          deployed   longhornio/longhorn-engine:v1.10.1             0          205d        134d

New instance-manager and engine-image pods also appear

kubectl -n longhorn get pod
NAME                                                READY   STATUS      RESTARTS   AGE
instance-manager-cf040c053e2f7fd2333ed533dec1a6bd   1/1     Running     0          134d
instance-manager-eb9561ac90cc783b9b80b708491bd8b2   1/1     Running     0          134d
instance-manager-46539f132c2240858e3159be8254eb65   1/1     Running     0          134d
engine-image-ei-3154f3aa-zr7hk                      1/1     Running     0          134d
engine-image-ei-3154f3aa-m5wn9                      1/1     Running     0          134d
engine-image-ei-3154f3aa-c4xwc                      1/1     Running     0          134d
engine-image-ei-3154f3aa-r8n88                      1/1     Running     0          134d
longhorn-ui-67fd69458-mnfvn                         1/1     Running     0          30d
longhorn-ui-67fd69458-nrs4h                         1/1     Running     0          30d
longhorn-manager-x2gtt                              2/2     Running     0          30d
longhorn-driver-deployer-86dfbfbfc5-xwvnx           1/1     Running     0          30d
longhorn-manager-45ljs                              2/2     Running     0          30d
longhorn-manager-pfpgm                              2/2     Running     0          30d
engine-image-ei-75a03ec3-27szt                      1/1     Running     0          30d
engine-image-ei-75a03ec3-t8wmk                      1/1     Running     0          30d
engine-image-ei-75a03ec3-99r6k                      1/1     Running     0          30d
engine-image-ei-75a03ec3-9qb5w                      1/1     Running     0          30d
csi-attacher-6975495887-zhb4n                       1/1     Running     0          30d
csi-provisioner-5d5ccf5b48-vvfn4                    1/1     Running     0          30d
csi-resizer-68bdd88584-l7k2v                        1/1     Running     0          30d
instance-manager-1b963018fb570dd58783ad16e5f07173   1/1     Running     0          30d
instance-manager-9c82ab43fd761052ec66b1c9cffda7a5   1/1     Running     0          30d
csi-snapshotter-5fbc58cd9c-d9k2w                    1/1     Running     0          30d
csi-provisioner-5d5ccf5b48-rsxjx                    1/1     Running     0          30d
csi-snapshotter-5fbc58cd9c-grqpq                    1/1     Running     0          30d
csi-attacher-6975495887-7lgtn                       1/1     Running     0          30d
instance-manager-d31436f5f4c6478e64e6704b6abc5cdf   1/1     Running     0          30d
instance-manager-52c28ff63bfb46d6989406408303ccb2   1/1     Running     0          30d
csi-attacher-6975495887-v4tvg                       1/1     Running     0          30d
csi-resizer-68bdd88584-m67g4                        1/1     Running     0          30d
csi-resizer-68bdd88584-ksp29                        1/1     Running     0          30d
csi-snapshotter-5fbc58cd9c-p7vpq                    1/1     Running     0          30d
csi-provisioner-5d5ccf5b48-gthqj                    1/1     Running     0          30d
longhorn-csi-plugin-q2zwc                           3/3     Running     0          30d
longhorn-csi-plugin-dvx49                           3/3     Running     0          30d
longhorn-csi-plugin-75hs4                           3/3     Running     0          30d
longhorn-csi-plugin-8rqcz                           3/3     Running     0          30d
instance-manager-8892c3082d2590b43a5b79b5756cf6c5   1/1     Running     0          30d
instance-manager-254e8828b6e9e9bc33e767f830501ce3   1/1     Running     0          30d
backup-daily-29676605-g9v62                         0/1     Completed   0          14h
backup-system-daily-29676670-gknwd                  0/1     Completed   0          13h

I have 20 volumes attached to 20 pods , I upgraded engine images of these 20 volumes from longhornio/longhorn-engine:v1.10.1 to docker.io/longhornio/longhorn-engine:v1.11.1

While working on that, I observed on the Longhorn web UI and noticed that the replicas of the transferred volumes were being managed by new instance-manager pods : instance-manager-52c28ff63bfb46d6989406408303ccb2 , instance-manager-8892c3082d2590b43a5b79b5756cf6c5 , instance-manager-9c82ab43fd761052ec66b1c9cffda7a5

I read these documents:
https://longhorn.io/docs/1.11.1/deploy/upgrade/upgrade-engine/
https://longhorn.io/docs/1.11.1/deploy/upgrade/instance-manager-pods-during-upgrade/
I checked and no replicas/volumes are being handled by the remaining instance-manager pods.

kubectl -n longhorn get replicas.longhorn.io -o json | jq -r '.items[] | select(.status.instanceManagerName == "instance-manager-cf040c053e2f7fd2333ed533dec1a6bd") | .metadata.name'
kubectl -n longhorn get replicas.longhorn.io -o json | jq -r '.items[] | select(.status.instanceManagerName == "instance-manager-eb9561ac90cc783b9b80b708491bd8b2") | .metadata.name'
kubectl -n longhorn get replicas.longhorn.io -o json | jq -r '.items[] | select(.status.instanceManagerName == "instance-manager-46539f132c2240858e3159be8254eb65") | .metadata.name'
kubectl -n longhorn get replicas.longhorn.io -o json | jq -r '.items[] | select(.status.instanceManagerName == "instance-manager-1b963018fb570dd58783ad16e5f07173") | .metadata.name'
kubectl -n longhorn get replicas.longhorn.io -o json | jq -r '.items[] | select(.status.instanceManagerName == "instance-manager-d31436f5f4c6478e64e6704b6abc5cdf") | .metadata.name'
kubectl -n longhorn get replicas.longhorn.io -o json | jq -r '.items[] | select(.status.instanceManagerName == "instance-manager-254e8828b6e9e9bc33e767f830501ce3") | .metadata.name'

kubectl -n longhorn get replicas.longhorn.io -o json | jq -r '.items[] | select(.status.instanceManagerName == "instance-manager-9c82ab43fd761052ec66b1c9cffda7a5") | .metadata.name'
pvc-8e8c614f-e0b7-4eb6-8838-eab9c3557cd7-r-9c3c8668
pvc-af62d412-784a-4c1a-bcba-f625a51a32f8-r-04028f95
pvc-89c796fb-360f-4c1e-a7ec-97ce0a1ee330-r-4124f0e6
pvc-64c57515-b09e-4c71-b812-3b11a2c26b80-r-db95af8b
pvc-4be811b0-17d1-4f56-acfe-0f394756b9be-r-6377397b
pvc-f6bd062e-7596-4376-b553-9c1efb607e96-r-1b2b9077
pvc-fd6a0fcb-8c41-446b-b011-c18ce60adc60-r-273004d6
pvc-c646939b-6fc0-47f9-bef9-88847e9a93f3-r-20ccef68
pvc-e94a5bbb-ddbd-413b-8bfe-7745bac6710f-r-b9b7eb0c
pvc-a1156762-db17-49f8-b129-c724c0af8b04-r-6c7273ee
pvc-a9c1563b-761a-450f-b28f-9ee2660428a9-r-5b4043d1
pvc-9ce83ee8-a7af-482b-8505-c43aba0fdef8-r-2c6d33e3
pvc-7c3c678b-bfd0-4c7e-a904-43ab0e8c09c1-r-83faae12
pvc-cb3058a2-1ca8-4424-bcbe-45f7d440898c-r-52afc02e
pvc-96d0b030-0f30-45fa-be5c-e25570bbb005-r-eeef6959
pvc-075d134d-3903-466c-9fd0-e5bdb234c0c8-r-d5fa1d47
pvc-0fbd22bb-2ec2-4166-bb70-05d26c89406e-r-4ea74eff
pvc-4b5a0556-ed10-4450-9fe7-9565197ac7fd-r-99ab0018
pvc-2be3a7bd-9ace-4a81-ab4d-a7a6b1b80edf-r-4152a66f

kubectl -n longhorn get replicas.longhorn.io -o json | jq -r '.items[] | select(.status.instanceManagerName == "instance-manager-52c28ff63bfb46d6989406408303ccb2") | .metadata.name'
pvc-8e8c614f-e0b7-4eb6-8838-eab9c3557cd7-r-eb3b05c4
pvc-af62d412-784a-4c1a-bcba-f625a51a32f8-r-7368f412
pvc-89c796fb-360f-4c1e-a7ec-97ce0a1ee330-r-4ac67e83
pvc-64c57515-b09e-4c71-b812-3b11a2c26b80-r-929407e5
pvc-4be811b0-17d1-4f56-acfe-0f394756b9be-r-6259059b
pvc-f6bd062e-7596-4376-b553-9c1efb607e96-r-9af0051d
pvc-fd6a0fcb-8c41-446b-b011-c18ce60adc60-r-3f2b6c40
pvc-c646939b-6fc0-47f9-bef9-88847e9a93f3-r-3fd4c19b
pvc-e94a5bbb-ddbd-413b-8bfe-7745bac6710f-r-676156f0
pvc-a1156762-db17-49f8-b129-c724c0af8b04-r-a7b0a901
pvc-a9c1563b-761a-450f-b28f-9ee2660428a9-r-7b8d00dd
pvc-9ce83ee8-a7af-482b-8505-c43aba0fdef8-r-144302aa
pvc-7c3c678b-bfd0-4c7e-a904-43ab0e8c09c1-r-1509f807
pvc-cb3058a2-1ca8-4424-bcbe-45f7d440898c-r-20ce654b
pvc-96d0b030-0f30-45fa-be5c-e25570bbb005-r-b12cb7d2
pvc-075d134d-3903-466c-9fd0-e5bdb234c0c8-r-1c06353e
pvc-0fbd22bb-2ec2-4166-bb70-05d26c89406e-r-5cf74c5f
pvc-4b5a0556-ed10-4450-9fe7-9565197ac7fd-r-ba7f990a
pvc-2be3a7bd-9ace-4a81-ab4d-a7a6b1b80edf-r-1840545d

kubectl -n longhorn get replicas.longhorn.io -o json | jq -r '.items[] | select(.status.instanceManagerName == "instance-manager-8892c3082d2590b43a5b79b5756cf6c5") | .metadata.name'
pvc-8e8c614f-e0b7-4eb6-8838-eab9c3557cd7-r-e6669eca
pvc-af62d412-784a-4c1a-bcba-f625a51a32f8-r-578489b6
pvc-89c796fb-360f-4c1e-a7ec-97ce0a1ee330-r-395a6a3b
pvc-64c57515-b09e-4c71-b812-3b11a2c26b80-r-f2a184e7
pvc-4be811b0-17d1-4f56-acfe-0f394756b9be-r-f645e338
pvc-f6bd062e-7596-4376-b553-9c1efb607e96-r-7addb9ec
pvc-fd6a0fcb-8c41-446b-b011-c18ce60adc60-r-8cff420f
pvc-c646939b-6fc0-47f9-bef9-88847e9a93f3-r-79b63b21
pvc-e94a5bbb-ddbd-413b-8bfe-7745bac6710f-r-396a23d3
pvc-a1156762-db17-49f8-b129-c724c0af8b04-r-468ac286
pvc-a9c1563b-761a-450f-b28f-9ee2660428a9-r-a3d2e6f2
pvc-9ce83ee8-a7af-482b-8505-c43aba0fdef8-r-7397231b
pvc-7c3c678b-bfd0-4c7e-a904-43ab0e8c09c1-r-1ba851cf
pvc-cb3058a2-1ca8-4424-bcbe-45f7d440898c-r-dfa87b5c
pvc-96d0b030-0f30-45fa-be5c-e25570bbb005-r-e42a28c0
pvc-075d134d-3903-466c-9fd0-e5bdb234c0c8-r-fb1fe77f
pvc-0fbd22bb-2ec2-4166-bb70-05d26c89406e-r-49a7ddd9
pvc-4b5a0556-ed10-4450-9fe7-9565197ac7fd-r-e32eb3a0
pvc-2be3a7bd-9ace-4a81-ab4d-a7a6b1b80edf-r-3cc8ca35

But old instance-manager pods are not terminated.

Do I have to scale down 20 pods to detach 20 volume at same time so longhorn can terminated old instance-manager pods ?
Please give me some advice, thank you very much.

Answered by jackchuong

Jun 8, 2026

I scaled down and up each of the 20 pods using Longhorn Volume to detach/attach each volume within the 20 Longhorn Volumes.
Old instance-manager pods are terminated automatically

View full answer

derekbit · 2026-06-05T10:06:10Z

derekbit
Jun 5, 2026
Maintainer

@jackchuong Please check https://longhorn.io/kb/troubleshooting-some-old-instance-manager-pods-are-still-running-after-upgrade/

1 reply

jackchuong Jun 5, 2026
Author

@derekbit so I have to scale down&up each pod in 20 pods to detach&attach 20 volumes so longhorn can terminated old instance-manager pods ?

jackchuong · 2026-06-08T04:13:03Z

jackchuong
Jun 8, 2026
Author

I scaled down and up each of the 20 pods using Longhorn Volume to detach/attach each volume within the 20 Longhorn Volumes.
Old instance-manager pods are terminated automatically

0 replies

Navigation Menu

old instance-manager pods are not terminated after upgrading (live) engine #13264

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

old instance-manager pods are not terminated after upgrading (live) engine #13264

Uh oh!

jackchuong Jun 5, 2026

Replies: 2 comments · 1 reply

Uh oh!

derekbit Jun 5, 2026 Maintainer

Uh oh!

jackchuong Jun 5, 2026 Author

Uh oh!

jackchuong Jun 8, 2026 Author

jackchuong
Jun 5, 2026

Replies: 2 comments 1 reply

derekbit
Jun 5, 2026
Maintainer

jackchuong Jun 5, 2026
Author

jackchuong
Jun 8, 2026
Author