Replica rebuilding may constantly fail with the error message mentioned in the title.
The replica server should be closed then the process should be removed after rebuilding failure
[longhorn-manager-n4xnc] time="2022-04-07T12:08:06Z" level=error msg="Failed rebuilding of replica 10.42.2.142:10000" controller=longhorn-engine error="failed to add replica address='tcp://10.42.2.142:10000' to controller 'vol': failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.2.2/longhorn [--url 10.42.4.123:10000 add tcp://10.42.2.142:10000], output , stderr, time=\"2022-04-07T12:08:06Z\" level=info msg=\"Adding replica tcp://10.42.2.142:10000 in WO mode\"\ntime=\"2022-04-07T12:08:06Z\" level=fatal msg=\"Error running add replica command: failed to create replica tcp://10.42.2.142:10000 for volume 10.42.4.123:10000: rpc error: code = Unknown desc = Too many snapshots created\"\n, error exit status 1" node=shuo-k8s-worker-1 volume=vol
[longhorn-manager-n4xnc] time="2022-04-07T12:08:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"vol-e-c2378df3\", UID:\"68e8592d-7fae-49cb-9229-f1d7f8e49106\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2013755\", FieldPath:\"\"}): type: 'Warning' reason: 'FailedRebuilding' Failed rebuilding replica with Address 10.42.2.142:10000: failed to add replica address='tcp://10.42.2.142:10000' to controller 'vol': failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.2.2/longhorn [--url 10.42.4.123:10000 add tcp://10.42.2.142:10000], output , stderr, time=\"2022-04-07T12:08:06Z\" level=info msg=\"Adding replica tcp://10.42.2.142:10000 in WO mode\"\ntime=\"2022-04-07T12:08:06Z\" level=fatal msg=\"Error running add replica command: failed to create replica tcp://10.42.2.142:10000 for volume 10.42.4.123:10000: rpc error: code = Unknown desc = Too many snapshots created\"\n, error exit status 1"
[longhorn-manager-n4xnc] time="2022-04-07T12:08:06Z" level=info msg="Removed failed rebuilding replica 10.42.2.142:10000" controller=longhorn-engine node=shuo-k8s-worker-1 volume=vol
[longhorn-manager-n4xnc] time="2022-04-07T12:08:06Z" level=info msg="Engine is still in backoff for replica vol-r-4541122b rebuild failure" controller=longhorn-engine node=shuo-k8s-worker-1 volume=vol
[longhorn-manager-n4xnc] time="2022-04-07T12:08:36Z" level=warning msg="Error syncing Longhorn engine" controller=longhorn-engine engine=longhorn-system/vol-e-c2378df3 error="fail to sync engine for longhorn-system/vol-e-c2378df3: fail to start rebuild for vol-r-4541122b of vol-e-c2378df3: timed out waiting for the condition" node=shuo-k8s-worker-1
[longhorn-manager-n4xnc] time="2022-04-07T12:08:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"vol-e-c2378df3\", UID:\"68e8592d-7fae-49cb-9229-f1d7f8e49106\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2013755\", FieldPath:\"\"}): type: 'Normal' reason: 'Rebuilding' Start rebuilding replica vol-r-4541122b with Address 10.42.2.142:10000 for normal engine vol"
[longhorn-manager-n4xnc] time="2022-04-07T12:08:36Z" level=error msg="Failed rebuilding of replica 10.42.2.142:10000" controller=longhorn-engine error="failed to add replica address='tcp://10.42.2.142:10000' to controller 'vol': failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.2.2/longhorn [--url 10.42.4.123:10000 add tcp://10.42.2.142:10000], output , stderr, time=\"2022-04-07T12:08:36Z\" level=info msg=\"Adding replica tcp://10.42.2.142:10000 in WO mode\"\ntime=\"2022-04-07T12:08:36Z\" level=fatal msg=\"Error running add replica command: failed to create replica tcp://10.42.2.142:10000 for volume 10.42.4.123:10000: rpc error: code = Unknown desc = Replica must be closed, Can not add in state: open\"\n, error exit status 1" node=shuo-k8s-worker-1 volume=vol
[longhorn-manager-n4xnc] time="2022-04-07T12:08:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"vol-e-c2378df3\", UID:\"68e8592d-7fae-49cb-9229-f1d7f8e49106\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2013755\", FieldPath:\"\"}): type: 'Warning' reason: 'FailedRebuilding' Failed rebuilding replica with Address 10.42.2.142:10000: failed to add replica address='tcp://10.42.2.142:10000' to controller 'vol': failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.2.2/longhorn [--url 10.42.4.123:10000 add tcp://10.42.2.142:10000], output , stderr, time=\"2022-04-07T12:08:36Z\" level=info msg=\"Adding replica tcp://10.42.2.142:10000 in WO mode\"\ntime=\"2022-04-07T12:08:36Z\" level=fatal msg=\"Error running add replica command: failed to create replica tcp://10.42.2.142:10000 for volume 10.42.4.123:10000: rpc error: code = Unknown desc = Replica must be closed, Can not add in state: open\"\n, error exit status 1"
[longhorn-manager-n4xnc] time="2022-04-07T12:08:36Z" level=info msg="Removed failed rebuilding replica 10.42.2.142:10000" controller=longhorn-engine node=shuo-k8s-worker-1 volume=vol
[longhorn-manager-n4xnc] time="2022-04-07T12:08:36Z" level=info msg="Engine is still in backoff for replica vol-r-4541122b rebuild failure" controller=longhorn-engine node=shuo-k8s-worker-1 volume=vol
Describe the bug
Replica rebuilding may constantly fail with the error message mentioned in the title.
Actually, the root cause is that the too-many-snapshot check blocks and fails the rebuilding. Then the replica server is not closed by the engine as usual, (and the replica server close call here does not work). Besides, the replica process itself won't be stopped (by the longhorn manager). Next time when the longhorn manager asks the engine to retry the rebuilding, the engine will detect that the replica server is in the invalid state
openat the beginning.To Reproduce
Steps to reproduce the behavior:
Expected behavior
The replica server should be closed then the process should be removed after rebuilding failure
Log or Support bundle
Environment
Additional context
N/A