[BUG] Unable to restore backup directly after recurring job completed on large volumes (due to lock acquisition failure) #4310

Describe the bug

I have a large volume (900GB) size and a large volume data to backup (~600GB), although the diff between the days are not that much, so the backups are fast.

I have a recurring job to backup at 00:05 every day and a 30 day retain window. The restore of the last backups of those recurring jobs are always failing with the following error:
replica longhorn-vol-pv-mongo-r-922370a1 failed the restore: tcp://192.168.67.9:10000: failed to restore backup data nfs://17.15.129.104:/mnt/backup/labrador-production?backup=backup-bf63752a6336485a&volume=longhorn-vol-pv-mongo to snapshot file volume-snap-33c67d9a-69ba-495d-b318-1c0c9956691a.img: rpc error: code = Unknown desc = error starting backup restore: error initiating full backup restore: failed lock backupstore/volumes/fe/ff/longhorn-vol-pv-mongo/locks/lock-ee3e1786db844cc6.lck type 1 acquisition

The restore of a manual backup always works.

To Reproduce

Steps to reproduce the behavior:

1. Have a large enough volume and dataset (~900GB volume, ~600GB data)
2. Scheduled backup job with retain window of 30 or more.
3. Try to restore a backup from this scheduled job
4. Backup restore fails

Expected behavior

Backup restore shall succeed.

Log or Support bundle

longhorn-support-bundle_e0e5ebb7-f8cb-4c34-ae59-3068868aaaba_2022-07-28T14-23-38Z.zip

Environment

• Longhorn version: 1.2.4
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Kubectl
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: custom install via Kubespray (1.17). Kubernetes version: 1.21.5
  • Number of management node in the cluster: 3
  • Number of worker node in the cluster: 3
• Node config
  • OS type and version: Ubuntu 20.04
  • CPU per node: 8
  • Memory per node: 16GB
  • Disk type(e.g. SSD/NVMe): SSD
  • Network bandwidth between the nodes: ?
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): VMs (OpenStack)
• Number of Longhorn volumes in the cluster: 2

Additional context