Good day all!

I have an issue. I have a 3 node k0s cluster running Longhorn. I have 4 longhorn storage class:

Here is my nodes in Longhorn:

My nodes seem to be imbalanced. None of these hosts in this "TEST/QA/DEV" environment will not have external NFS shares for Kubernetes. In my production environment in which I am still standing up, but not yet setup longhorn, still configuring the basics, will have Longhorn Local replication and also NFS, (Test Setup 3; Prod will have 5 worker nodes.) The biggest issue I have right now is that I want to ensure that in this test environment, process that have shared storage are mounting correctly, but if I force the entire solution on one node, it works.

The most common error I get, even with NFS client and no firewall between the hosts turned on, I get this error:

MountVolume.MountDevice failed for volume "pvc-47d244c4-9f18-46e4-beb6-9ce7242e1bee" : rpc error: code = Internal desc = mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o defaults /dev/longhorn/pvc-47d244c4-9f18-46e4-beb6-9ce7242e1bee /var/lib/k0s/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/e6cdaca58c3584cee9760b29ad09c2c50c84665d50709ea4183efa47640c992e/globalmount Output: mount: /var/lib/k0s/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/e6cdaca58c3584cee9760b29ad09c2c50c84665d50709ea4183efa47640c992e/globalmount: /dev/longhorn/pvc-47d244c4-9f18-46e4-beb6-9ce7242e1bee already mounted or mount point busy. dmesg(1) may have more information after failed mount system call. . (9h)

This particular is set to: longhorn-static not even the rwm ones, but happens on all.

My environment in Ubuntu 24.04.3 LTS running k0s latest build (as of today). Longhorn version 1.9.2 is installed via Helm.

I have worked regularly for the past 3 months looking into all these errors, doing deep dive log review, AI, etc. Nothing. I even during this time did a etcd migration from the Bimini image to the standard image without issue.

So my guess is because the volumes replication is not working between the three nodes and once I got to 5 and there is a max of 3 hosts hosting, i am assuming volumes are not replicating correctly on the target node so that it can mount and unmount correctly. I tried disabling, enabling scheduling in longhorn, modify setting, but I am at a lost and asking the community for help!

Update 1: Another error message I get is this:

Multi-Attach error for volume "pvc-47d244c4-9f18-46e4-beb6-9ce7242e1bee" Volume is already exclusively attached to one node and can't be attached to another. (13s)