Replies: 5 comments 7 replies
-
|
I assume the live replica on the drained node should become error state and failed, so the volume controller will request to create a new replica based on https://longhorn.io/docs/1.4.0/references/settings/#replica-replenishment-wait-interval, and see if there is a failed replica able to reuse. This is a quite general process w/o considering an intended failed replica caused by the drain. Probably we should have a gradual case for intended draining for better space efficiency. @shuo-wu @derekbit @PhanLe1010 any thoughts? |
Beta Was this translation helpful? Give feedback.
-
That's why we launch setting |
Beta Was this translation helpful? Give feedback.
-
|
I think I am also seeing the same...
The whole process took probably less than 5min. After the node came back online all the replicas where marked as failed. That is expected but they where ALL moved away and re-built on another node. I am not really sure if this is expected or not in the end. |
Beta Was this translation helpful? Give feedback.
-
https://longhorn.io/docs/1.7.2/maintenance/maintenance/#updating-the-node-os-or-container-runtime Longhorn will just do replica rebuilding if it can't reuse failed replicas from the maintenance node.
|
Beta Was this translation helpful? Give feedback.
-
|
Yes I read that, but why is not re-using the replicas is my question. :) Not the end of the world but not idea either :) |
Beta Was this translation helpful? Give feedback.
-
|
@innobead @shuo-wu We are anecdotally seeing this on a SUSE Storage supported customer. We will log a case with SUSE Support and move from there but I wanted to check in and see if there has been any progress on this? Replica Replenish Wait Interval is set correctly but the same behaviour is occurring. |
Beta Was this translation helpful? Give feedback.
-
What's the value you set? Could you elaborate more on your steps to this issue? |
Beta Was this translation helpful? Give feedback.
-
|
We have 600 set (default) and have not adjusted the default settings. Rebooting, for example when patching a node, is enough to trigger the behaviour. |
Beta Was this translation helpful? Give feedback.
-
|
How long did you take to patch and reboot a node? |
Beta Was this translation helpful? Give feedback.
-
|
Less than 600 seconds. |
Beta Was this translation helpful? Give feedback.
-
|
@davidalexander83 i didn't reload my Talos nodes since I commented here last time but I had this issues I think since the beginning of my hang Longhorn with version 1.7.something. I have also the 600s time out and have daily snapshots also configured so in theory it should work but every time is a full reload. And definitely doesn't take 600 seconds to reboot a node. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Related to #1958
When i.e. kured prepares a machine for a short maintenance restart, it will properly use the drain feature. However, this will cause longhorn ALWAYS to rebuild replicas, even so this is completely unnecessary.
The handling for this is just not well implemented, is there any way to instruct longhorn to REUSE the existing replica? Longhorn must be able to handle such normal and easy operations, unless you as the creators have a secret plan to destroy the lifetime of all the SSDs out there, in this case yes this makes totally sense!
Beta Was this translation helpful? Give feedback.
All reactions