Fix PVC mounting issues when 1st server pod is unavailable#1088
Fix PVC mounting issues when 1st server pod is unavailable#1088greenday9 wants to merge 3 commits into
Conversation
|
@leelavg , @amarts , @aravindavk , any thoughts on this change? |
|
Possible to fix lint errors? I'll look at test failures when possible. Change lgtm, however, pls note I could only review & merge but can't cut out a release. |
|
Updated is_server_pod_reachable to be semi-async to reduce blocking Signed-off-by: Kyle Greenwell <kyle.greenwell@comtechtel.com>
Code review comment Co-authored-by: Leela Gangavarapu <leelavg@thoughtexpo.com> Signed-off-by: Kyle Greenwell <kyle.greenwell@comtechtel.com>
3775a8e to
f3b793e
Compare
pylint looks clean now. I don't see any issues except the suppressed C0209(f-string). I would still be in favor of merging even if we can't release although that is preferred. |
Fixed pylint issue
|
@greenday9 could you pls provide a sample test output or an ack that you've tested the fix? I'll merge after an ack as this doesn't create any security issues. I see no responses from others and so thinking to merge PRs which doesn't create security vulnerabilities (by looking at code). |
When the first server pod is unavailable, kadalu is unable to mount with errors like "DeadlineExceeded". This is due to the is_server_pod_reachable code which will iterate over the same server pod multiple times. The original code would try 1st server pod, timeout in 20, sleep 30 seconds, and then retry 4 times. This results in up to 200 seconds to determine that 1st server pod is not reachable before trying the next. This is too long to wait before responding when the issue may be isolated to the 1st server pod.
Given that it appears that the purpose of is_server_pod_reachable is to see if any server pod is available, I rewrote it to function semi-asynchronously. Basically, it will attempt to connect to all server pods at once and see if any succeeds. If none succeeded, then sleep 30 seconds and retry.
What this PR does / why we need it:
It reduces the blocking in is_server_pod_reachable to minimum to determine whether any server pod is reachable. This allows PVC mounting to continue when 1st server is not reachable but other server pods are reachable.
Which issue(s) this PR fixes:
Fixes mounting issues when 1st server pod is unavailable.