[ENHANCEMENT] harvester-cloud-provider handle the conflict when updating the service object #9019
Description
Is your enhancement request of an existing feature related to a problem? Please describe (👍 if you like this request)
When providing loadbalancer type service on guest cluster, following error & event are easy to see.
I0901 17:08:07.903193 1 event.go:389] "Event occurred" object="default/lb2" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I0901 17:08:08.972151 1 event.go:389] "Event occurred" object="default/lb2" fieldPath="" kind="Service"
apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: update load balancer IP of service default/lb2 failed, error: Operation cannot be fulfilled on services \"lb2\": the object has been modified; please apply your changes to the latest version and try again"
E0901 17:08:08.976388 1 controller.go:303] "Unhandled Error" err="error processing service default/lb2 (retrying with exponential backoff): failed to ensure load balancer: update load balancer IP of service default/lb2 failed, error: Operation cannot be fulfilled on services \"lb2\": the object has been modified; please apply your changes to the latest version and try again" logger="UnhandledError"
The back-and-force re-creating service & LB object just delays the whole processing.
A second bug is also observed:
when failed to get ip from pool, the error log:
I0903 13:29:37.867333 1 event.go:389] "Event occurred" object="default/lb-nginx-5" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: update load balancer IP of service default/lb-nginx-5 failed, error: timeout waiting for IP address, last error:%!w(<nil>)"
E0903 13:29:37.867950 1 controller.go:303] "Unhandled Error" err="error processing service default/lb-nginx-5 (retrying with exponential backoff): failed to ensure load balancer: update load balancer IP of service default/lb-nginx-5 failed, error: timeout waiting for IP address, last error:%!w(<nil>)" logger="UnhandledError"
after the fix, the error log is:
E0903 13:57:48.877676 1 controller.go:303] "Unhandled Error" err="error processing service default/lb-nginx-5 (retrying with exponential backoff): failed to ensure load balancer: update load balancer IP of service default/lb-nginx-5 failed, error: timeout waiting for IP address, last error:ip is not allocated, mode: pool, message: no matched IPPool with requirement &{Network:default/vm-non-tag Project: Namespace:ns-gc-2 Cluster:gc23}" logger="UnhandledError"
I0903 13:57:48.877816 1 event.go:389] "Event occurred" object="default/lb-nginx-5" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: update load balancer IP of service default/lb-nginx-5 failed, error: timeout waiting for IP address, last error:ip is not allocated, mode: pool, message: no matched IPPool with requirement &{Network:default/vm-non-tag Project: Namespace:ns-gc-2 Cluster:gc23}"
Describe the solution you'd like
harvester-cloud-provider adds retry when updating service object in case the return error is IsConflict, to eliminate such unnecessary events/errors.
Describe alternatives you've considered
No response
Additional context
also refer kubernetes/kubernetes#84430 (comment)
No response
Metadata
Metadata
Assignees
Labels
Type
Projects
Status