Why are these changes needed?

This PR addresses the following questions:

1. KubeRay is aggressive in deleting running Pods

Without this PR, the RayFTTestCase test is somewhat flaky. When GCS fault tolerance is enabled, KubeRay will inject both liveness and readiness probes into both the head and worker Pods. However, KubeRay will delete the Pod immediately if it receives an unhealthy event from the probes no matter the value of failureThreshold. Hence, it is very easy to trigger the deletion of running Pods, which is not a good idea from any perspective. In CI, the initialDelaySeconds of the probes is 30 seconds, but the GCS server may take more than 30 seconds to be ready. Hence, sometimes the probe will create an unhealthy event if the GCS server isn't ready in time. KubeRay will then delete this head Pod and create a new one in the next reconcile loop.

• Example CI failure
  

2. KubeRay listens to "all unhealthy events" in the Kubernetes cluster.

Without this PR, KubeRay listens to "all unhealthy events" in the Kubernetes cluster, regardless of whether they are related to KubeRay or not. This may cause OOM if there are a lot of unhealthy events in the Kubernetes cluster, and see #601 (comment) for more details.

3. Undefined lifecycles of "Failed" and "Succeeded" Pods

KubeRay does not define the lifecycle of Pods in terminated states (i.e., "Failed" or "Succeeded"). To be more specific, for the head Pod, KubeRay will neither delete nor create a new one when it fails. For the worker Pod, KubeRay will not delete the failed one but will create a new worker Pod.

• Example 1: restartPolicy: Never, no Ray Autoscaler, delete the ray start process in the head Pod

  • The head Pod's status will change to "Failed" because the container's primary process is killed, and the failed Pod will remain there indefinitely. Subsequently, the worker Pod will also fail because it cannot connect to the GCS beyond a certain threshold.

• Example 2: Same gist as example 1. In this example, we delete the ray start process in the worker Pod. The failed worker Pod will not be deleted, and a new worker Pod will be created.

• Example 3: restartPolicy: Never, add command: ["exit 1"] to the worker's spec.

  • The worker Pod will fail automatically. The KubeRay operator will then create a new worker Pod, which will also fail, leading to the creation of another worker Pod, and so on. The loop will repeat forever and eat all resources in the Kubernetes cluster.

Behavior of this PR

• If a Pod is able to restart, KubeRay will not delete it. On the other hand, KubeRay only deletes Pods which are in terminated states (i.e., "Failed" or "Succeeded") and not able to restart (i.e., restartPolicy is not Always).
• KubeRay will not try to delete any "Running" Pods at this moment.
• KubeRay does not listen to Kubernetes events anymore.

Follow up

• Good default values for probes
• Consider the case with sidecar containers
• Delete Pods by the value of restartCount

Related issue number

I closed most issues that were used to track the progress of the flakiness of GCS FT tests.

Closes #1144
Closes #618
Closes #990
Closes #638
Closes #691
Closes #692

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

• I used the following script to run the fault tolerance test 50 times on my devbox. It passed 50 times and failed 0 times.

  #!/bin/bash
  for iter in {1..50}
  do
      RAY_IMAGE=rayproject/ray:2.5.0 OPERATOR_IMAGE=controller:latest python3 tests/compatibility-test.py RayFTTestCase 2>&1 | tee logs/log_${iter} 
  done

• Verify: for i in {1..50}; do tail -n 1 log_${i}; done | grep OK | wc -l => 50