ray-project · kevin85421 · Mar 27, 2023 · Mar 26, 2023 · Mar 26, 2023 · Mar 26, 2023
diff --git a/docs/guidance/tls.md b/docs/guidance/tls.md
@@ -0,0 +1,161 @@
+# TLS Authentication
+
+Ray can be configured to use TLS on its gRPC channels. This means that 
+connecting to the Ray head will require an appropriate 
+set of credentials and also that data exchanged between various processes 
+(client, head, workers) will be encrypted ([Ray's document](https://docs.ray.io/en/latest/ray-core/configure.html?highlight=tls#tls-authentication)).
+
+This document provides detailed instructions for generating a public-private
+key pair and CA certificate for configuring KubeRay.
+
+> Warning: Enabling TLS will cause a performance hit due to the extra
+overhead of mutual authentication and encryption. Testing has shown that 
+this overhead is large for small workloads and becomes relatively smaller
+for large workloads. The exact overhead will depend on the nature of your
+workload.
+
+# Prerequisites
+
+To fully understand this document, it's highly recommended that you have a
+solid understanding of the following concepts:
+
+* private/public key
+* CA (certificate authority)
+* CSR (certificate signing request)
+* self-signed certificate
+
+This [YouTube video](https://youtu.be/T4Df5_cojAs) is a good start.
+
+# TL;DR
+
+> Please note that this document is designed to support KubeRay version 0.5.0 or later. If you are using an older version of KubeRay, some of the instructions or configurations may not apply or may require additional modifications.
+
+> Warning: Please note that the `ray-cluster.tls.yaml` file is intended for demo purposes only. It is crucial that you **do not** store
+your CA private key in a Kubernetes Secret in your production environment.
+
+```sh
+# Install v0.5.0 KubeRay operator
+# `ray-cluster.tls.yaml` will cover from Step 1 to Step 3 (path: kuberay/)
+kubectl apply -f ray-operator/config/samples/ray-cluster.tls.yaml
+
+# Jump to Step 4 "Verify TLS authentication" to verify the connection.
+```
+
+`ray-cluster.tls.yaml` will create:
+* A Kubernetes Secret containing the CA's private key (`ca.key`) and self-signed certificate (`ca.crt`) (**Step 1**)
+* A Kubernetes ConfigMap containing the scripts `gencert_head.sh` and `gencert_worker.sh`, which allow Ray Pods to generate private keys 
+(`tls.key`) and self-signed certificates (`tls.crt`) (**Step 2**)
+* A RayCluster with proper TLS environment variables configurations (**Step 3**)
+
+The certificate (`tls.crt`) for a Ray Pod is encrypted using the CA's private key (`ca.key`). Additionally, all Ray Pods have the CA's public key included in `ca.crt`, which allows them to decrypt certificates from other Ray Pods.
+
+# Step 1: Generate a private key and self-signed certificate for CA
+
+In this document, a self-signed certificate is used, but users also have the
+option to choose a publicly trusted certificate authority (CA) for their TLS
+authentication.
+
+```sh
+# Step 1-1: Generate a self-signed certificate and a new private key file for CA.
+openssl req -x509 \
+            -sha256 -days 3650 \
+            -nodes \
+            -newkey rsa:2048 \
+            -subj "/CN=*.kuberay.com/C=US/L=San Francisco" \
+            -keyout ca.key -out ca.crt
+
+# Step 1-2: Check the CA's public key from the self-signed certificate.
+openssl x509 -in ca.crt -noout -text
+
+# Step 1-3
+# Method 1: Use `cat $FILENAME | base64` to encode `ca.key` and `ca.crt`.
+#           Then, paste the encoding strings to the Kubernetes Secret in `ray-cluster.tls.yaml`.
+
+# Method 2: Use kubectl to encode the certifcate as Kubernetes Secret automatically.
+#           (Note: You should comment out the Kubernetes Secret in `ray-cluster.tls.yaml`.)
+kubectl create secret generic ca-tls --from-file=ca.key --from-file=ca.crt
+```
+* `ca.key`: CA's private key
+* `ca.crt`: CA's self-signed certificate 
+
+This step is optional because the `ca.key` and `ca.crt` files have
+already been included in the Kubernetes Secret specified in [ray-cluster.tls.yaml](../../ray-operator/config/samples/ray-cluster.tls.yaml).
+
+# Step 2: Create separate private key and self-signed certificate for Ray Pods
+
+In [ray-cluster.tls.yaml](../../ray-operator/config/samples/ray-cluster.tls.yaml), each Ray
+Pod (both head and workers) generates its own private key file (`tls.key`) and self-signed
+certificate file (`tls.crt`) in its init container. We generate separate files for each Pod 
+because worker Pods do not have deterministic DNS names, and we cannot use the same 
+certificate across different Pods.
+
+In the YAML file, you'll find a ConfigMap named `tls` that contains two shell scripts: 
+`gencert_head.sh` and `gencert_worker.sh`. These scripts are used to generate the private key 
+and self-signed certificate files (`tls.key` and `tls.crt`) for the Ray head and worker Pods.
+An alternative approach for users is to prebake the shell scripts directly into the docker image that's utilized
+by the init containers, rather than relying on a ConfigMap.
+
+Please find below a brief explanation of what happens in each of these scripts:
+1. A 2048-bit RSA private key is generated and saved as `/etc/ray/tls/tls.key`.
+2. A Certificate Signing Request (CSR) is generated using the private key file (`tls.key`)
+and the `csr.conf` configuration file.
+3. A self-signed certificate (`tls.crt`) is generated using the private key of the 
+Certificate Authority (`ca.key`) and the previously generated CSR.
+
+The only difference between `gencert_head.sh` and `gencert_worker.sh` is the `[ alt_names ]`
+section in `csr.conf` and `cert.conf`. The worker Pods use the fully qualified domain name 
+(FQDN) of the head Kubernetes Service to establish a connection with the head Pod. 
+Therefore, the `[alt_names]` section for the head Pod needs to include the FQDN of the head 
+Kubernetes Service. By the way, the head Pod uses `$POD_IP` to communicate with worker Pods.
+
+```sh
+# gencert_head.sh
+[alt_names]
+DNS.1 = localhost
+DNS.2 = $FQ_RAY_IP
+IP.1 = 127.0.0.1
+IP.2 = $POD_IP
+
+# gencert_worker.sh
+[alt_names]
+DNS.1 = localhost
+IP.1 = 127.0.0.1
+IP.2 = $POD_IP
+```
+
+In [Kubernetes networking model](https://github.com/kubernetes/design-proposals-archive/blob/main/network/networking.md#pod-to-pod), the IP that a Pod sees itself as is the same IP that others see it as. That's why Ray Pods can self-register for the certificates.
+
+# Step 3: Configure environment variables for Ray TLS authentication
+
+To enable TLS authentication in your Ray cluster, set the following environment variables:
+
+- `RAY_USE_TLS`: Either 1 or 0 to use/not-use TLS. If this is set to 1 then all of the environment variables below must be set. Default: 0.
+- `RAY_TLS_SERVER_CERT`: Location of a certificate file which is presented to other endpoints so as to achieve mutual authentication (i.e. `tls.crt`).
+- `RAY_TLS_SERVER_KEY`: Location of a private key file which is the cryptographic means to prove to other endpoints that you are the authorized user of a given certificate (i.e. `tls.key`). 
+- `RAY_TLS_CA_CERT`: Location of a CA certificate file which allows TLS to decide whether an endpoint’s certificate has been signed by the correct authority (i.e. `ca.crt`).
+
+For more information on how to configure Ray with TLS authentication, please refer to [Ray's document](https://docs.ray.io/en/latest/ray-core/configure.html#tls-authentication).
+
+# Step 4: Verify TLS authentication
+
+```sh
+# Log in to the worker Pod
+kubectl exec -it ${WORKER_POD} -- bash
+
+# Since the head Pod has the certificate of $FQ_RAY_IP, the connection to the worker Pods 
+# will be established successfully, and the exit code of the ray health-check command 
+# should be 0.
+ray health-check --address $FQ_RAY_IP:6379
+echo $? # 0
+
+# Since the head Pod has the certificate of $RAY_IP, the connection will fail and an error
+# message similar to the following will be displayed: "Peer name raycluster-tls-head-svc is
+# not in peer certificate".
+ray health-check --address $RAY_IP:6379
+
+# If you add `DNS.3 = $RAY_IP` to the [alt_names] section in `gencert_head.sh`,
+# the head Pod will generate the certificate of $RAY_IP. 
+# 
+# For KubeRay versions prior to 0.5.0, this step is necessary because Ray workers in earlier
+# versions use $RAY_IP to connect with Ray head.
+```