Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a toolkit of libraries (Ray AIR) for accelerating ML workloads.

A toolkit to run Ray applications on Kubernetes

C++ based gRPC (C++, Python, Ruby, Objective-C, PHP, C#)

SGLang is a fast serving framework for large language models and vision language models.

veRL: Volcano Engine Reinforcement Learning for LLM

Repository for out-of-tree scheduler plugins based on scheduler framework.

Kubernetes-native Job Queueing

TensorDict is a pytorch dedicated tensor container.

Helm charts for the KubeRay project

Pygloo provides Python bindings for Gloo.

Tracking Ray Enhancement Proposals

Kubernetes IN Docker - local clusters for testing Kubernetes

A comprehensive guide to building RAG-based LLM applications for production.

NGINX Gateway Fabric provides an implementation for the Gateway API using NGINX as the data plane.

Repository for the next iteration of composite service (e.g. Ingress) and load balancing APIs.

Holistic job manager on Kubernetes

SkyPilot: Run LLMs, AI, and Batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed execution—all with a simple interface.

Ray Aviary - evaluate multiple LLMs easily

FlytePropeller is a Kubernetes native operator, that executes Flyte Workflows and Tasks. It has its own kubectl-flyte CLI to interact and is extensible using the flyteplugins/pluginmachinery interface

Kubernetes-native workflow automation platform for complex, mission-critical data and ML processes at scale. It has been battle-tested at Lyft, Spotify, Freenome, and others and is truly open-source.

An open-source ML pipeline development platform

A repository for Kustomize manifests