Documentation · Blog · White paper · X · Slack
AIBrix is a Kubernetes control plane for production LLM inference. You bring the engine — vLLM, SGLang, or TensorRT-LLM — and AIBrix routes each request from live engine state, scales on inference metrics, and manages models, LoRA adapters, and KV cache.
Use it when you run many models or replicas, mix GPU types, or need to hold cost and latency targets.
| Capability | What it does |
|---|---|
| Gateway and routing | OpenAI-compatible API. Picks a pod from live engine state, including prefix-cache, load, and prefill/decode placement, with session affinity for agentic workloads. |
| Multi-engine | Serve vLLM, SGLang, and TensorRT-LLM behind one gateway. |
| LoRA | Pack many low-rank adapters onto shared base-model pods. |
| Model lifecycle | Declare models with ModelClaim, and preload images and weights with ModelWarmup so scale-up is faster. |
| Autoscaling | Scale replicas on engine metrics such as queue depth, KV cache, and tokens. |
| Distributed inference | Run multi-node and multi-role deployments, including prefill/decode, with StormService. |
| Distributed KV cache | Offload KV cache and reuse it across engines. |
| Heterogeneous GPUs | Mix GPU types and still target cost and SLO. Experimental. |
| Runtime | Sidecar that downloads weights, loads and unloads adapters, and normalizes engine metrics. |
Requests hit an OpenAI-compatible gateway and go straight to an engine pod. Kubernetes controllers decide what should be running: model lifecycle, warmup, multi-node topology, autoscaling, and observability. The architecture guide walks through both planes.
These commands install AIBrix, including its Envoy Gateway dependency, into the cluster your current kubectl context points at. KubeRay is included and is only required for Ray-based multi-node inference.
The snippet uses v0.7.0, the latest published release, so the downloads succeed today. This repository’s manifests track the v0.8.0-rc.1 release candidate; that GitHub release is still a draft, and its asset URLs return 404 until it is published. Switch VERSION when it is.
VERSION=v0.7.0
# Dependencies (Envoy Gateway, KubeRay, and friends)
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/${VERSION}/aibrix-dependency-${VERSION}.yaml" --server-side
# CRDs ship separately so uninstalling the operator does not delete user resources
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/${VERSION}/aibrix-core-crds-${VERSION}.yaml" --server-side
# Controllers, gateway plugins, and supporting services
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/${VERSION}/aibrix-core-${VERSION}.yaml"
kubectl get pods -n aibrix-systemWhen the pods are ready, deploy a model and call the gateway. The quickstart has a base-model manifest and the OpenAI-compatible curl commands. Helm install steps are in the installation guide.
Images in config/default are tagged nightly.
git clone https://github.com/vllm-project/aibrix.git
cd aibrix
kubectl apply -k config/dependency --server-side
kubectl apply -k config/crd --server-side
kubectl apply -k config/defaultFull guides live at aibrix.readthedocs.io.
| You want to | Start here |
|---|---|
| Serve a first model | Overview, then the quickstart |
| Make a running deployment cheaper | Autoscaling, LoRA, KV cache, observability |
| Hit scale or latency targets | Prefill/decode, multi-node inference, heterogeneous GPUs |
- [2026-10-09] v0.8.0-rc.1 release candidate.
- [2026-06-16] v0.7.0 · blog
- [2026-03-05] v0.6.0 · blog
- [2025-11-10] v0.5.0 · blog
Older releases
- [2025-11-12] KubeCon North America keynote: Kubernetes-native GenAI Inference Infrastructure
- [2025-06-10] KubeCon China: Cost-effective Kubernetes control plane for vLLM
- [2025-04-04] KubeCon EU keynote, with Google: LLM-aware load balancing in Kubernetes
- [2025-03-30] ASPLOS'25 workshop: Large-scale LLM inference infrastructure for systems research
Contributions are welcome. Read the contributing guide, talk in #aibrix on the vLLM Slack, or email maintainers@aibrix.ai.
Bugs and feature requests go on GitHub Issues. AIBrix is licensed under Apache 2.0.