SGLang is a fast serving framework for large language models and vision language models.

A Datacenter Scale Distributed Inference Serving Framework

TensorZero is an open-source stack for industrial-grade LLM applications. It unifies an LLM gateway, observability, optimization, evaluation, and experimentation.

Genai-bench is a powerful benchmark tool designed for comprehensive token-level performance evaluation of large language model (LLM) serving systems.

A high-throughput and memory-efficient inference and serving engine for LLMs

OME is a Kubernetes operator for enterprise-grade management and serving of Large Language Models (LLMs)

Nano vLLM

KAI Scheduler is an open source Kubernetes Native scheduler for AI workloads at large scale

vLLM’s reference system for K8S-native cluster-wide deployment with community-driven performance optimization