Production-Grade Container Scheduling and Management

A Datacenter Scale Distributed Inference Serving Framework

Next Generation Agentic Proxy for AI Agents and MCP servers

Supercharge Your LLM with the Fastest KV Cache Layer

Kubernetes enhancements for Network Topology Aware Gang Scheduling & Autoscaling

A high-throughput and memory-efficient inference and serving engine for LLMs

A CLI to estimate inference memory requirements for Hugging Face models, written in Python.

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory…

Development repository for the Triton language and compiler

An open and reliable container runtime

Simplified model deployment on llm-d

A simple, performant and scalable Jax LLM!