A unified inference runtime for VLA models.

TokenSpeed is a speed-of-light LLM inference engine.

Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing,…

AlpaSim is an open-source autonomous vehicle simulation platform designed for development and testing of end-to-end AV policies

A safetensors extension to efficiently store sparse quantized tensors on disk

Flash Vision-Language-Action Inference for Autonomous Driving

CLI proxy that reduces LLM token consumption by 60-90% on common dev commands. Single Rust binary, zero dependencies

Open Source AI Platform - AI Chat with advanced features that works with every LLM

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

DSPy: The framework for programming—not prompting—language models

ARIS ⚔️ (Auto-Research-In-Sleep) — Lightweight Markdown-only skills for autonomous ML research: cross-model review loops, idea discovery, and experiment automation. No framework, no lock-in — works…

Common recipes to run vLLM

Offline optimization of your disaggregated Dynamo graph

A Datacenter Scale Distributed Inference Serving Framework

Distributed MoE in a Single Kernel [NeurIPS '25]

High-Performance KV Cache Storage Engine on CXL Shared Memory for LLM Inference

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

A collection of prompts, system prompts and LLM instructions

Extracted system prompts from Anthropic - Claude Fable 5, Opus 5, Claude Design, Claude Code. OpenAI - ChatGPT GPT-5.6-Sol, Codex. Google - Gemini 3.5 Flash, 3.1 Pro, Antigravity. xAI - Grok, Curso…

Causal depthwise conv1d in CUDA, with a PyTorch interface

🚀 Efficient implementations for emerging model architectures

Material for gpu-mode lectures

An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)

TPU inference for vLLM, with unified JAX and PyTorch support.

Easy, Fast, and Scalable Multimodal AI

[NeurIPS'25 Oral] Query-agnostic KV cache eviction: 3–4× reduction in memory and 2× decrease in latency (Qwen3/2.5, Gemma3, LLaMA3)

A curriculum for learning about gpu performance engineering, from scratch to what the frontier AI labs do

DeepEP: an efficient expert-parallel communication library