A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.

Nano vLLM

Mirage Persistent Kernel: Compiling LLMs into a MegaKernel

Accelerate inference without tears

TPU inference for vLLM, with unified JAX and PyTorch support.

NanoGPT (124M) in 3 minutes

Tenstorrent MLIR compiler

🤘 TT-NN operator library, and TT-Metalium low level kernel programming model.

Universal LLM Deployment Engine with ML Compilation

Open Machine Learning Compiler Framework

Efficient Triton Kernels for LLM Training

Blazingly fast LLM inference.

Open-source search and retrieval database for AI applications.

TensorZero is an open-source stack for industrial-grade LLM applications. It unifies an LLM gateway, observability, optimization, evaluation, and experimentation.

Efficent platform for inference and serving local LLMs including an OpenAI compatible API server.

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

A minimal tensor processing unit (TPU), inspired by Google's TPU V2 and V1

A Datacenter Scale Distributed Inference Serving Framework

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

A high-throughput and memory-efficient inference and serving engine for LLMs

Use your Neovim like using Cursor AI IDE!