High Performance LLM Inference Operator Library

Big & Small LLMs working together

A unified inference and post-training framework for accelerated video generation.

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

Tile primitives for speedy kernels

CUDA Templates and Python DSLs for High-Performance Linear Algebra

CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning

verl: Volcano Engine Reinforcement Learning for LLMs

DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling

Block Diffusion for Ultra-Fast Speculative Decoding

ArcticInference: vLLM plugin for high-throughput, low-latency inference

DeepEP: an efficient expert-parallel communication library

Ongoing research training transformer models at scale

A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Sionna Research Kit: A GPU-Accelerated Research Platform for AI-RAN

FlashMLA: Efficient Multi-head Latent Attention Kernels

The simplest, fastest repository for training/finetuning medium-sized GPTs.

Nano vLLM

Obsidian plugin to create custom progress bars

Disaggregated serving system for Large Language Models (LLMs).

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.