Stars
A workload for deploying LLM inference services on Kubernetes
Checkpoint-engine is a simple middleware to update model weights in LLM inference engines
Since the emergence of chatGPT in 2022, the acceleration of Large Language Model has become increasingly important. Here is a list of papers on accelerating LLMs, currently focusing mainly on infer…
Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction | A tiny BERT model can tell you the verbosity of an LLM (with low latency overhead!)
Cost-efficient and pluggable Infrastructure components for GenAI inference
The simplest, fastest repository for training/finetuning small-sized VLMs.
GoReplay is an open-source tool for capturing and replaying live HTTP traffic into a test environment in order to continuously test your system with real data. It can be used to increase confidence…
16-fold memory access reduction with nearly no loss
Supercharge Your LLM with the Fastest KV Cache Layer
[IEEE T-PAMI 2024] All you need for End-to-end Autonomous Driving
A sparse attention kernel supporting mix sparse patterns
Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.
A collection of AI Agents papers (Updated biweekly)
Hackable and optimized Transformers building blocks, supporting a composable construction.
[ICCV 2025] Official Repository of VideoLLaMB: Long Video Understanding with Recurrent Memory Bridges
A throughput-oriented high-performance serving framework for LLMs
Our Clone of Orca used for experimentation
Official PyTorch implementation for "Large Language Diffusion Models"
A curated list of state-of-the-art research in embodied AI, focusing on vision-language-action (VLA) models, vision-language navigation (VLN), and related multimodal learning approaches.
A curated reading list of research in Mixture-of-Experts(MoE).
An Easy-to-use, Scalable and High-performance RLHF Framework based on Ray (PPO & GRPO & REINFORCE++ & vLLM & Ray & Dynamic Sampling & Async Agentic RL)
Development repository for the Triton language and compiler
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
My learning notes/codes for ML SYS.
A Datacenter Scale Distributed Inference Serving Framework