Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

Tile primitives for speedy kernels

DLSlime: Flexible & Efficient Heterogeneous Transfer Toolkit

MSCCL++: A GPU-driven communication stack for scalable AI applications

A throughput-oriented high-performance serving framework for LLMs

A guidance language for controlling large language models.

Venus Collective Communication Library, supported by SII and Infrawaves.

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Seamless operability between C++11 and Python

Efficient Compute-Communication Overlap for Distributed LLM Inference

SGLang is a fast serving framework for large language models and vision language models.

ArcticInference: vLLM plugin for high-throughput, low-latency inference

Domain-specific language designed to streamline the development of high-performance GPU/CPU/Accelerators kernels

A high-throughput and memory-efficient inference and serving engine for LLMs

注释的nano_vllm仓库，并且完成了MiniCPM4的适配以及注册新模型的功能

Official inference repo for FLUX.1 models

An AI agent development platform with all-in-one visual tools, simplifying agent creation, debugging, and deployment like never before. Coze your way to AI Agent creation.

Next-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation…

Nano vLLM

rocSHMEM intra-kernel networking runtime for AMD dGPUs on the ROCm platform.

Distributed Triton for Parallel Systems

Perplexity GPU Kernels

Disaggregated serving system for Large Language Models (LLMs).

extensible collectives library in triton

This package contains the original 2012 AlexNet code.

A fast communication-overlapping library for tensor/expert parallelism on GPUs.

DeepEP: an efficient expert-parallel communication library