A high-throughput and memory-efficient inference and serving engine for LLMs

The source of LMSYS website and blogs

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

Benchmark SGLang on SLURM

verl: Volcano Engine Reinforcement Learning for LLMs

Utilities intended for use with Llama models.

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. Tensor…

Development repository for the Triton language and compiler

SGLang is a high-performance serving framework for large language models and multimodal models.

Agent Zero AI framework

An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).

Puzzles for learning Triton

Fast and memory-efficient exact attention

MLX: An array framework for Apple silicon

CUDA Templates and Python DSLs for High-Performance Linear Algebra

Building a quick conversation-based search demo with Lepton AI.

so-vits-svc fork with realtime support, improved interface and more features.

[DEPRECATED] Moved to ROCm/rocm-libraries repo. NOTE: develop branch is maintained as a read-only mirror

AITemplate is a Python framework which renders neural network into high performance CUDA/HIP C++ code. Specialized for FP16 TensorCore (NVIDIA GPU) and MatrixCore (AMD GPU) inference.

High-efficiency floating-point neural network inference operators for mobile, server, and Web

Repo for counting stars and contributing. Press F to pay respect to glorious developers.

Tensors and Dynamic neural networks in Python with strong GPU acceleration

Quantized Neural Network PACKage - mobile-optimized implementation of quantized neural network operators

Open deep learning compiler stack for cpu, gpu and specialized accelerators

Open Machine Learning Compiler Framework

Quantized Neural Network PACKage - mobile-optimized implementation of quantized neural network operators