Domain-specific language designed to streamline the development of high-performance GPU/CPU/Accelerators kernels

Kernels, of the mega variety :)

基金投资管理回测引擎

CUDA Templates for Linear Algebra Subroutines

Fast and memory-efficient exact attention

Mirage Persistent Kernel: Compiling LLMs into a MegaKernel

A fast communication-overlapping library for tensor/expert parallelism on GPUs.

Open Machine Learning Compiler Framework

Universal LLM Deployment Engine with ML Compilation

SGLang is a fast serving framework for large language models and vision language models.

NVIDIA NVSHMEM is a parallel programming interface for NVIDIA GPUs based on OpenSHMEM. NVSHMEM can significantly reduce multi-process communication and coordination overheads by allowing programmer…

FlashInfer: Kernel Library for LLM Serving

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and support state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorR…

Quantized Attention achieves speedup of 2-5x and 3-11x compared to FlashAttention and xformers, without lossing end-to-end metrics across language, image, and video models.

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

TinyNeuralNetwork is an efficient and easy-to-use deep learning model compression framework.

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

A high-throughput and memory-efficient inference and serving engine for LLMs

Infrastructure to enable deployment of ML models to low-power resource-constrained embedded targets (including microcontrollers and digital signal processors).