FlashInfer: Kernel Library for LLM Serving

[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.

Flash Attention in ~100 lines of CUDA (forward pass only)

[ICML2025] SpargeAttention: A training-free sparse attention that accelerates any model inference.

An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).

CPM.cu is a lightweight, high-performance CUDA implementation for LLMs, optimized for end-device inference and featuring cutting-edge techniques in sparse architecture, speculative sampling and qua…

Quantized Attention that achieves speedups of 2.1-3.1x and 2.7-5.1x compared to FlashAttention2 and xformers, respectively, without lossing end-to-end metrics across various models.

Sage attention for turning.