[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.

This repository provides a comprehensive guide to optimizing GPU kernels for performance, with a focus on NVIDIA GPUs. It covers key tools and techniques such as CUDA, PyTorch, and Triton, aimed at improving computational efficiency for deep learning and scientific computing tasks.

Building Triton and CUDA kernels side-by-side to create a cuBLAS-performant GEMM kernel.

collection of high-performance CUDA implementations, ranging from naive to highly optimized versions.

Learnings and experimentation with GPU programming

A simple example how to write CUDA (C++) kernels in VSCode

A collection of submissions to LeetGPU.