📚LeetCUDA: Modern CUDA Learn Notes with PyTorch for Beginners🐑, 200+ CUDA Kernels, Tensor Cores, HGEMM, FA-2 MMA.🎉

DeepEP: an efficient expert-parallel communication library

Tile primitives for speedy kernels

This is a series of GPU optimization topics. Here we will introduce how to optimize the CUDA kernel in detail. I will introduce several basic kernel optimizations, including: elementwise, reduce, s…

Flash Attention in ~100 lines of CUDA (forward pass only)

Fast CUDA matrix multiplication from scratch

cuVS - a library for vector search and clustering on the GPU

flash attention tutorial written in python, triton, cuda, cutlass

A simple high performance CUDA GEMM implementation.

Optimizing SGEMM kernel functions on NVIDIA GPUs to a close-to-cuBLAS performance.

Yinghan's Code Sample