LLM training in simple, raw C/CUDA

DeepEP: an efficient expert-parallel communication library

📚LeetCUDA: Modern CUDA Learn Notes with PyTorch for Beginners🐑, 200+ CUDA Kernels, Tensor Cores, HGEMM, FA-2 MMA.🎉

DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling

FlashInfer: Kernel Library for LLM Serving

NCCL Tests

This is a series of GPU optimization topics. Here we will introduce how to optimize the CUDA kernel in detail. I will introduce several basic kernel optimizations, including: elementwise, reduce, s…

CUDA Data Parallel Primitives Library

Optimizing SGEMM kernel functions on NVIDIA GPUs to a close-to-cuBLAS performance.