Stars
slime is an LLM training framework for RL Scaling.
verl/HybridFlow: A Flexible and Efficient RL Post-Training Framework
LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.
JAX bindings for the flash-attention3 kernels
Flash Attention 2 implementation for Turing GPUs
Implementation of FlashAttention (FA1-FA4) in PyTorch for educational and algorithmic clarity
Implement Flash Attention using Cute.
FP8-Attention: A CUDA-accelerated kernel for efficient transformer attention computation that uses E4M3 FP8 quantization and FlashAttention-style tiling to reduce memory usage and increase inferenc…
This repository contains multiple implementations of Flash Attention optimized with Triton kernels, showcasing progressive performance improvements through hardware-aware optimizations. The impleme…
Code from the "CUDA Crash Course" YouTube series by CoffeeBeforeArch
Best practices for training DeepSeek, Mixtral, Qwen and other MoE models using Megatron Core.
(best/better) practices of megatron on veRL and tuning guide
Kernel Library for Large Headdim Attention (64~1024, BF16/FP8/FP4), 1.5x~15x↑ vs PyTorch SDPA.
An Open-source book with Modern CUDA learning Notes for Beginners - BF16/FP8/FP4, HGEMM, CuTe, Flash-Attention, etc.
Llama causal LM fully recreated in LibTorch. Designed to be used in Unreal Engine 5
This is a code repository for pytorch c++ (or libtorch) tutorial.
Domain-specific language designed to streamline the development of high-performance GPU/CPU/Accelerators kernels
12 Weeks, 24 Lessons, AI for All!
Distributed Compiler and Optimized Parallel Kernels
🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash attention v2.