An extension of the nanoGPT repository for training small MOE models.

Ongoing research training transformer models at scale

Pact improves Pi by retaining recent messages when compacting.

A PyTorch native platform for training generative AI models

PyTorch building blocks for the OLMo ecosystem

Autonomous experiment loop extension for pi

This library empowers users to seamlessly port pretrained models and checkpoints on the HuggingFace (HF) hub (developed using HF transformers library) into inference-ready formats that run efficien…

🐳 Efficient Triton implementations for "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention"

NanoGPT (124M) in 90 seconds

Speedrunning LoRA fine-tuning: frozen task, frozen hardware, public wall-clock leaderboard. modded-nanogpt for fine-tuning.

Tiered optimizer state allocation for memory-efficient MoE training. Cuts optimizer memory by 97.4%, outperforming AdamW/Muon/Lion while fitting a 6.78B MoE on a single 40GB GPU.

Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793

Code for the signSGD paper

🦁 Lion, new optimizer discovered by Google Brain using genetic algorithms that is purportedly better than Adam(w), in Pytorch

TokenSpeed is a speed-of-light LLM inference engine.

Root Mean Square Layer Normalization

Repo for "Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture"

Building blocks for foundation models.

[ICLR2025] Codebase for "ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing", built on Megatron-LM.

Official code for UnICORNN (ICML 2021)

Achieve state of the art inference performance with modern accelerators on Kubernetes

DeepSpec: a full-stack codebase for training and evaluating speculative decoding algorithms

A dynamic binary instrumentation tool for tracing and analyzing CUDA kernel instructions.

🎬 3.7× faster video generation E2E 🖼️ 1.6× faster image generation E2E ⚡ ColumnSparseAttn 9.3× vs FlashAttn‑3 💨 ColumnSparseGEMM 2.5× vs cuBLAS

Causal depthwise conv1d in CUDA, with a PyTorch interface

Ongoing research training transformer language models at scale, including: BERT & GPT-2

[AAAI 2026] Official implementation of "FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models". If you find this repository helpful, please consider starring 🌟 it to support the p…

Official repository of the xLSTM.