Stars
Code for the paper "Language Models are Unsupervised Multitask Learners"
[ICLR 2025] Official PyTorch Implementation of Gated Delta Networks: Improving Mamba2 with Delta Rule
Collection of scripts and notebooks for OpenAI's latest GPT OSS models
Flash Attention Triton kernel with support for second-order derivatives
Utility scripts for PyTorch (e.g. Make Perfetto show some disappearing kernels, Memory profiler that understands more low-level allocations such as NCCL, ...)
rl from zero pretrain, can it be done? yes.
Trainable fast and memory-efficient sparse attention
gpt-oss-120b and gpt-oss-20b are two open-weight language models by OpenAI
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
Distribute and run LLMs with a single file.
FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.
Agent Reinforcement Trainer: train multi-step agents for real-world tasks using GRPO. Give your agents on-the-job training. Reinforcement learning for Qwen2.5, Qwen3, Llama, and more!
[Up-to-date] Awesome Agentic Deep Research Resources
verl-agent is an extension of veRL, designed for training LLM/VLM agents via RL. verl-agent is also the official code for paper "Group-in-Group Policy Optimization for LLM Agent Training"
Copy-paste Liquid Glass shader with SVG
Pipeline Parallelism Emulation and Visualization
A bidirectional pipeline parallelism algorithm for computation-communication overlap in DeepSeek V3/R1 training.
General Reasoner: Advancing LLM Reasoning Across All Domains [NeurIPS25]
Recreating PyTorch from scratch (C/C++, CUDA, NCCL and Python, with multi-GPU support and automatic differentiation!)
depyf is a tool to help you understand and adapt to PyTorch compiler torch.compile.
Leaderboard Comparing LLM Performance at Producing Hallucinations when Summarizing Short Documents
[ICLR2025 Spotlight] SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models
Explore the Multimodal “Aha Moment” on 2B Model