Ulysses sequence-parallel all-to-all as a torch custom op, moved by the GPU copy engines into torch symmetric memory. Zero SM usage; 1.66-2.17x over torch.distributed on NVLink.

Agent Substrate: the core system

K8S on FoundationDB

Tensors and Dynamic neural networks in Python with strong GPU acceleration

High Performance LLM Inference Operator Library

LoRAFusion: Efficient LoRA Fine-Tuning for LLMs

Supercharge Your Model Training

Accelerating MoE with IO and Tile-aware Optimizations

Training API and CLI

An experimental implementation of compiler-driven automatic sharding of models across a given device mesh.

A storage solution for PyTorch tensors with distributed tensor support.

[ICLR 2026]QeRL enables RL for 32B LLMs on a single H100 GPU.

https://github.com/eunomia-bpf homepage, documents and blogs

Perplexity GPU Kernels

slime is an LLM post-training framework for RL Scaling.

Cosmos-RL is a flexible and scalable Reinforcement Learning framework specialized for Physical AI applications.

Official implementation of "Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding"

Exploring context retrieval strategies for code editing

A std::execution style runtime context and High Performance RPC Transport for using OpenUCX. Including CUDA/ROCM/... devices with RDMA.

Thread-safe asyncio-aware queue for Python

Scalable toolkit for efficient model reinforcement

[DEPRECATED] Moved to ROCm/rocm-libraries repo

The largest collection of PyTorch image encoders / backbones. Including train, eval, inference, export scripts, and pretrained weights -- ResNet, ResNeXT, EfficientNet, NFNet, Vision Transformer (V…

TensorDict is a pytorch dedicated tensor container.

Research and development for optimizing transformers

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

Fast Multi-dimensional Sparse Attention

🤗 Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch.