Efficient Long-context Language Model Training by Core Attention Disaggregation

[ASPLOS'25] Towards End-to-End Optimization of LLM-based Applications with Ayo

Official Repo for Open-Reasoner-Zero

S-LoRA: Serving Thousands of Concurrent LoRA Adapters

Microsoft Automatic Mixed Precision Library

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

Fast and memory-efficient exact attention

A repo for distributed training of language models with Reinforcement Learning via Human Feedback (RLHF)

The simplest, fastest repository for training/finetuning medium-sized GPTs.

Pipeline Parallelism for PyTorch

Intel® Performance Counter Monitor (Intel® PCM)

A multi-party collaborative machine learning framework

Ongoing research training transformer models at scale

A PyTorch implementation of the Transformer model in "Attention is All You Need".

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

A lightweight parameter server interface

A high performance and generic framework for distributed DNN training

Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.