RAT+: Train Dense, Infer Sparse - Recurrence Augmented Attention for Dilated Inference (ICML2026)

The official code of "Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers"

An easy-to-use package for implementing SmoothQuant for LLMs

Ongoing research training transformer language models at scale, including: BERT & GPT-2

The official PyTorch implementation of the NeurIPS2022 (spotlight) paper, Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models

The official PyTorch implementation of the ICLR2022 paper, QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization

Expression language and expression evaluation for Go