Ongoing research training transformer language models at scale, including: BERT & GPT-2

Fast and memory-efficient exact attention

Transformers at any scale

DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.

An open source framework that provides a simple, universal API for building distributed applications. Ray is packaged with RLlib, a scalable reinforcement learning library, and Tune, a scalable hyp…

PyTorch differentiable Multi-Scale Structural Similarity (MS-SSIM) loss