Ongoing research training transformer models at scale

Megatron's multi-modal data loader

Collection of best practices, reference architectures, model training examples and utilities to train large models on AWS.

Scripts to customize AWS ParallelCluster

EFA/NCCL base AMI build Packer and CodeBuild/Pipeline files. Also base Docker build files to enable EFA/NCCL in containers

🧑‍🏫 60+ Implementations/tutorials of deep learning papers with side-by-side notes 📝; including transformers (original, xl, switch, feedback, vit, ...), optimizers (adam, adabelief, sophia, ...), ga…