DeepSpeed Accelerator Abstraction Interface
Contents Contents Introduction Write accelerator agnostic models Port accelerator runtime calls Port accelerator device name Te...
Tutorials
DeepSpeed Accelerator Setup Guides
Contents Contents Introduction Intel Architecture (IA) CPU Intel XPU Huawei Ascend NPU Intel Gaudi Apple Silicon (MPS)
Installation Details
The quickest way to get started with DeepSpeed is via pip, this will install the latest release of DeepSpeed which is not tied to specific PyTorch or CUDA ve...
Automatic Tensor Parallelism for HuggingFace Models
Note: This tutorial covers AutoTP for inference. For training with tensor parallelism and ZeRO optimization, see Automatic Tensor Parallelism (Training).
Automatic Tensor Parallelism (Training)
This tutorial covers Automatic Tensor Parallelism for combining tensor parallelism with ZeRO optimization during training. For inference-only tensor parallel...
Autotuning
Automatically discover the optimal DeepSpeed configuration that delivers good training speed
Getting Started with DeepSpeed on Azure
This tutorial will help you get started with DeepSpeed on Azure.
CIFAR-10 Tutorial
Train your first model with DeepSpeed!
Communication Logging
Log all DeepSpeed communication calls
DeepSpeed Data Efficiency: A composable library that makes better use of data, increases training efficiency, and improves model quality
What is DeepSpeed Data Efficiency: DeepSpeed Data Efficiency is a library purposely built to make better use of data, increases training efficiency, and impr...
DataStates-LLM Checkpointing Engine
This tutorial will show how to use DataStates-LLM for asynchronous checkpointing. DataStates-LLM introduces a lazy asynchronous checkpointing mechanism tailo...
DeepNVMe
This tutorial will show how to use DeepNVMe for data transfers between persistent storage and tensors residing in host or device memory. DeepNVMe improves th...
Domino
Domino achieves near-complete communication hiding behind computation for tensor parallel training. Please find our Domino-tutorial in DeepSpeedExample repo.
Getting Started with DeepSpeed-Ulysses for Training Transformer Models with Extreme Long Sequences
In this tutorial we describe how to enable DeepSpeed-Ulysses for Megatron-Deepspeed. DeepSpeed-Ulysses is a simple but highly communication and memory effici...
DS4Sci_EvoformerAttention eliminates memory explosion problems for scaling Evoformer-centric structural biology models
1. What is DS4Sci_EvoformerAttention DS4Sci_EvoformerAttention is a collection of kernels built to scale the Evoformer computation to larger number of sequen...
Flops Profiler
Measure the parameters, latency, and floating-point operations of your model
DCGAN Tutorial
Train your first GAN model with DeepSpeed!
Getting Started
First steps with DeepSpeed
Hybrid Engine OPT cache compatibility
Hybrid Engine’s injected OPT layers implement the legacy tuple-cache interface. OPT decoder layers exposing cache_position or past_key_values use a newer cac...
Getting Started with DeepSpeed for Inferencing Transformer based Models
DeepSpeed-Inference v2 is here and it’s called DeepSpeed-FastGen! For the best performance, latest features, and newest model support please see our DeepS...
Learning Rate Range Test
This tutorial shows how to use to perform Learning Rate range tests in PyTorch.
Megatron-LM GPT2
If you haven’t already, we advise you to first read through the Getting Started guide before stepping through this tutorial.
Mixed Precision ZeRO++
Mixed Precision ZeRO++ (MixZ++) is a set of optimization strategies based on ZeRO and ZeRO++ to improve the efficiency and reduce memory usage for large mode...
Getting Started with DeepSpeed-MoE for Inferencing Large-Scale MoE Models
DeepSpeed-MoE Inference introduces several important features on top of the inference optimization for dense models (DeepSpeed-Inference blog post). It embra...
Mixture of Experts for NLG models
In this tutorial, we introduce how to apply DeepSpeed Mixture of Experts (MoE) to NLG models, which reduces the training cost by 5 times and reduce the MoE m...
Mixture of Experts (DeepSpeed MoE)
DeepSpeed v0.5 introduces new support for training Mixture of Experts (MoE) models. MoE models are an emerging class of sparsely activated models that have s...
Monitor
Monitor your model’s training metrics live and log for future analysis
1-Cycle Schedule
This tutorial shows how to implement 1Cycle schedules for learning rate and momentum in PyTorch.
Pipeline Parallelism
DeepSpeed v0.3 includes new support for pipeline parallelism! Pipeline parallelism improves both the memory and compute efficiency of deep learning training ...
Using PyTorch Profiler with DeepSpeed for performance debugging
This tutorial describes how to use PyTorch Profiler with DeepSpeed.
DeepSpeed Transformer Kernel
This tutorial shows how to enable the DeepSpeed transformer kernel and set its different configuration parameters.
Arctic Long Sequence Training (ALST) for HF Transformers integration
Ulysses Sequence Parallelism for Hugging Face (HF) Transformers implements an efficient way of training on long sequences by employing sequence parallelis...
DeepSpeed Ulysses-Offload
DeepSpeed Ulysses-Offload is a system of chunking and offloading long-context transformer model training scheme built on top of ZeRO and DeepSpeed Ulysses. I...
Universal Checkpointing with DeepSpeed: A Practical Guide
DeepSpeed Universal Checkpointing feature is a powerful tool for saving and loading model checkpoints in a way that is both efficient and flexible, enabling ...
ZenFlow
ZenFlow is an extension of ZeRO-Offload that decouples and asynchronously updates gradients during training. It reduces CPU-induced stalls when using offload...
ZeRO-Offload
ZeRO-3 Offload consists of a subset of features in our newly released ZeRO-Infinity. Read our ZeRO-Infinity blog to learn more!
ZeRO++
ZeRO++ is a system of communication optimization strategies built on top of ZeRO to offer unmatched efficiency for large model training regardless of the sca...