Skip to content

Releases: huggingface/accelerate

Release list

v1.15.0: FSDP2 activation memory, dtensor improvements

v1.15.0: FSDP2 activation memory, dtensor improvements Latest
Latest

Choose a tag to compare

View all tags
@SunMarc SunMarc released this 09 Sep 13:30
v1.15.0
6afc1e5

v1.15.0: FSDP2 activation memory, dtensor improvements

FSDP2

A large batch of FSDP2 work this release: two fixes that cut activation memory at long sequence lengths, tied-embedding support on torch >= 2.13, and a round of checkpointing correctness and scale fixes.

Activation checkpointing was wrapping each child of the matched layer (self_attn, mlp, the norms) instead of the layer itself, so every inter-child activation stayed saved for backward. It now wraps the layer.

There's also a new FSDP2-only activation_checkpointing_offload, which moves the remaining per-layer checkpoint inputs to pinned CPU memory. Gradients are exactly those of plain activation checkpointing:

# fsdp2.yaml
fsdp_config:
  fsdp_version: 2
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_activation_checkpointing: true
  fsdp_activation_checkpointing_offload: true
accelerate launch --config_file fsdp2.yaml train.py
  • FSDP2 activation checkpointing: wrap the matched transformer layer itself, not each of its children by @qgallouedec in #4172
  • Add FSDP2 activation_checkpointing_offload: offload checkpointed layer inputs to pinned CPU memory by @qgallouedec in #4175
  • Fix FSDP2 tied-embedding models on torch >= 2.13: put the output embedding in the same fully_shard group by @qgallouedec in #4171
  • Fix FSDP2/PEFT/FULL_STATE_DICT dropping every rank's adapter shard except rank 0 by @AmineDiro in #4206
  • FSDP2: per-rank torch.save/load for SHARDED_STATE_DICT to fix 2800+ NPU checkpoint timeout by @gygdh-001 in #4105
  • Fix FSDP sharded checkpoint path resolution by @HaomingSong in #4119
  • Raise a clear error when FSDP is enabled on a mesh with no shard dimension by @qgallouedec in #4180

DTensor

Two fixes for DTensor-sharded models, which you hit with FSDP2, tensor parallelism, or any N-D parallelism setup: gradient clipping no longer fails on the foreach op when plain tensors and DTensors are mixed, and prepare_model leaves an already-sharded model where it is:

Offloading & Quantization

An entire model can now be dispatched to disk, including tied weights — useful for tools like llm-compressor that compress large models on machines that can't hold them:

Trackers

Custom trackers can be registered by name and then selected from log_with= like any built-in one:

from accelerate import Accelerator
from accelerate.tracking import register_tracker_class

register_tracker_class(MyTracker)  # MyTracker.name == "my_tracker"
accelerator = Accelerator(log_with="my_tracker")

Device support

Neuron gains a torch dynamo backend (so --torch-compile works with the Transformers Trainer) and MPS is now reported and handled properly by accelerate env and find_executable_batch_size.

CLI

  • Handle missing accelerate executable in env command by @tandede in #4168
  • Fix accelerate launch --cpu never setting the KMP variables by @yupengtang in #4222
  • Fix estimate-memory for timm>=1.0.29 by adding the hf-hub: prefix by @iamsharduld in #4213

Minor fixes

  • Refuse context parallelism for models with sliding-window or chunked attention layers by @qgallouedec in #4177
  • Fix convert_model_to_fp8_ao converting the first and last linear layers by @vineethsaivs in #4147
  • Fix get_non_persistent_buffers mutating module._non_persistent_buffers_set by @sohumt123 in #4116
  • fix: preserve dataloader iteration state when resuming from checkpoint by @TomQunChao in #4071
  • Fix load_accelerator_state only restoring one RNG backend by @Rakshit-gen in #4217
  • Reach through the DeepSpeed optimizer wrapper in AcceleratedOptimizer.eval() by @vineethsaivs in #4130
  • Fix compile_regions running the uncompiled module by @hjinnkim in #4188
  • Forward reduce_batch_size_fn through the find_executable_batch_size decorator form by @vineethsaivs in #4160
  • Fix distributedType -> distributed_type typo in downcast_bf16 guard by @uttam12331 in #4149
  • Fix duplicated words in user-facing messages by @Sreekant13 in #4091
  • Add missing type hints in checkpointing.py by @RudrenduPaul in #4153
  • Add missing type hints in utils/memory.py by @RudrenduPaul in #4152
  • docs: fix parameter names and a ghost entry in docstrings by @DaoyuanLi2816 in #4076
  • docs: fix documented parameter names that do not match the signature by @massimiliano1991 in #4156
  • docs: fix garbled sentence in CONTRIBUTING.md by @simpleqt in #4231
  • docs: fix three dead links left by the docs restructure by @simpleqt in #4230
  • Fix mixed precision argument typo in examples README by @noobyalan in #4192
  • Make doc builds faster by @mishig25 in #4103
  • Add security policy by @SunMarc in #4134
  • Bump pre-commit ruff hook to v0.13.1 to match the quality extra by @devangpratap in #4089
  • Fix flaky behavior in gated-model CLI test by making the test deterministic by @sywangyi in #4073
  • [test] Update min version torch by @SunMarc in #4170
  • Fix CI by @SunMarc in #4229
  • chore(deps): bump the actions group across 1 directory with 7 updates by @dependabot[bot] in #4167
  • chore(deps): bump the actions group across 1 directory with 4 updates by @dependabot[bot] in #4182
  • chore: update build_and_run_tests.yml by @hf-security-analysis[bot] in #4169

New Contributors

Full Changelog: v1.14.0...v1.15.0

Contributors

shoemoney, iamsharduld, and 30 other contributors
Assets 2
Loading

v1.14.0: AMD ROCm support, FSDP2 hardening

v1.14.0: AMD ROCm support, FSDP2 hardening

Choose a tag to compare

View all tags
@SunMarc SunMarc released this 11 Jun 13:58
v1.14.0
beb0672

FSDP2 Improvements

This release brings a large batch of FSDP2 fixes and quality-of-life improvements: correct dtype handling on load, sharding of embeddings/norms, QLoRA crash prevention, and a more robust auto-wrap policy.

  • Fsdp2 fully_shard embedding and norm by @SunMarc in #4015
  • Fix fsdp2 load full state dict dtype mismatch by @SunMarc in #4021
  • Fix region compilation fsdpv2 by @SunMarc in #4022
  • [FSDP2] Cast model to uniform dtype before fully_shard to fix mixed-dtype AssertionError by @roycho96 in #3985
  • [FSDP2] Auto-exclude non-floating frozen Params4bit from fully_shard to prevent QLoRA crash by @roycho96 in #3987
  • fix(FSDP2): auto-wrap policy ignoring _no_split_modules fallback by @JohnGiorgi in #3999
  • fix: use key-based matching in fsdp2_load_full_state_dict by @roycho96 in #3982
  • fix: add missing model_has_params4bit guard to fsdp2_load_full_state_dict call by @roycho96 in #3981
  • Fix to-fsdp2: drop REMOVED / NOT_YET_IMPLEMENTED FSDP1 keys instead of leaking them by @lollinng in #4065
  • Prevent double-wrapping models in prepare_model() by @joshuaswanson in #3977

AMD ROCm support

Accelerate now works end-to-end on AMD ROCm devices. Thanks @Abdennacer-Badaoui!

Neuron

Further Neuron improvements to reduce recompilation and cover missing device cases.

Quantization & Offloading

We improved offloading support for quantized models, including Torchao, int8, and tied-weight handling.

Data Loading

  • Feat: Support dynamic batch size in BatchSamplerShard with even_batches by @yuxinyuan in #3969
  • Fix iterable dataset sharding condition when n_shards == num_processes by @SunMarc in #3958
  • Fix implicit padding in split_between_processes when apply_padding=False and num_samples < num_processes by @3manifold in #4052

Minor fixes

  • [DeepSpeed] allow kernels flash-attn in SP by @kashif in #3959
  • Fix: Conditionally import torch.distributed.algorithms.join in accelerator.py by @0xDELUXA in #3962
  • Fix is_hf_initialized attribute by @SunMarc in #3976
  • feat(utils): add max reduction type by @imstevenpmwork in #4027
  • fix(state): make MLU backend part of the _prepare_backend elif chain by @Anai-Guo in #4057
  • fix notebook launcher cuda init by @SunMarc in #4059
  • pytorch-triton-xpu rename to triton-xpu by @sywangyi in #4007
  • Relax numerical tolerance for XPU in test_big_modeling by @YangKai0616 in #4001
  • Fix gloo backend error in test_load_checkpoint_and_dispatch_with_broadcast on XPU by @kaixuanliu in #4056
  • Raise ValueError instead of a bare string in ParallelismConfig.get_device_mesh by @lollinng in #4064
  • tests: Gracefully handle missing set_device for mps by @booxter in #4028
  • test: add regression test for no_split_module_classes accepting set type by @UFO0506 in #4048
  • Fix all tests by @SunMarc in #4072
  • docs: add aggregate profiler memory example by @aryanputta in #4054
  • DOC: document missing parameters in load_accelerator_state, find_executable_batch_size, and send_to_device by @kratos0718 in #4051
  • docs: Fix docstring of fsdp2_prepare_auto_wrap_policy by @slocoro in #4037
  • Fix DistributedType documentation by @3manifold in #3980
  • Fix grammar, spelling, and consistency issues across docs and examples by @cihandemir in #3961
  • docs: fix typos in docstrings, comments, and user docs by @mokashang in #4040
  • chore: update doc-builder workflow SHA by @rtrompier in #4009
  • chore: bump doc-builder SHA for main doc build workflow by @rtrompier in #4018
  • [CI] Bump style-bot SHA + switch to GitHub App by @paulinebm in #4031
  • Fix TrackioTracker.log() ignoring step parameter by @joshuaswanson in #3975
  • fix: pass step parameter in TrackioTracker.log() by @liuyun7345 in #3970
  • fix(tracking): default step=None on tracker.log and accept extra kwargs in MLflowTracker by @1fanwang in #4039
  • Fix MLflowTracker.store_init_configuration mutating the caller's config dict by @ATOM00blue in #4046
  • fix(tracker): guard init_trackers and log against None kwargs by @xodn348 in #4026
  • 🔒 Pin GitHub Actions to commit SHAs by @paulinebm in #3992
  • chore: update build-docker-images-release.yml by @hf-security-analysis[bot] in #4069
  • chore: enable Dependabot weekly GitHub Actions bumps by @hf-dependantbot-rollout[bot] in #4049
  • Bump the actions group with 8 updates by @dependabot[bot] in #4068

Full Changelog: v1.13.0...v1.14.0

Contributors

kashif, booxter, and 30 other contributors
Loading

v1.13.0: Neuron support, IPEX removal, and distributed training fixes

v1.13.0: Neuron support, IPEX removal, and distributed training fixes

Choose a tag to compare

View all tags
@SunMarc SunMarc released this 04 Mar 20:15
v1.13.0
e6ee133

AWS Neuron support

We now have support for AWS Neuron (Trainium/Inferentia) devices. Thanks @michaelbenayoun for adding this.

XPU Improvements

We've removed IPEX dependency and improved device-agnostic code for XPU.

FSDP2 Improvements

We've added a bunch of important fixes for FSDP2 users: upcasting only grad-requiring params, better tied embedding errors, DCP optimizer loading, bf16 optimizer step crash fix, and torch < 2.7.0 compatibility.

  • Upcast FSDP2 parameters only if requires_grad by @ojh31 in #3848
  • Fix FSDP2 tied embedding errors with targeted ValueError guidance by @amanzoni1 in #3878
  • bug: fsdp cannot load optimizer state using dcp by @flymin in #3904
  • fix crash in optimizer.step when fsdp2 is enabled and model is bfloat16 by @sywangyi in #3905
  • Fix FSDP2 crash with ignored_params on torch < 2.7.0 by @Mr-Neutr0n in #3924

DeepSpeed Sequence Parallelism

We've added several fixes to the DeepSpeed + Sequence Parallelism integration introduced in v1.12.0, including evaluation support during SP training and proper process group handling.

  • [SP] fix loss computation example by @kashif in #3858
  • [SP and CP] error out if both CP and SP enabled by @kashif in #3862
  • DeepSpeed has its own process group by @kashif in #3916
  • [Deepspeed] skip device mesh creation when deepspeed and sp_size >1 by @kashif in #3914
  • Enable evaluation during deepspeed Sequence Parallel by @jp1924 in #3917

FP8

We've enhanced FP8 training. Thanks @shimizust for fixing torchao support.

Performance

Accelerate now imports faster by deferring heavy dependencies, and torch.compile hooks are disabled lazily.

Minor fixes

Contributors

kashif, hexfaker, and 26 other contributors
Loading

v1.12.0: Deepspeed Ulysses/ALST

v1.12.0: Deepspeed Ulysses/ALST

Choose a tag to compare

View all tags
@SunMarc SunMarc released this 21 Nov 12:47
v1.12.0
61ff833

Deepspeed Ulysses/ALST integration

Deepspeed Ulysses/ALST is an efficient way of training on long sequences by employing sequence parallelism and attention head parallelism. You can learn more about this technology in this paper https://arxiv.org/abs/2506.13996 or this deepspeed tutorial https://www.deepspeed.ai/tutorials/ulysses-alst-sequence-parallelism/.

0d8bd9e0

To enable Deepspeed Ulysses, you first need to create ParallelismConfig and setting sp related args:

parallelism_config = ParallelismConfig(
    sp_backend="deepspeed",
    sp_size=2,
    sp_handler=DeepSpeedSequenceParallelConfig(...),
)

Then, you need to make sure to compute the correct loss as described on our docs

        ...
        losses_per_rank = torch.distributed.nn.functional.all_gather(loss, group=sp_group)
        good_tokens = (shift_labels != -100).view(-1).sum()
        good_tokens_per_rank = torch.distributed.nn.functional.all_gather(good_tokens, group=sp_group)
        total_loss = sum(
            losses_per_rank[rank] * good_tokens_per_rank[rank]
            for rank in range(sp_world_size)
            if good_tokens_per_rank[rank] > 0
        )
        total_good_tokens = sum(good_tokens_per_rank)
        loss = total_loss / max(total_good_tokens, 1)

Thanks @S1ro1 for starting this work and for @stas00 for finishing this work. Also thanks @kashif for adding docs and reviewing/testing this PR !

This feature will also be available in HF Trainer thanks for this PR from @stas00: huggingface/transformers#41832

Minor changes

New Contributors

Full Changelog: v1.11.0...v1.12.0

Contributors

kashif, stas00, and 6 other contributors
Loading

v1.11.0: TE MXFP8, FP16/BF16 with MPS, Python 3.10

v1.11.0: TE MXFP8, FP16/BF16 with MPS, Python 3.10

Choose a tag to compare

View all tags
@SunMarc SunMarc released this 20 Oct 16:08
v1.11.0
9a81156

TE MXFP8 support

We've added support for MXFP8 in our TransformerEngine integration. To use that, you need to set use_mxfp8_block_scaling in fp8_config. See nvidia docs [here]. (https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html#MXFP8-and-block-scaling)

  • Add support for TE MXFP8 recipe in accelerate by @pstjohn in #3688

FP16/BF16 Training for MPS devices

BF16 and FP16 support for MPS devices is finally here. You can now pass mixed_precision = "fp16" or "bf16" when training on a mac (fp16 requires torch 2.8 and bf16 requires torch 2.6)

  • Add bf16/fp16 support for amp with mps device by @SunMarc in #3373

FSDP updates

The following PRs add respectively support to ignored_params and no_sync() for FSDPv2:

  • feat: add ignored_params support for fsdp2 by @kmehant in #3731
  • fix: model.set_requires_gradient_sync(False) should be called to turn off gradient synchronization in FSDP2 by @EquationWalker in #3762

Mixed precision can now be passed as a dtype string from accelerate cli flag or fsdp_config in accelerate config file:

  • feat: allow mixed precision policy as dtype by @kmehant in #3751

Nd-parallel updates

Some minor updates concerning nd-parallelism.

Bump to Python 3.10

We've dropped support for python 3.9 as it reached EOL in October.

Lots of minor fixes:

New Contributors

Full Changelog: v1.10.1...v1.11.0

Contributors

Qubitium, pstjohn, and 12 other contributors
Loading

v1.10.1: Patchfix

v1.10.1: Patchfix

Choose a tag to compare

View all tags
@SunMarc SunMarc released this 25 Aug 13:57
v1.10.1
623cc0b

Full Changelog: v1.10.0...v1.10.1

Contributors

S1ro1 and SunMarc
Loading

v1.10.0: N-D Parallelism

v1.10.0: N-D Parallelism

Choose a tag to compare

View all tags
@SunMarc SunMarc released this 07 Aug 13:39
v1.10.0
5dcd16e

N-D Parallelism

Training large models across multiple GPUs can be complex, especially when combining different parallelism strategies (e.g TP, CP, DP). To simplify this process, we've collaborated with Axolotl to introduce an easy-to-use integration that allows you to apply any combination of parallelism strategies directly in your training script. Just pass a ParallelismConfig specifying the size of each parallelism type—it's that simple.
Learn more about how it works in our latest blogpost.

parallelism_config = ParallelismConfig(
    dp_shard_size=2,
    dp_replicate_size=2,
    cp_size=2,
    tp_size=2,
)
accelerator = Accelerator(
    parallelism_config=parallelism_config,
   ...
)
model = AutoModelForCausalLM.from_pretrained("your-model-name", device_mesh=accelerator.torch_device_mesh)
model = accelerator.prepare(model)
  • Parallelism config + TP + HSDP + BYODM (Bring Your Own Device Mesh) by @salmanmohammadi in #3682
  • Feat: context parallel v2.0 by @S1ro1 in #3700
  • set default submesh_tp_size to prevent unset local variable error by @winglian in #3687
  • Add Parallelism getter property to Accelerator class by @WoosungMyung in #3703
  • Fix: prepare works even if nothing except tp specified (rare) by @S1ro1 in #3707
  • Set parallelism_config in constructor due to Trainer reset of State by @winglian in #3713
  • Fix: tp size wouldn't read from env by @S1ro1 in #3716
  • Remove ParallelismConfig from PartialState by @SunMarc in #3720

FSDP improvements

We've fixed ignored modules attribute. With this, it is now possible to train PEFT model that moe layers that contrains q_proj and v_proj parameters. This is especially important for fine-tuning gpt-oss model.

Minor improvements

  • feature: CpuOffload pre_forward don't attempt to move if already on device by @JoeGaffney in #3695
  • Fix: Ensure environment variable values are case-insensitive in Accelerate by @jp1924 in #3712
  • remove use_ipex by @SunMarc in #3721

New Contributors

Full Changelog: v1.9.0...v1.10.0

Contributors

winglian, BenjaminBossan, and 6 other contributors
Loading

v1.9.0: Trackio support, Model loading speedup, Minor distributed improvements

v1.9.0: Trackio support, Model loading speedup, Minor distributed improvements

Choose a tag to compare

View all tags
@SunMarc SunMarc released this 16 Jul 16:35
v1.9.0
5565412

Trackio tracker support

We've added support for a trackio, lightweight, 💯 free experiment tracking Python library built on top of 🤗 Datasets and Spaces.

Screen Recording 2025-06-11 at 5 39 32 PM

Main features are:

  • Local-first design: dashboard runs locally by default. You can also host it on Spaces by specifying a space_id.
  • Persists logs locally (or in a private Hugging Face Dataset)
  • Visualize experiments with a Gradio dashboard locally (or on Hugging Face Spaces)
  • Everything here, including hosting on Hugging Faces, is free!

To use it with accelerate, you need to set log_with and initialize the trackers

accelerator = Accelerator(log_with="trackio")
config={"learning_rate": 0.001, "batch_size": 32}
# init_kwargs in order to host the dashboard on spaces
init_kwargs = {"trackio": {"space_id": "hf_username/space_name"}
accelerator.init_trackers("example_project", config=config, init_kwargs=init_kwargs})

Thanks @pcuenca for the integration !

Model loading speedup when relying set_module_tensor_to_device

Setting tensor while clearing cache is very slow, so we added clear_device option to disable it.
Another small optimization is using non_blocking everywhere and syncing just before returning control to the user. This makes the loading slightly faster.

FDSP, Deepspeed, FP8 minor improvements

  • Add support for e5e2 and default to hybrid when launcher is used by @IlyasMoutawwakil in #3640
  • Fix FP8 tests, enable FP8 to be used without direct Accelerator() configuring by @pstjohn in #3677
  • Bunch of FSDP improvements by @S1ro1 in #3671
  • Fix: properly error when DDP + Dtensor model by @S1ro1 in #3629
  • Fix fsdp2 example typo by @shimizust in #3657
  • Added a check in no_sync() to avoid errors when using deepspeed zero2/3 by @xliu0105 in #3656

🚨🚨🚨 Breaking changes 🚨🚨🚨

find_executable_batch_size() will no longer halves the batch after every OOM. Instead, we will multiply the batch size by 0.9. This should help user not waste gpu capacity.

  • “Stop Halving My Batch!” · Default back-off 0.5 → 0.9 by @SunMarc in #3684

What's Changed

New Contributors

Full Changelog: v1.8.1...v1.9.0

Contributors

pcuenca, pstjohn, and 10 other contributors
Loading

v1.8.1: Patchfix

v1.8.1: Patchfix

Choose a tag to compare

View all tags
@SunMarc SunMarc released this 20 Jun 15:43
v1.8.1
f5513f1

Full Changelog: v1.8.0...v1.8.1

Contributors

SunMarc and IlyasMoutawwakil
Loading

v1.8.0: FSDPv2 + FP8, Regional Compilation for DeepSpeed, Faster Distributed Training on Intel CPUs, ipex.optimize deprecation

v1.8.0: FSDPv2 + FP8, Regional Compilation for DeepSpeed, Faster Distributed Training on Intel CPUs, ipex.optimize deprecation

Choose a tag to compare

View all tags
@SunMarc SunMarc released this 19 Jun 15:37
v1.8.0
d2ced37

FSDPv2 refactor + FP8 support

We've simplified how to prepare FSDPv2 models, as there were too many ways to compose FSDP2 with other features (e.g., FP8, torch.compile, activation checkpointing, etc.). Although the setup is now more restrictive, it leads to fewer errors and a more performant user experience. We’ve also added support for FP8. You can read about the results here. Thanks to @S1ro1 for this contribution!

Faster Distributed Training on Intel CPUs

We updated the CCL_WORKER_COUNT variable and added KMP parameters for Intel CPU users. This significantly improves distributed training performance (e.g., Tensor Parallelism), with up to a 40% speed-up on Intel 4th Gen Xeon when training transformer TP models.

Regional Compilation for DeepSpeed

We added support for regional compilation with the DeepSpeed engine. DeepSpeed’s .compile() modifies models in-place using torch.nn.Module.compile(...), rather than the out-of-place torch.compile(...), so we had to account for that. Thanks @IlyasMoutawwakil for this feature!

ipex.optimize deprecation

ipex.optimize is being deprecated. Most optimizations have been upstreamed to PyTorch, and future improvements will land there directly. For users without PyTorch 2.8, we’ll continue to rely on IPEX for now.

Better XPU Support

We've greatly expanded and stabilized support for Intel XPUs:

Trackers

We've added support for SwanLab as an experiment tracking backend. Huge thanks to @ShaohonChen for this contribution ! We also deferred all tracker initializations to prevent premature setup of distributed environments.

  • Integrate SwanLab for offline/online experiment tracking for Accelerate by @ShaohonChen in #3605
  • Fix: Defer Tracker Initialization to Prevent Premature Distributed Setup by @yuanjua in #3581

What's Changed

New Contributors

Full Changelog: v1.7.0...v1.8.0

Contributors

kashif, yao-matrix, and 12 other contributors
Loading