Skip to content

Releases: huggingface/accelerate

v1.14.0: AMD ROCm support, FSDP2 hardening

11 Jun 13:58
@SunMarc SunMarc
v1.14.0
beb0672

Choose a tag to compare

View all tags
v1.14.0: AMD ROCm support, FSDP2 hardening Latest
Latest

FSDP2 Improvements

This release brings a large batch of FSDP2 fixes and quality-of-life improvements: correct dtype handling on load, sharding of embeddings/norms, QLoRA crash prevention, and a more robust auto-wrap policy.

  • Fsdp2 fully_shard embedding and norm by @SunMarc in #4015
  • Fix fsdp2 load full state dict dtype mismatch by @SunMarc in #4021
  • Fix region compilation fsdpv2 by @SunMarc in #4022
  • [FSDP2] Cast model to uniform dtype before fully_shard to fix mixed-dtype AssertionError by @roycho96 in #3985
  • [FSDP2] Auto-exclude non-floating frozen Params4bit from fully_shard to prevent QLoRA crash by @roycho96 in #3987
  • fix(FSDP2): auto-wrap policy ignoring _no_split_modules fallback by @JohnGiorgi in #3999
  • fix: use key-based matching in fsdp2_load_full_state_dict by @roycho96 in #3982
  • fix: add missing model_has_params4bit guard to fsdp2_load_full_state_dict call by @roycho96 in #3981
  • Fix to-fsdp2: drop REMOVED / NOT_YET_IMPLEMENTED FSDP1 keys instead of leaking them by @lollinng in #4065
  • Prevent double-wrapping models in prepare_model() by @joshuaswanson in #3977

AMD ROCm support

Accelerate now works end-to-end on AMD ROCm devices. Thanks @Abdennacer-Badaoui!

Neuron

Further Neuron improvements to reduce recompilation and cover missing device cases.

Quantization & Offloading

We improved offloading support for quantized models, including Torchao, int8, and tied-weight handling.

Data Loading

  • Feat: Support dynamic batch size in BatchSamplerShard with even_batches by @yuxinyuan in #3969
  • Fix iterable dataset sharding condition when n_shards == num_processes by @SunMarc in #3958
  • Fix implicit padding in split_between_processes when apply_padding=False and num_samples < num_processes by @3manifold in #4052

Minor fixes

  • [DeepSpeed] allow kernels flash-attn in SP by @kashif in #3959
  • Fix: Conditionally import torch.distributed.algorithms.join in accelerator.py by @0xDELUXA in #3962
  • Fix is_hf_initialized attribute by @SunMarc in #3976
  • feat(utils): add max reduction type by @imstevenpmwork in #4027
  • fix(state): make MLU backend part of the _prepare_backend elif chain by @Anai-Guo in #4057
  • fix notebook launcher cuda init by @SunMarc in #4059
  • pytorch-triton-xpu rename to triton-xpu by @sywangyi in #4007
  • Relax numerical tolerance for XPU in test_big_modeling by @YangKai0616 in #4001
  • Fix gloo backend error in test_load_checkpoint_and_dispatch_with_broadcast on XPU by @kaixuanliu in #4056
  • Raise ValueError instead of a bare string in ParallelismConfig.get_device_mesh by @lollinng in #4064
  • tests: Gracefully handle missing set_device for mps by @booxter in #4028
  • test: add regression test for no_split_module_classes accepting set type by @UFO0506 in #4048
  • Fix all tests by @SunMarc in #4072
  • docs: add aggregate profiler memory example by @aryanputta in #4054
  • DOC: document missing parameters in load_accelerator_state, find_executable_batch_size, and send_to_device by @kratos0718 in #4051
  • docs: Fix docstring of fsdp2_prepare_auto_wrap_policy by @slocoro in #4037
  • Fix DistributedType documentation by @3manifold in #3980
  • Fix grammar, spelling, and consistency issues across docs and examples by @cihandemir in #3961
  • docs: fix typos in docstrings, comments, and user docs by @mokashang in #4040
  • chore: update doc-builder workflow SHA by @rtrompier in #4009
  • chore: bump doc-builder SHA for main doc build workflow by @rtrompier in #4018
  • [CI] Bump style-bot SHA + switch to GitHub App by @paulinebm in #4031
  • Fix TrackioTracker.log() ignoring step parameter by @joshuaswanson in #3975
  • fix: pass step parameter in TrackioTracker.log() by @liuyun7345 in #3970
  • fix(tracking): default step=None on tracker.log and accept extra kwargs in MLflowTracker by @1fanwang in #4039
  • Fix MLflowTracker.store_init_configuration mutating the caller's config dict by @ATOM00blue in #4046
  • fix(tracker): guard init_trackers and log against None kwargs by @xodn348 in #4026
  • 🔒 Pin GitHub Actions to commit SHAs by @paulinebm in #3992
  • chore: update build-docker-images-release.yml by @hf-security-analysis[bot] in #4069
  • chore: enable Dependabot weekly GitHub Actions bumps by @hf-dependantbot-rollout[bot] in #4049
  • Bump the actions group with 8 updates by @dependabot[bot] in #4068

Full Changelog: v1.13.0...v1.14.0

Contributors

kashif, booxter, and 30 other contributors
Assets 2
Loading

v1.13.0: Neuron support, IPEX removal, and distributed training fixes

04 Mar 20:15
@SunMarc SunMarc
v1.13.0
e6ee133

Choose a tag to compare

View all tags
v1.13.0: Neuron support, IPEX removal, and distributed training fixes

AWS Neuron support

We now have support for AWS Neuron (Trainium/Inferentia) devices. Thanks @michaelbenayoun for adding this.

XPU Improvements

We've removed IPEX dependency and improved device-agnostic code for XPU.

FSDP2 Improvements

We've added a bunch of important fixes for FSDP2 users: upcasting only grad-requiring params, better tied embedding errors, DCP optimizer loading, bf16 optimizer step crash fix, and torch < 2.7.0 compatibility.

  • Upcast FSDP2 parameters only if requires_grad by @ojh31 in #3848
  • Fix FSDP2 tied embedding errors with targeted ValueError guidance by @amanzoni1 in #3878
  • bug: fsdp cannot load optimizer state using dcp by @flymin in #3904
  • fix crash in optimizer.step when fsdp2 is enabled and model is bfloat16 by @sywangyi in #3905
  • Fix FSDP2 crash with ignored_params on torch < 2.7.0 by @Mr-Neutr0n in #3924

DeepSpeed Sequence Parallelism

We've added several fixes to the DeepSpeed + Sequence Parallelism integration introduced in v1.12.0, including evaluation support during SP training and proper process group handling.

  • [SP] fix loss computation example by @kashif in #3858
  • [SP and CP] error out if both CP and SP enabled by @kashif in #3862
  • DeepSpeed has its own process group by @kashif in #3916
  • [Deepspeed] skip device mesh creation when deepspeed and sp_size >1 by @kashif in #3914
  • Enable evaluation during deepspeed Sequence Parallel by @jp1924 in #3917

FP8

We've enhanced FP8 training. Thanks @shimizust for fixing torchao support.

Performance

Accelerate now imports faster by deferring heavy dependencies, and torch.compile hooks are disabled lazily.

Minor fixes

Contributors

kashif, hexfaker, and 26 other contributors
Loading

v1.12.0: Deepspeed Ulysses/ALST

21 Nov 12:47
@SunMarc SunMarc
v1.12.0
61ff833

Choose a tag to compare

View all tags
v1.12.0: Deepspeed Ulysses/ALST

Deepspeed Ulysses/ALST integration

Deepspeed Ulysses/ALST is an efficient way of training on long sequences by employing sequence parallelism and attention head parallelism. You can learn more about this technology in this paper https://arxiv.org/abs/2506.13996 or this deepspeed tutorial https://www.deepspeed.ai/tutorials/ulysses-alst-sequence-parallelism/.

0d8bd9e0

To enable Deepspeed Ulysses, you first need to create ParallelismConfig and setting sp related args:

parallelism_config = ParallelismConfig(
    sp_backend="deepspeed",
    sp_size=2,
    sp_handler=DeepSpeedSequenceParallelConfig(...),
)

Then, you need to make sure to compute the correct loss as described on our docs

        ...
        losses_per_rank = torch.distributed.nn.functional.all_gather(loss, group=sp_group)
        good_tokens = (shift_labels != -100).view(-1).sum()
        good_tokens_per_rank = torch.distributed.nn.functional.all_gather(good_tokens, group=sp_group)
        total_loss = sum(
            losses_per_rank[rank] * good_tokens_per_rank[rank]
            for rank in range(sp_world_size)
            if good_tokens_per_rank[rank] > 0
        )
        total_good_tokens = sum(good_tokens_per_rank)
        loss = total_loss / max(total_good_tokens, 1)

Thanks @S1ro1 for starting this work and for @stas00 for finishing this work. Also thanks @kashif for adding docs and reviewing/testing this PR !

This feature will also be available in HF Trainer thanks for this PR from @stas00: huggingface/transformers#41832

Minor changes

New Contributors

Full Changelog: v1.11.0...v1.12.0

Contributors

kashif, stas00, and 6 other contributors
Loading

v1.11.0: TE MXFP8, FP16/BF16 with MPS, Python 3.10

20 Oct 16:08
@SunMarc SunMarc
v1.11.0
9a81156

Choose a tag to compare

View all tags
v1.11.0: TE MXFP8, FP16/BF16 with MPS, Python 3.10

TE MXFP8 support

We've added support for MXFP8 in our TransformerEngine integration. To use that, you need to set use_mxfp8_block_scaling in fp8_config. See nvidia docs [here]. (https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html#MXFP8-and-block-scaling)

  • Add support for TE MXFP8 recipe in accelerate by @pstjohn in #3688

FP16/BF16 Training for MPS devices

BF16 and FP16 support for MPS devices is finally here. You can now pass mixed_precision = "fp16" or "bf16" when training on a mac (fp16 requires torch 2.8 and bf16 requires torch 2.6)

  • Add bf16/fp16 support for amp with mps device by @SunMarc in #3373

FSDP updates

The following PRs add respectively support to ignored_params and no_sync() for FSDPv2:

  • feat: add ignored_params support for fsdp2 by @kmehant in #3731
  • fix: model.set_requires_gradient_sync(False) should be called to turn off gradient synchronization in FSDP2 by @EquationWalker in #3762

Mixed precision can now be passed as a dtype string from accelerate cli flag or fsdp_config in accelerate config file:

  • feat: allow mixed precision policy as dtype by @kmehant in #3751

Nd-parallel updates

Some minor updates concerning nd-parallelism.

Bump to Python 3.10

We've dropped support for python 3.9 as it reached EOL in October.

Lots of minor fixes:

New Contributors

Full Changelog: v1.10.1...v1.11.0

Contributors

Qubitium, pstjohn, and 12 other contributors
Loading

v1.10.1: Patchfix

25 Aug 13:57
@SunMarc SunMarc
v1.10.1
623cc0b

Choose a tag to compare

View all tags
v1.10.1: Patchfix

Full Changelog: v1.10.0...v1.10.1

Contributors

S1ro1 and SunMarc
Loading

v1.10.0: N-D Parallelism

07 Aug 13:39
@SunMarc SunMarc
v1.10.0
5dcd16e

Choose a tag to compare

View all tags
v1.10.0: N-D Parallelism

N-D Parallelism

Training large models across multiple GPUs can be complex, especially when combining different parallelism strategies (e.g TP, CP, DP). To simplify this process, we've collaborated with Axolotl to introduce an easy-to-use integration that allows you to apply any combination of parallelism strategies directly in your training script. Just pass a ParallelismConfig specifying the size of each parallelism type—it's that simple.
Learn more about how it works in our latest blogpost.

parallelism_config = ParallelismConfig(
    dp_shard_size=2,
    dp_replicate_size=2,
    cp_size=2,
    tp_size=2,
)
accelerator = Accelerator(
    parallelism_config=parallelism_config,
   ...
)
model = AutoModelForCausalLM.from_pretrained("your-model-name", device_mesh=accelerator.torch_device_mesh)
model = accelerator.prepare(model)
  • Parallelism config + TP + HSDP + BYODM (Bring Your Own Device Mesh) by @salmanmohammadi in #3682
  • Feat: context parallel v2.0 by @S1ro1 in #3700
  • set default submesh_tp_size to prevent unset local variable error by @winglian in #3687
  • Add Parallelism getter property to Accelerator class by @WoosungMyung in #3703
  • Fix: prepare works even if nothing except tp specified (rare) by @S1ro1 in #3707
  • Set parallelism_config in constructor due to Trainer reset of State by @winglian in #3713
  • Fix: tp size wouldn't read from env by @S1ro1 in #3716
  • Remove ParallelismConfig from PartialState by @SunMarc in #3720

FSDP improvements

We've fixed ignored modules attribute. With this, it is now possible to train PEFT model that moe layers that contrains q_proj and v_proj parameters. This is especially important for fine-tuning gpt-oss model.

Minor improvements

  • feature: CpuOffload pre_forward don't attempt to move if already on device by @JoeGaffney in #3695
  • Fix: Ensure environment variable values are case-insensitive in Accelerate by @jp1924 in #3712
  • remove use_ipex by @SunMarc in #3721

New Contributors

Full Changelog: v1.9.0...v1.10.0

Contributors

winglian, BenjaminBossan, and 6 other contributors
Loading

v1.9.0: Trackio support, Model loading speedup, Minor distributed improvements

16 Jul 16:35
@SunMarc SunMarc
v1.9.0
5565412

Choose a tag to compare

View all tags
v1.9.0: Trackio support, Model loading speedup, Minor distributed improvements

Trackio tracker support

We've added support for a trackio, lightweight, 💯 free experiment tracking Python library built on top of 🤗 Datasets and Spaces.

Screen Recording 2025-06-11 at 5 39 32 PM

Main features are:

  • Local-first design: dashboard runs locally by default. You can also host it on Spaces by specifying a space_id.
  • Persists logs locally (or in a private Hugging Face Dataset)
  • Visualize experiments with a Gradio dashboard locally (or on Hugging Face Spaces)
  • Everything here, including hosting on Hugging Faces, is free!

To use it with accelerate, you need to set log_with and initialize the trackers

accelerator = Accelerator(log_with="trackio")
config={"learning_rate": 0.001, "batch_size": 32}
# init_kwargs in order to host the dashboard on spaces
init_kwargs = {"trackio": {"space_id": "hf_username/space_name"}
accelerator.init_trackers("example_project", config=config, init_kwargs=init_kwargs})

Thanks @pcuenca for the integration !

Model loading speedup when relying set_module_tensor_to_device

Setting tensor while clearing cache is very slow, so we added clear_device option to disable it.
Another small optimization is using non_blocking everywhere and syncing just before returning control to the user. This makes the loading slightly faster.

FDSP, Deepspeed, FP8 minor improvements

  • Add support for e5e2 and default to hybrid when launcher is used by @IlyasMoutawwakil in #3640
  • Fix FP8 tests, enable FP8 to be used without direct Accelerator() configuring by @pstjohn in #3677
  • Bunch of FSDP improvements by @S1ro1 in #3671
  • Fix: properly error when DDP + Dtensor model by @S1ro1 in #3629
  • Fix fsdp2 example typo by @shimizust in #3657
  • Added a check in no_sync() to avoid errors when using deepspeed zero2/3 by @xliu0105 in #3656

🚨🚨🚨 Breaking changes 🚨🚨🚨

find_executable_batch_size() will no longer halves the batch after every OOM. Instead, we will multiply the batch size by 0.9. This should help user not waste gpu capacity.

  • “Stop Halving My Batch!” · Default back-off 0.5 → 0.9 by @SunMarc in #3684

What's Changed

New Contributors

Full Changelog: v1.8.1...v1.9.0

Contributors

pcuenca, pstjohn, and 10 other contributors
Loading

v1.8.1: Patchfix

20 Jun 15:43
@SunMarc SunMarc
v1.8.1
f5513f1

Choose a tag to compare

View all tags
v1.8.1: Patchfix

Full Changelog: v1.8.0...v1.8.1

Contributors

SunMarc and IlyasMoutawwakil
Loading

v1.8.0: FSDPv2 + FP8, Regional Compilation for DeepSpeed, Faster Distributed Training on Intel CPUs, ipex.optimize deprecation

19 Jun 15:37
@SunMarc SunMarc
v1.8.0
d2ced37

Choose a tag to compare

View all tags
v1.8.0: FSDPv2 + FP8, Regional Compilation for DeepSpeed, Faster Distributed Training on Intel CPUs, ipex.optimize deprecation

FSDPv2 refactor + FP8 support

We've simplified how to prepare FSDPv2 models, as there were too many ways to compose FSDP2 with other features (e.g., FP8, torch.compile, activation checkpointing, etc.). Although the setup is now more restrictive, it leads to fewer errors and a more performant user experience. We’ve also added support for FP8. You can read about the results here. Thanks to @S1ro1 for this contribution!

Faster Distributed Training on Intel CPUs

We updated the CCL_WORKER_COUNT variable and added KMP parameters for Intel CPU users. This significantly improves distributed training performance (e.g., Tensor Parallelism), with up to a 40% speed-up on Intel 4th Gen Xeon when training transformer TP models.

Regional Compilation for DeepSpeed

We added support for regional compilation with the DeepSpeed engine. DeepSpeed’s .compile() modifies models in-place using torch.nn.Module.compile(...), rather than the out-of-place torch.compile(...), so we had to account for that. Thanks @IlyasMoutawwakil for this feature!

ipex.optimize deprecation

ipex.optimize is being deprecated. Most optimizations have been upstreamed to PyTorch, and future improvements will land there directly. For users without PyTorch 2.8, we’ll continue to rely on IPEX for now.

Better XPU Support

We've greatly expanded and stabilized support for Intel XPUs:

Trackers

We've added support for SwanLab as an experiment tracking backend. Huge thanks to @ShaohonChen for this contribution ! We also deferred all tracker initializations to prevent premature setup of distributed environments.

  • Integrate SwanLab for offline/online experiment tracking for Accelerate by @ShaohonChen in #3605
  • Fix: Defer Tracker Initialization to Prevent Premature Distributed Setup by @yuanjua in #3581

What's Changed

New Contributors

Full Changelog: v1.7.0...v1.8.0

Contributors

kashif, yao-matrix, and 12 other contributors
Loading

v1.7.0 : Regional compilation, Layerwise casting hook, FSDPv2 + QLoRA

15 May 12:33
@SunMarc SunMarc
v1.7.0
9cb1a6b

Choose a tag to compare

View all tags
v1.7.0 : Regional compilation, Layerwise casting hook, FSDPv2 + QLoRA

Regional compilation

Instead of compiling the entire model at once, regional compilation targets repeated blocks (such as decoder layers) first. This allows the compiler to cache and reuse optimized code for subsequent blocks, significantly reducing the cold start compilation time typically seen during the first inference. Thanks @IlyasMoutawwakil for the feature ! You can view the full benchmark here, and check out our updated compilation guide for more details!

compilation_time-1

To enable this feature, set use_regional_compilation=True in the TorchDynamoPlugin configuration.

# Configure the compilation backend
dynamo_plugin = TorchDynamoPlugin(
    use_regional_compilation=True,
    ... # other parameters
)
# Initialize accelerator with the plugin
accelerator = Accelerator(dynamo_plugin=dynamo_plugin)
# This will apply compile_regions to your model
model = accelerator.prepare(model)

Layerwise casting hook

We've introduced a new hook that enables per-layer upcasting and downcasting (e.g., for Linear layers) during inference. This allows users to run models with separate storage and compute dtypes, resulting in memory savings. The concept was first implemented in diffusers, where downcasting models to FP8 proved effective without major quality degradation. Contributed by @sayakpaul in #3427

model = ....
storage_dtype = torch.float8_e4m3fn
compute_dtype = torch.bfloat16
attach_layerwise_casting_hooks(
            model,
            storage_dtype=storage_dtype,
            compute_dtype=compute_dtype,
        )

Better FSDP2 support

This release includes numerous new features and bug fixes. Notably, we’ve added support for FULL_STATE_DICT, a widely used option in FSDP, now enabling .save_pretrained() in transformers to work with FSDP2 wrapped models. QLoRA training is now supported as well but more testing is needed. We have also resolved a backend issue related to parameter offloading to CPU. Additionally, a significant memory spike that occurred when cpu_ram_efficient_loading=True was enabled has been fixed. Several other minor improvements and fixes are also included—see the What’s Changed section for full details.

  • FULL_STATE_DICT have been enabled by @S1ro1 in #3527
  • QLoRA support by @winglian in #3546
  • set backend correctly for CUDA+FSDP2+cpu-offload in #3574
  • memory spike fixed when using cpu_ram_efficient_loading=True by @S1ro1 in #3482

Better HPU support:

We have added a documentation for Intel Gaudi hardware !
The support is already available since v1.5.0 through this PR.

Torch.compile breaking change for dynamic argument

We've updated the logic for setting self.dynamic to explicitly preserve None rather than defaulting to False when the USE_DYNAMIC environment variable is unset. This change aligns the behavior with the PyTorch documentation for torch.compile. Thanks to @yafshar for contributing this improvement in #3567.

What's Changed

Read more

Contributors

winglian, pstjohn, and 30 other contributors
Loading