Skip to content

Fix s1_train DDP crash on Windows single-GPU (sm_120 / Blackwell) - #2774

Open
lshhhhhhh wants to merge 1 commit into
RVC-Boss:mainfrom
lshhhhhhh:fix-win-single-gpu-ddp-crash
Open

lshhhhhhh wants to merge 1 commit into
RVC-Boss:mainfrom
lshhhhhhh:fix-win-single-gpu-ddp-crash

Conversation

@lshhhhhhh

Copy link
Copy Markdown

Summary

On Windows + single GPU + CUDA 12.8 + PyTorch 2.7+ on Blackwell (sm_120) hardware, s1_train.py crashes with a native access violation (exit code 3221225477 / 0xC0000005) shortly after pytorch_lightning's Trainer.fit() is called, before the first batch runs.

Repro

OS:      Windows 11
GPU:     RTX 5090 (compute capability 12.0, sm_120, Blackwell)
CUDA:    12.8
PyTorch: 2.11.0+cu128
Lightning: 2.6.1

Run any voice training through the standard webui / s1_train.py pipeline. The s1 (GPT) stage prints model summary, dataloader warnings, then exits with 3221225477 and no Python traceback.

Root cause

s1_train.py constructs the Trainer with DDPStrategy(backend="gloo") on Windows, regardless of GPU count. The gloo + sm_120 + CUDA 12.8 combination has a known incompatibility — confirmed by an independent PyTorch forum report titled "[Solved] RTX 5090 (sm_120) Training Segfault — DDP Was the Cause" — and produces a native crash inside Lightning's training-loop setup.

For a single GPU, DDP brings only process-management overhead and no parallelism benefit. Letting Lightning pick `strategy="auto"` selects `single_device`, avoiding DDP entirely.

Changes

Two narrowly-scoped patches, both Windows-conditional:

GPT_SoVITS/s1_train.py

  • On Windows, use strategy=\"auto\" and devices=1 instead of DDPStrategy(\"gloo\") + devices=-1.
  • Linux / macOS behaviour is unchanged — still NCCL DDP across all GPUs.

GPT_SoVITS/AR/data/bucket_sampler.py

  • DistributedBucketSampler previously assumed a distributed group was always initialized. With strategy=\"auto\" on a single GPU, Lightning does not initialize the group, so dist.get_world_size() raises.
  • Now: if the group is not initialized, fall back to a single-replica configuration (num_replicas=1, rank=0).
  • Defensive change — behaviour is unchanged whenever DDP is initialized.

Testing

  • Windows 11 + RTX 5090 (sm_120) + CUDA 12.8 + PyTorch 2.11+cu128 + Lightning 2.6.1
  • 15-epoch v4 LoRA training on a 30-min Chinese dataset (`batch_size=4`, `lora_rank=32`)
  • Pre-patch: crashes with `3221225477` after model load, no checkpoints saved.
  • Post-patch: training completes cleanly through all 15 epochs, weights saved to `GPT_weights_v4/` as expected.

Closes

Closes #2626 ("`RuntimeError: makeDeviceForHostname(): unsupported gloo device`" — same root cause).

On Windows with a single GPU running CUDA 12.8 + PyTorch 2.7+ on Blackwell
(sm_120) hardware, s1_train.py crashes with an access violation (exit code
3221225477) shortly after pytorch_lightning's Trainer initialization, before
the first batch runs.

Root cause: DDPStrategy with the gloo backend is forced on Windows even
when there's only one GPU. The gloo + sm_120 + CUDA 12.8 combination has a
known incompatibility (see PyTorch forum "[Solved] RTX 5090 sm_120 Training
Segfault - DDP Was the Cause") that produces a native crash inside the
Lightning training loop.

Two changes, scoped to Windows + CUDA only:

  * GPT_SoVITS/s1_train.py: on Windows, use Lightning's "auto" strategy,
    which picks `single_device` for one GPU and skips DDP entirely. Also
    pin devices=1 on Windows so multi-GPU users don't accidentally enable
    DDP. Non-Windows behaviour is unchanged (NCCL DDP, all available GPUs).
  * GPT_SoVITS/AR/data/bucket_sampler.py: when the distributed process
    group isn't initialized (i.e. running under single_device strategy),
    fall back to a single-replica configuration instead of crashing in
    dist.get_world_size(). Defensive change — behaviour is unchanged when
    DDP is properly initialized.

Tested on:
  * Windows 11 + RTX 5090 (sm_120) + CUDA 12.8 + PyTorch 2.11+cu128
    15-epoch s1 training completes cleanly, weights saved as expected.

Closes RVC-Boss#2626.
@aoi-umi

aoi-umi commented Aug 7, 2026

Copy link
Copy Markdown

s2_train exists same issue, use V2/V2Pro

dist.init_process_group(
backend="gloo" if os.name == "nt" or not torch.cuda.is_available() else "nccl",

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RuntimeError: makeDeviceForHostname(): unsupported gloo device

3 participants