Conversation
On Windows with a single GPU running CUDA 12.8 + PyTorch 2.7+ on Blackwell
(sm_120) hardware, s1_train.py crashes with an access violation (exit code
3221225477) shortly after pytorch_lightning's Trainer initialization, before
the first batch runs.
Root cause: DDPStrategy with the gloo backend is forced on Windows even
when there's only one GPU. The gloo + sm_120 + CUDA 12.8 combination has a
known incompatibility (see PyTorch forum "[Solved] RTX 5090 sm_120 Training
Segfault - DDP Was the Cause") that produces a native crash inside the
Lightning training loop.
Two changes, scoped to Windows + CUDA only:
* GPT_SoVITS/s1_train.py: on Windows, use Lightning's "auto" strategy,
which picks `single_device` for one GPU and skips DDP entirely. Also
pin devices=1 on Windows so multi-GPU users don't accidentally enable
DDP. Non-Windows behaviour is unchanged (NCCL DDP, all available GPUs).
* GPT_SoVITS/AR/data/bucket_sampler.py: when the distributed process
group isn't initialized (i.e. running under single_device strategy),
fall back to a single-replica configuration instead of crashing in
dist.get_world_size(). Defensive change — behaviour is unchanged when
DDP is properly initialized.
Tested on:
* Windows 11 + RTX 5090 (sm_120) + CUDA 12.8 + PyTorch 2.11+cu128
15-epoch s1 training completes cleanly, weights saved as expected.
Closes RVC-Boss#2626.
|
s2_train exists same issue, use V2/V2Pro GPT-SoVITS/GPT_SoVITS/s2_train.py Lines 80 to 81 in d523079 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On Windows + single GPU + CUDA 12.8 + PyTorch 2.7+ on Blackwell (
sm_120) hardware,s1_train.pycrashes with a native access violation (exit code 3221225477/0xC0000005) shortly afterpytorch_lightning'sTrainer.fit()is called, before the first batch runs.Repro
Run any voice training through the standard webui /
s1_train.pypipeline. The s1 (GPT) stage prints model summary, dataloader warnings, then exits with3221225477and no Python traceback.Root cause
s1_train.pyconstructs the Trainer withDDPStrategy(backend="gloo")on Windows, regardless of GPU count. Thegloo+sm_120+ CUDA 12.8 combination has a known incompatibility — confirmed by an independent PyTorch forum report titled "[Solved] RTX 5090 (sm_120) Training Segfault — DDP Was the Cause" — and produces a native crash inside Lightning's training-loop setup.For a single GPU, DDP brings only process-management overhead and no parallelism benefit. Letting Lightning pick `strategy="auto"` selects `single_device`, avoiding DDP entirely.
Changes
Two narrowly-scoped patches, both Windows-conditional:
GPT_SoVITS/s1_train.pystrategy=\"auto\"anddevices=1instead ofDDPStrategy(\"gloo\")+devices=-1.GPT_SoVITS/AR/data/bucket_sampler.pyDistributedBucketSamplerpreviously assumed a distributed group was always initialized. Withstrategy=\"auto\"on a single GPU, Lightning does not initialize the group, sodist.get_world_size()raises.num_replicas=1,rank=0).Testing
Closes
Closes #2626 ("`RuntimeError: makeDeviceForHostname(): unsupported gloo device`" — same root cause).