Repository navigation
[vllm] fix: prevent scheduler resume race during async weight sync - #7846
Merged
wuxibin89 merged 1 commit intoSep 18, 2026
Merged
Conversation
lxb007981
force-pushed
the
fix_vllm_dp_resume_gate_race
branch
from
September 14, 2026 07:24
837f40f to
2396c83
Compare
lxb007981
marked this pull request as ready for review
September 14, 2026 07:41
lxb007981
requested review from
HollowMan6,
PeterSH6 and
wuxibin89
as code owners
September 14, 2026 07:41
lxb007981
force-pushed
the
fix_vllm_dp_resume_gate_race
branch
from
September 16, 2026 02:27
2396c83 to
673561a
Compare
Fully async weight synchronization can deadlock when a duplicate scheduler resume overlaps with the start of a new DP request wave. Restoring the KV cache implicitly resumes scheduling, and the subsequent resume_generation() call resumes it again. In affected vLLM versions, resume_scheduler() branches on the rank-local engines_running flag. A new request wave can cause some ranks to skip the resume collective while others enter it, leaving EngineCores and model workers waiting on each other until Gloo times out. Split replica resume into two ordered phases: - Keep request admission closed while the head server resumes all DP EngineCores and waits for the resume RPC to complete. - Open submission gates on all servers only after engine resume returns. This prevents waiting generation tasks from starting a new DP wave while the duplicate resume is still in progress. Also order the server-level resume_generation() wrapper to resume engines before opening its gate. Keep the explicit resume because it is still required when KV-cache restoration is skipped.
lxb007981
force-pushed
the
fix_vllm_dp_resume_gate_race
branch
from
September 17, 2026 09:41
673561a to
e573a97
Compare
wuxibin89
approved these changes
Sep 18, 2026
8 tasks
28 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fully async weight synchronization can deadlock when a duplicate scheduler resume overlaps with the start of a new DP request wave.
Restoring the KV cache implicitly resumes vllm scheduling, and the subsequent resume_generation() call resumes it again. In the current vLLM version, resume_scheduler() branches on the rank-local
engines_runningflag. A new request wave can cause some ranks to skip the resume collective while others enter it, leaving EngineCores and model workers waiting on each other until Gloo times out.Split replica resume into two ordered phases:
This prevents waiting generation tasks from starting a new DP wave while the duplicate resume is still in progress. Also order the server-level resume_generation() wrapper to resume engines before opening its gate.
Keep the explicit resume because it is still required when KV-cache restoration is skipped.
See here for detailed walkthrough about the race
There is a race that triggers a deadlock during weight synchronization in fully async mode. Some ranks are blocked inside
DPEngineCoreProc.resume_scheduler(), while some ranks are waiting inside model execution until they both timeout. The root cause is the race between duplicate scheduler resume and the start of a new DP request wave.During each weight update, verl calls the same vLLM
DPEngineCoreProc.resume_scheduler()method twice:wake_up(tags=["kv_cache"])restores the remaining memory and resumes scheduling.resume_generation()call.The affected vLLM implementation is not idempotent: a duplicate resume can enter a collective even though scheduling is already unpaused. It branches on the rank-local
engines_runningflag. Since request-wave notifications and resume RPCs arrive independently, one rank can seeTrueand return immediately, while another seesFalseand enters the resume barrier. That creates a circular wait:This deadlock eventually reported as Gloo timeouts.
vLLM PR #55229, currently a draft, addresses this vllm issue by recognizing duplicate resumes using scheduler pause state and the wave gate. Already-unpaused ranks skip synchronization consistently, while genuine resumes retain the barrier.
Related PRs: #7511
Checklist Before Starting
[{modules}] {type}: {description}(This will be checked by the CI){modules}includefsdp,megatron,veomni,sglang,vllm,rollout,trainer,ci,training_utils,recipe,hardware,deployment,ray,worker,single_controller,misc,perf,model,algo,env,tool,ckpt,doc,data,cfg,reward,fully_async,one_step_off,like[megatron, fsdp, doc]{type}is infeat,fix,refactor,chore,test[BREAKING]to the beginning of the title.[BREAKING][fsdp, megatron] feat: dynamic batchingTest
Verified under
verl.experimental.fully_async_policy.fully_async_mainxfully_async_ppo_megatron_trainer.yamlpresets, fully async training on GLM-5.2.API and Usage Example
No APIs changed.