Skip to content

Raise on every rank when the native DeepEP constructor fails on one - #46

Open
almogtavor wants to merge 1 commit into
mainfrom
native-ctor-failure
Open

almogtavor wants to merge 1 commit into
mainfrom
native-ctor-failure

Conversation

@almogtavor

Copy link
Copy Markdown
Member

Follow-up to the review of #37: a failure inside the native constructor on one rank left its peers waiting in NCCL.

  • The native constructor keeps only the steps that can fail on one rank alone (property query, ncclMemAlloc, host workspace).
  • The collective steps (ncclDevCommCreate, ncclCommWindowRegister) move to a new connect().
  • afdep/deepep.py gathers the construction status of every rank between the two, so every rank raises together.

Limit: a failure inside connect() itself is still one-sided.

Checked on 4x H100, native source digest 62530ba60db1838a85338c79a2d6226b37b87a689229e90aa27e5c51506d0cc9: tests.gpu.construct_failure (every rank raises), tests.gpu.deepep with --payload fp8 and bf16, unit tests, Ruff.

The native constructor now holds only the steps that can fail on one rank
alone. The collective steps (ncclDevCommCreate, ncclCommWindowRegister) move
to connect(), and the ranks gather their construction status between the two.
@almogtavor
almogtavor requested a review from Etelis as a code owner November 20, 2026 23:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant