Skip to content

[ray, doc] fix: make Slurm Ray network interface configurable - #7386

Merged
wuxibin89 merged 1 commit into
verl-project:mainfrom
SkyFishMoon:codex/slurm-ray-network-interface
Aug 13, 2026
Merged

wuxibin89 merged 1 commit into
verl-project:mainfrom
SkyFishMoon:codex/slurm-ray-network-interface

Conversation

@SkyFishMoon

@SkyFishMoon SkyFishMoon commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

This PR makes the network interface used by Ray configurable in the Slurm launch example.

On multi-NIC clusters, users can set RAY_NETWORK_INTERFACE to resolve the IPv4 address of that interface independently on every allocated node. The example then passes the resolved address to ray start --node-ip-address for both the head and worker nodes. If the selected interface does not have an IPv4 address on a node, the script exits with a clear error instead of silently using another interface.

The existing behavior remains unchanged when RAY_NETWORK_INTERFACE is not set. The multinode documentation now explains how to use the option and clarifies that Gloo and NCCL have separate interface settings.

Checklist Before Starting

Test

The following checks were run:

bash -n examples/tutorial/slurm/ray_on_slurm.slurm
git diff --check
python3 tests/special_sanity/check_docs_time_info.py
pre-commit run --all-files --show-diff-on-failure --color=always

The IPv4 resolver was also tested with mocked srun output for both successful CIDR-to-IPv4 extraction and the failure path where the selected interface has no IPv4 address.

The change was validated on a four-node Slurm cluster with a 1 Gbps management interface (lan0) and a 200 Gbps HDR interface (ib0). Before explicitly selecting the interface, Ray actors registered with 10.241.* addresses from lan0. With RAY_NETWORK_INTERFACE=ib0, the Ray head and workers registered with 10.245.* addresses from ib0, and the distributed training job completed successfully.

API and Usage Example

The new setting is optional. Existing users do not need to change their launch command.

On a multi-NIC cluster, select the interface when submitting the job:

RAY_NETWORK_INTERFACE=ib0 \
GLOO_SOCKET_IFNAME=ib0 \
NCCL_SOCKET_IFNAME=ib0 \
sbatch examples/tutorial/slurm/ray_on_slurm.slurm

RAY_NETWORK_INTERFACE controls Ray's node addresses only. GLOO_SOCKET_IFNAME and NCCL_SOCKET_IFNAME remain explicit because they configure separate communication layers.

Design & Code Changes

  • Add the optional RAY_NETWORK_INTERFACE environment variable to the Slurm example.
  • Resolve the selected interface's IPv4 address on each node through srun on the host.
  • Bind both the Ray head and workers with --node-ip-address when the option is set.
  • Fail early with the node and interface name when address resolution fails.
  • Preserve the original address-discovery behavior when the option is unset.
  • Document multi-NIC usage and the boundary between Ray, Gloo, and NCCL settings.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

  • Read the Contribute Guide.
  • Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always.
  • Add / Update the documentation.
  • Add unit or end-to-end test(s) to the CI workflow, or explain why this is not feasible: an end-to-end test requires a multi-node Slurm allocation with multiple network interfaces. The shell logic was tested with mocked srun output and validated on a real four-node Slurm cluster.
  • Once the PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace or the Feishu group.
  • Recipe submodule update: not applicable.

AI assistance disclosure

OpenAI Codex assisted with the implementation and documentation. I reviewed every changed line and ran the validation described above.

Assisted-by: OpenAI Codex
Signed-off-by: SkyFishMoon <yangtianyu612@gmail.com>
@CLAassistant

CLAassistant commented Aug 13, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@SkyFishMoon
SkyFishMoon marked this pull request as ready for review August 13, 2026 09:09
@wuxibin89
wuxibin89 merged commit e208d93 into verl-project:main Aug 13, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants