This is the rocm-dev-infra branch — the default branch of the ROCm/xla fork of openxla/xla.
It hosts GitHub Actions workflows, CI/CD infrastructure, Debug and Performance Tools, the JAX Scaling Book for AMD GPUs, and serves as a central index for all ROCm JAX/XLA release branches.
The main branch is an exact mirror of openxla/xla:main and should never
contain fork-specific commits. This separation ensures the merge-upstream API
(fast-forward only) continues to work for automatic syncing.
- XLA Branches for JAX
- Recommended Docker Image && Build Instructions
- OpenXLA Upstream PR Status
- OpenXLA Upstream CI Checks
- Debug Tools
- Performance Tools
- Scaling Book
- Dashboards
- Workflows
For supported ROCm versions and GFX targets per release, see the AMD ROCm compatibility matrix.
| JAX Release | XLA Branch | JAX Branch |
|---|---|---|
| jax-ml/jax:main | openxla/xla:main |
jax-ml/jax:main |
| jax-v0.12.0 (rocm-pjrt-plugin) | rocm-jaxlib-v0.12.0 |
rocm-jaxlib-v0.12.0 |
| jax-v0.11.2 (rocm-pjrt-plugin) | rocm-jaxlib-v0.11.2 |
rocm-jaxlib-v0.11.2 |
| jax-v0.11.1 (rocm-pjrt-plugin) | rocm-jaxlib-v0.11.1 |
rocm-jaxlib-v0.11.1 |
Preceding released branches
Use the same ROCm-less build container as CI for reproducible local builds and CI parity:
docker pull ghcr.io/rocm/jax-build-ubu24.rocmless@sha256:68854ed30b800d3e8c390f6e651721e2bead58830da1c59e1b828354eb8f077f- This is the digest pinned in the
rocm-configjob of themainCI workflows; the image'slatesttag is a newer build that CI does not use. - The image is Ubuntu 24.04 with Python 3.12, Bazel (via Bazelisk), and the build dependencies needed by JAX and XLA, but it has no ROCm install (no
/opt/rocm).- Instead, Bazel downloads a TheRock ROCm tarball and builds against it with the hermetic Clang toolchain, so neither a system ROCm nor a system Clang is required.
- Mount your local
jaxandxlasource checkouts into the container (e.g. as/tf/jaxand/tf/xla), and pass--device=/dev/kfd --device=/dev/dri --group-add videotodocker runso tests can see the GPU, before running the build commands below.
Bazel picks the TheRock tarball from these flags, which the main CI jobs all pass:
--repo_env=ROCM_PATH=
--repo_env=ROCM_DISTRO_URL=https://stable.repo.amd.com/rocm/core/tarball/therock-dist-linux-multiarch-10.0.0.tar.gz
--repo_env=ROCM_DISTRO_HASH=1c5e807875d26a2470ecc7323daa5b5b9009208a55c3290ac255a909cde15fc6-
Keep
ROCM_PATHempty, and don't pass--rocm_pathtobuild/build.py. A non-emptyROCM_PATHtakes priority overROCM_DISTRO_URL, so Bazel would use a local ROCm install instead of the tarball. -
TheRock 10.0.0 is also XLA's default hermetic ROCm (
rocm_latest_multiarchinthird_party/gpus/rocm/rocm_redist.bzl); the twoROCM_DISTRO_*flags pin it explicitly. -
To build against a TheRock nightly, set
ROCM_DISTRO_HASH=no_hash(which skips the checksum) and pointROCM_DISTRO_URLat that nightly's tarball. The nightly URL differs between release lines and is subject to change, so check thetherock_nightlyentries inrocm_xla_ci_nightly.ymlandrocm_xla_ci_jax.ymlfor the current ones. At the time of writing:- 10.1:
https://rocm.nightlies.amd.com/tarball-multi-arch/therock-dist-linux-multiarch-<version>.tar.gz, with the version index athttps://rocm.nightlies.amd.com/whl-multi-arch/. - 10.2:
https://nightly.repo.amd.com/rocm/core/tarball/therock-dist-linux-multiarch-<version>.tar.gz, with the version index athttps://nightly.repo.amd.com/rocm/whl-next/.
To print the latest
<version>, pass the line's version index and number, for example for 10.2:GPU_TARGET=device-all THEROCK_INDEX_URL=https://nightly.repo.amd.com/rocm/whl-next/ ROCM_SDK_VERSION=10.2 \ bash build_tools/rocm/detect-therock-version.sh
- 10.1:
-
CI runs the XLA unit tests with
build_tools/rocm/run_xla_ci_build.shfromopenxla/xla, and builds and tests JAX withci/run_bazel_test_rocm_rbe.shfromjax-ml/jax, both with these flags on top of the ROCm RBE configs. Seerocm_xla_ci_nightly.ymlandrocm_xla_ci_jax.ymlfor the exact commands.
Inside the recommended rocmless container, with the JAX repo mounted at /tf/jax and the XLA repo mounted at /tf/xla:
cd /tf/jax
# Remove any pre-existing JAX/jaxlib installs first, in case a mixed-up
# jaxlib is present that would otherwise shadow the freshly built wheels.
python3 -m pip uninstall jax jaxlib jax-rocm10-pjrt jax-rocm10-plugin jax-rocm7-pjrt jax-rocm7-plugin -y
rm -rf dist/*
python build/build.py build \
--wheels=jax,jaxlib,jax-rocm-plugin,jax-rocm-pjrt \
--rocm_amdgpu_targets=gfx950 \
--rocm_version=10 \
--bazel_options=--override_module=xla=/tf/xla \
--bazel_options=--config=rocm_clang_hermetic \
--bazel_options=--repo_env=ROCM_PATH= \
--bazel_options=--repo_env=ROCM_DISTRO_URL=https://stable.repo.amd.com/rocm/core/tarball/therock-dist-linux-multiarch-10.0.0.tar.gz \
--bazel_options=--repo_env=ROCM_DISTRO_HASH=1c5e807875d26a2470ecc7323daa5b5b9009208a55c3290ac255a909cde15fc6 \
--bazel_options=--repo_env=TF_ROCM_AMDGPU_TARGETS=gfx950 \
--bazel_startup_options="--bazelrc=build/rocm/rocm.bazelrc"
pip install dist/*.whl
# The rocmless image has no ROCm runtime, so point the loader at the TheRock
# libraries Bazel extracted during the build.
ROCM_DIST="$(bazel --bazelrc=build/rocm/rocm.bazelrc info output_base)/external/xla++rocm_configure_ext+local_config_rocm/rocm/rocm_dist"
export LD_LIBRARY_PATH="${ROCM_DIST}/lib:${ROCM_DIST}/lib/rocm_sysdeps/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}"
# Run outside /tf/jax so Python imports the installed wheels, not the source tree.
cd /tmp && python3 -c "import jax; print(jax.devices())"Notes:
- Use
--override_module=xla=/tf/xla, NOT--override_repository=xla=.... Current JAX builds run under bzlmod (.bazelrcsets--enable_bzlmod --noenable_workspace), where XLA is a Bazel module whose canonical repo name isxla+(note the trailing+). The legacy--override_repository=xla=...targets a repo literally namedxla, which does not exist under bzlmod, so it is silently ignored — Bazel then builds XLA from the bzlmod-pinned commit and your local checkout is never compiled. - The example builds for
gfx950, the architecture CI builds and tests on. Adjust--rocm_amdgpu_targetsand--repo_env=TF_ROCM_AMDGPU_TARGETSto match your hardware (e.g.gfx90a,gfx942,gfx950,gfx1100), keeping both the same.- Multiple targets can be passed comma-separated.
--rocm_amdgpu_targetsalone does not reach Bazel's ROCm setup, becausebuild/rocm/rocm.bazelrcsetsTF_ROCM_AMDGPU_TARGETSto 27 architectures with--repo_env, which makes the build much slower and the wheels much larger.
- Multiple targets can be passed comma-separated.
- The
--wheels=jax,jaxlib,jax-rocm-plugin,jax-rocm-pjrtlist produces all four required wheels indist/;pip install dist/*.whlinstalls the matching set. Built against TheRock 10.0.0, the ROCm wheels are namedjax_rocm10_pluginandjax_rocm10_pjrt. - The last command should print
[RocmDevice(id=0), ...]. Without theLD_LIBRARY_PATHstep, JAX prints a plugin error (librocprofiler-sdk.so.1: cannot open shared object file) and falls back to[CpuDevice(id=0)]. SetLD_LIBRARY_PATHagain in every new shell, and rebuild after abazel clean --expunge, which deletes the extracted libraries.
Building the rocm-jaxlib-v0.8.x / rocm-jaxlib-v0.9.x release branches
For the 0.8.x and 0.9.x JAX releases (rows in the XLA Branches for JAX table above), the wheel build is driven by the ROCm/rocm-jax repository on its rocm-jax-infra branch — not by build/build.py in ROCm/jax directly. This is the legacy stack.py / build/ci_build path that pairs the matching rocm-jaxlib-v0.8.x / rocm-jaxlib-v0.9.x branches of ROCm/xla and ROCm/jax.
| JAX Version | Repo used for building | Builds jaxlib wheel? |
|---|---|---|
| 0.8.x | ROCm/rocm-jax |
Yes |
| 0.9.x | ROCm/rocm-jax |
Yes |
| 0.10.x and newer | ROCm/jax (artifact workflow / build/build.py) |
No (only when jaxlib changes) |
Starting with JAX 0.10.0, wheel builds move to the ROCm/jax fork's artifact workflow (ci/build_rocm_artifacts.sh → build/build.py); the stack.py entrypoint in rocm-jax is deprecated and the rocm-jax-infra branch is retained only for Dockerfiles and image-build infrastructure.
To build a 0.8.x / 0.9.x wheel set, follow the build and CI scripts in the rocm-jax-infra branch of ROCm/rocm-jax, pointing them at the corresponding rocm-jaxlib-v0.8.x / rocm-jaxlib-v0.9.x branches of this repository (ROCm/xla) and ROCm/jax. For main and the 0.11.x+ branches, use the build/build.py invocation shown in the previous section instead. The 0.10.x branches' XLA predates TheRock tarball support (ROCM_DISTRO_URL), so build those in the ROCm 7.2.4 image (ghcr.io/rocm/jax-ubu22.rocm7.2.4) with the same command, replacing the three --repo_env=ROCM_* options with --rocm_path=/opt/rocm and using --rocm_version=7.
Track pending and in-progress ROCm upstream contributions to OpenXLA on the OpenXLA:GPU - AMD/ROCm project board.
Every openxla/xla PR runs the upstream rocm_ci.yml workflow on a gfx950 runner, using the ROCm-less image and TheRock 10.0.0 tarball described in Recommended Docker Image && Build Instructions. It has two jobs:
| Job | Entry point | Purpose |
|---|---|---|
xla |
build_tools/rocm/run_xla_ci_build.sh (in openxla/xla) |
XLA test sweep. Runs the //xla/... single-GPU and CPU test suites. |
jax |
ci/run_bazel_test_rocm_rbe.sh (in jax-ml/jax) |
JAX test sweep against XLA-from-source. Builds jaxlib and JAX with Bazel against the PR's XLA and runs the JAX GPU tests. Runs only for PRs targeting main. |
Both jobs use the EngFlow internal.actinolite.cluster.engflow.com RBE cluster through the rocm_rbe / rocm_rbe_dynamic configs in XLA's build_tools/rocm/rocm_xla.bazelrc and JAX's build/rocm/rocm.bazelrc.
Debugging utilities for triaging issues while running MaxText-TE / JAX / TF workloads on AMD GPUs (ROCm). See debug_tools/ for full details. Individual tools live in their own subdirectories with self-contained usage notes.
| Tool | Purpose |
|---|---|
debug_tools/queue_dump |
Hang investigation. Drives gdb / rocgdb / umr / the ROCR Debug Agent / RDL to inspect a hung process and its GPU state (queues, kernels, wavefronts, fences). |
debug_tools/nan_check_tool |
NaN / numerical-correctness investigation. Scans intermediate buffers for NaN / Inf and localises the first offending HLO instruction, buffer index, and rank. |
Performance and regression micro-benchmarks for AMD GPUs (ROCm). See perf_tools/ for full details.
| Tool | Purpose |
|---|---|
perf_tools/rocm_latency_check_tool |
HIP-runtime-only micro-benchmark measuring end-to-end per-request dispatch latency (H2D → compute → D2D → compute → D2H) and comparing it across ROCm versions. Reports a latency percentile histogram plus QPS and can append version-stamped rows to a CSV for regression tracking. |
perf_tools/hlo_eval_tools |
Per-module HLO micro-benchmarks built on multihost_hlo_runner. Replays the pre-optimization HLO dumps (training and inference) from real JAX models one module at a time to measure each submodule's cost on XLA, and appends the timings to a CSV for regression tracking across ROCm/XLA versions. Covers models across four categories: LLM, vision, multimodal, and science. |
The scaling book is a companion to Google's JAX Scaling Book for AMD GPU users, covering performance features for JAX training at the implementation layer (ROCm libraries, MaxText, XLA internals, and JAX APIs).
The scaling_book/ directory contains the Jekyll site served on GitHub Pages. Page content can be found at scaling_book/pages/ with images at scaling_book/pages/img/.
Reference reproductions for the case studies can be found at the following locations:
| Folder | Description |
|---|---|
scaling_book/llama7b-jax-fundamentals |
Rematerialisation, attention backends, and profiling examples |
scaling_book/llama70b-mixed-precision-training |
Mixed quantization experiments in BF16, FP16, FP8, MXFP8, and MXFP4 |
scaling_book/mixtral8-22b-distributed-strategies |
Sharding and grouped GEMM experiments |
Three sections are published to GitHub Pages from this branch. Two of them are trend dashboards, and they are easy to mix up by name — they measure different things and share nothing but a layout:
| ci-metrics | perf-metrics | |
|---|---|---|
| Question | How long does CI take to answer a pull request? | How fast does the top of tree openxla/main runs 1gpu training and inference on collected hlo dumps? Secondly how fast rocm_latency_check_tool runs with the latest theRock nightly |
| Measures | time to signal, queue wait, pass rate, suite size, per lane | HLO module execution time and HIP latency, per ROCm stack |
| Source | the GitHub Actions API and job logs, per PR head commit | perf_tools/ replayed on a gfx950 |
| A point is | one day, pooled from overlapping nightly snapshots | one run, against one pinned openxla/xla commit |
| Collected by | rocm_xla_jax_metrics_nightly.yml → ci-metrics-data |
rocm_xla_perf_nightly.yml → perf-metrics-data |
| Built from | ci-metrics/ |
perf-metrics/ |
| Published by | ci_metrics_pages.yml |
perf_metrics_pages.yml |
The third section is the scaling book,
built from scaling_book/.
A Pages deployment replaces the whole site, so each of the three workflows
rebuilds all three sections through the shared build-pages-site action; a
workflow that published only its own section would delete the other two.
GitHub Actions workflows and PR labels that drive automation in this fork.
| Workflow | Schedule | Description |
|---|---|---|
sync_upstream.yml |
Every 3h (weekdays), daily (weekends) | Syncs main with openxla/xla:main via the GitHub merge-upstream API |
claude_auto_review.yml |
Every PR opened against a branch that has pr_event_dispatch.yml |
Automatically runs Claude-Opus-5.5 powered code review on every pull request |
rocm_xla_ci_nightly.yml |
Daily at 3:47 AM UTC with sanitizers and every 6 hours full unit test suite of XLA and JAX | Runs the latest XLA (openxla/xla) unit test suite, the TSAN and ASAN sanitizer builds (rocm_xla_ci_sanitizers.yml) and the latest JAX (jax-ml/jax) unit tests (rocm_xla_ci_jax.yml), each against the TheRock 10.0.0 release tarball and the latest TheRock nightly tarball (the XLA tests track the 10.2 nightlies, the sanitizer and JAX jobs the 10.1 nightlies) |
rocm_infra_monitor.yml |
Every 1h (also workflow_dispatch, with an optional run_id input to replay one specific monitored run on demand) |
Monitors rocm_xla_ci_nightly.yml and the sanitizer/JAX jobs it triggers for build, test, config and infra failures. Each failure signature not already analyzed gets a comment on a single long-lived tracking issue (created once, reused thereafter) and automatically triggers a Claude Code debug run on real GPU hardware; a recurrence of an already-analyzed signature is skipped entirely (no new comment, no new debug run) - see Nightly Failure Debugging (Claude) below. |
rocm_xla_jax_metrics_nightly.yml |
Daily at 3:47 AM UTC | Analyzes 1 day window of PRs submitted to upstream openxla/xla and jax-ml/jax. Collects CI latency, time to signal, mean and p90 values for ROCm and vendor CI actions. Dumps markdown report at ci-metrics-report and data at ci-metrics-data |
rocm_xla_perf_nightly.yml |
Daily at 2:00 AM UTC | Runs on upstream openxla/xla hlo_eval_tools on 1 gpu for the collected inference and training hlo workloads and rocm_latency_check_tools on 1, 2 and 4 gpus on theRock 10.0 10.1 and 10.2 (rc). Perf data is pushed to perf-metrics-data |
scaling_book_pages.yml |
Push to rocm-dev-infra when scaling_book/ changes |
Builds the scaling book from scaling_book/ and deploys to GitHub Pages at rocm.github.io/xla/scaling-book |
ci_metrics_pages.yml |
After each rocm_xla_jax_metrics_nightly.yml run completes, and on push to rocm-dev-infra when ci-metrics/ changes |
Builds the CI trend dashboard - how long CI takes to answer a PR - from ci-metrics/ and the ci-metrics-data branch, and deploys to GitHub Pages at rocm.github.io/xla/ci-metrics. A Pages deployment replaces the whole site, so it rebuilds every section (the scaling book and the perf dashboard too) via the shared build-pages-site action |
perf_metrics_pages.yml |
After each rocm_xla_perf_nightly.yml run completes, and on push to rocm-dev-infra when perf-metrics/ changes |
Builds the perf trend dashboard - how fast the benchmarked modules run, per ROCm stack - from perf-metrics/ and the perf-metrics-data branch, and deploys to GitHub Pages at rocm.github.io/xla/perf-metrics. Rebuilds every section for the same reason as above. Deploys on a failed perf nightly too, since a nightly that loses one matrix mode still publishes the others; only a cancelled run is skipped |
PRs opened against branches in the ROCm/xla fork can opt-in to additional automated checks by attaching labels:
| Label | Effect |
|---|---|
claude-review |
Triggers an automated Claude code review on the PR. |
sanitizers |
Runs sanitizer-instrumented builds and tests: ThreadSanitizer (TSAN) and AddressSanitizer (ASAN). |
rocm_infra_monitor.yml watches rocm_xla_ci_nightly.yml and the sanitizer/JAX jobs it calls. Every hour it checks for monitored runs that finished since its own last successful run, and for every build/test/step failure it finds:
- Skips it if it's already been analyzed. Every failure is identified by a
job/stepsignature. If that exact signature already has a debug report comment on the tracking issue, the whole thing is a no-op: no new comment, no new debug run - a long-standing, already-understood failure doesn't keep adding noise every time it recurs. - Otherwise, reports it. A single, long-lived GitHub issue titled "ROCm CI Nightly Failures Detected" (label
rocm-infra-error) is created once and reused forever after - every new (not-yet-analyzed) failure signature is appended to it as its own comment, never a new issue, so the issue is a running history of distinct failures rather than every recurrence of them. - Debugs it, once per failure signature. A "debug" job is queued (as a matrix entry of the same workflow run - see
build_tools/rocm/collect_claude_debug_targets.sh) on thebuild-only-xlarunner (the GPU runner pool can't reach the LLM gateway) inside therocmlesscontainer, checking out the exact commit the failing job actually tested (parsed from that job's own log, not the triggering run's metadata - seebuild_tools/rocm/get_checkout_source.sh, since the monitored jobs testopenxla/xla@mainwhile the run that triggers them lives inROCm/xla), and runs thedebug-nightly-failureClaude Code skill against it. - Triages and estimates, fast - and checks again for a race. As its own first step, the skill re-checks the tracking issue for an existing report on this signature (defense in depth against a race between two monitor runs) and posts just a reference to it if found. Otherwise it classifies the failure into
nightly-build-failure,nightly-test-failure,nightly-config-failure, ornightly-infra-failure, applies the corresponding label, and posts a preliminary comment with that category and an explicitly-caveated "possible root cause" guess. This whole triage/analysis phase (including checking whether a later upstream commit already fixed it, and a quick code read) is hard-capped at 5 minutes. - Reproduces only if there's good reason to and time allows. Building/testing afterward doesn't count against that 5-minute cap, but a full RBE reproduction (narrowed to the single failing target - never the whole
//xla/...sweep the original CI step used) withgdb/VLOG/kernel-disassembly debugging is optional and only attempted when the skill is confident it'll finish in time; otherwise it reports a quick estimate instead. - Reports the outcome as one final comment on the tracking issue, quoting the detection comment it addresses: an already-fixed-upstream commit, a verified fix (confirmed by re-running the target), an unverified "possible fix" estimate, or a detailed analysis if it couldn't resolve anything. It never pushes a branch, opens a PR, or edits anything outside that one issue.
To test the whole pipeline against one specific monitored run instead of waiting for the hourly scan, dispatch it manually with that run's ID: gh workflow run rocm_infra_monitor.yml --repo ROCm/xla -f run_id=<run-id>.