HIP Multi-arch and amdgcnspirv offload target update by khuck-AMD · Pull Request #9384 · kokkos/kokkos

khuck-AMD · 2026-07-28T14:28:09Z

This patch adds support for HIP multi-arch builds and adds amdgcnspirv as an explicit Kokkos architecture target.

What is done:

Adds Kokkos_ARCH_AMD_GCNSPIRV / KOKKOS_ARCH_AMD_GCNSPIRV.
Allows selecting multiple AMD GPU targets in CMake and propagates them to HIP offload flags.
Updates exported package checks so downstream GPU_TARGETS can be a subset of the enabled Kokkos HIP architectures.
Switches HIP warp-size-dependent host logic to runtime values from hipDeviceProp_t.
Moves HIP arch capability checks (including XNACK-related behavior) to runtime detection.
Updates HIP tests and adds TestHIP_RuntimeArch.cpp coverage.

Result:

Multi-arch HIP builds now supported
amdgcnspirv is recognized end-to-end.
Performance comparison vs the prior compile-time path shows no meaningful regression (largest observed changes were low single-digit percentages).

For this PR, the runtime HIP architecture path with multi-arch support was tested to perform essentially the same as the compile-time path on MI300A (within system noise).

STREAM: mixed small changes, best +0.95% (Set), worst -1.46% (Add)
Reduction: effectively unchanged (-0.65% and -0.06%)
Launch latency: +0.08% (V=1) and +1.13% (V=30000)

Related issues / PRs

none.

Changelog Entry

HIP configurations now support the HIP offload target architecture -DKokkos_ARCH_AMD_GCNSPIRV which provides runtime support for any architecture
HIP configurations now support multiple offload targets in the same configuration/build.
There is now one gfx942 binary target - there is runtime branching for APU-vs-descrete-GPU behavior using HIP properties rather than separate compile-time specialization for MI300A vs MI300X, although the existing Kokkos arch CMake options are still honored.

This patch adds support for HIP multi-arch builds and adds `amdgcnspirv` as an explicit Kokkos architecture target. What is done: - Adds `Kokkos_ARCH_AMD_GCNSPIRV` / `KOKKOS_ARCH_AMD_GCNSPIRV`. - Allows selecting multiple AMD GPU targets in CMake and propagates them to HIP offload flags. - Updates exported package checks so downstream `GPU_TARGETS` can be a subset of the enabled Kokkos HIP architectures. - Switches HIP warp-size-dependent host logic to runtime values from `hipDeviceProp_t`. - Moves HIP arch capability checks (including XNACK-related behavior) to runtime detection. - Updates HIP tests and adds `TestHIP_RuntimeArch.cpp` coverage. Result: - Multi-arch HIP builds now supported - `amdgcnspirv` is recognized end-to-end. - Performance comparison vs the prior compile-time path shows no meaningful regression (largest observed changes were low single-digit percentages). For this PR, the runtime HIP architecture path performs essentially the same as the compile-time path on MI300A. - STREAM: mixed small changes, best +0.95% (`Set`), worst -1.46% (`Add`) - Reduction: effectively unchanged (-0.65% and -0.06%) - Launch latency: +0.08% (`V=1`) and +1.13% (`V=30000`) Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Co-authored-by: Trévis Morvany <63788850+tretre91@users.noreply.github.com> Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

tretre91

There are some integer comparison warnings in the CI related to HIPTraits::WarpSize() https://cloud1.cees.ornl.gov/jenkins-ci/job/Kokkos/job/PR-9384/4/stages/

tretre91 · 2026-08-04T13:47:51Z

+  if(KOKKOS_HIP_ARCHITECTURES)
+    list(LENGTH KOKKOS_HIP_ARCHITECTURES KOKKOS_HIP_ARCH_COUNT_FOR_OPENACC)
+    if(KOKKOS_HIP_ARCH_COUNT_FOR_OPENACC GREATER 1)
+      message(
+        FATAL_ERROR
+          "Kokkos_ENABLE_OPENACC currently supports only one AMD GPU architecture, but got ${KOKKOS_HIP_ARCHITECTURES}."
+      )
+    endif()
+  endif()


Does SYCL support multiple archs? If not we should add the same check with KOKKOS_ENABLE_SYCL

Since multiple intel archs are not supported, I would assume the same for multiple AMD archs

macro(CHECK_MULTIPLE_INTEL_ARCH) if(KOKKOS_ARCH_INTEL_GPU) message(FATAL_ERROR "Specifying multiple Intel GPU architectures is not allowed!") endif() set(KOKKOS_ARCH_INTEL_GPU ON) endmacro()

@masterleinad can you confirm ?

Yes, we haven't allowed multiple GPU or CPU architectures for any backend.

khuck-AMD · 2026-08-04T17:38:08Z

There are some integer comparison warnings in the CI related to HIPTraits::WarpSize() https://cloud1.cees.ornl.gov/jenkins-ci/job/Kokkos/job/PR-9384/4/stages/

good catch. Fixing that now...

Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Co-authored-by: Trévis Morvany <63788850+tretre91@users.noreply.github.com> Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

dalg24

(partial review)

dalg24 · 2026-08-13T14:07:21Z

    }
    block_size >>= 1;
-  } while (block_size >= HIPTraits::WarpSize);
+  } while (block_size >= static_cast<unsigned>(HIPTraits::WarpSize()));


Why were we not getting warnings before?

Probably because it was replaced with a compile-time constant that did not have a type, just a value?

Co-authored-by: Damien L-G <dalg24+github@gmail.com> Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Signed-off-by: Kevin Huck <kevin.huck@gmail.com>

dalg24

(unfinished review but sending it now because I don't want to sit on these for too long)

Co-authored-by: Damien L-G <dalg24+github@gmail.com> Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Co-authored-by: Trévis Morvany <63788850+tretre91@users.noreply.github.com> Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

…as well as amdgcnspirv special case. Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

tretre91

Other than the comment below and a comment from Damien about mixing west-east const, LGTM

tretre91 · 2026-09-21T21:45:37Z

+  if(KOKKOS_HIP_ARCHITECTURES)
+    list(LENGTH KOKKOS_HIP_ARCHITECTURES KOKKOS_HIP_ARCH_COUNT_FOR_OPENACC)
+    if(KOKKOS_HIP_ARCH_COUNT_FOR_OPENACC GREATER 1)
+      message(
+        FATAL_ERROR
+          "Kokkos_ENABLE_OPENACC currently supports only one AMD GPU architecture, but got ${KOKKOS_HIP_ARCHITECTURES}."
+      )
+    endif()
+  endif()


Since multiple intel archs are not supported, I would assume the same for multiple AMD archs

macro(CHECK_MULTIPLE_INTEL_ARCH) if(KOKKOS_ARCH_INTEL_GPU) message(FATAL_ERROR "Specifying multiple Intel GPU architectures is not allowed!") endif() set(KOKKOS_ARCH_INTEL_GPU ON) endmacro()

@masterleinad can you confirm ?

tretre91 · 2026-09-21T22:20:51Z

You should also rebase to get some fixes for the CI failures

tretre91 reviewed Jul 30, 2026

View reviewed changes

Comment thread core/src/impl/Kokkos_Core.cpp

Comment thread core/src/HIP/Kokkos_HIP.cpp Outdated

Comment thread core/src/HIP/Kokkos_HIP_IsXnack.cpp Outdated

khuck-AMD and others added 3 commits July 30, 2026 16:08

Print all configured HIP architectures, not just the first one

73dcf46

Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Update core/src/HIP/Kokkos_HIP.cpp

819529f

Co-authored-by: Trévis Morvany <63788850+tretre91@users.noreply.github.com> Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Update core/src/HIP/Kokkos_HIP_IsXnack.cpp

896bbb1

Co-authored-by: Trévis Morvany <63788850+tretre91@users.noreply.github.com> Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

tretre91 reviewed Jul 31, 2026

View reviewed changes

Comment thread core/src/HIP/Kokkos_HIP.cpp

Don't issue warning for arch mismatch with amdgcnspirv

432aecb

Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

KillianBabilotte mentioned this pull request Aug 4, 2026

Add direct register query to clock_tic_device() for backends without wrappers. #9392

Draft

tretre91 reviewed Aug 4, 2026

View reviewed changes

khuck-AMD and others added 3 commits August 4, 2026 13:00

Fixing mismatched unsigned/int type comparisons exposed by -Werror flag.

1dc0c17

Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Update core/src/HIP/Kokkos_HIP.cpp

b3646a5

Co-authored-by: Trévis Morvany <63788850+tretre91@users.noreply.github.com> Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Merge branch 'develop' into multiarch-support

7821d1c

Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

dalg24 reviewed Aug 17, 2026

View reviewed changes

khuck-AMD and others added 2 commits August 25, 2026 13:30

Update core/src/HIP/Kokkos_HIP_IsXnack.cpp

317f3ee

Co-authored-by: Damien L-G <dalg24+github@gmail.com> Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Update core/src/HIP/Kokkos_HIP_Instance.hpp

ecb3328

Co-authored-by: Damien L-G <dalg24+github@gmail.com> Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

dalg24 reviewed Aug 26, 2026

View reviewed changes

dalg24 mentioned this pull request Aug 26, 2026

Signedness of word size type used in CUDA and HIP reductions changes with the size of reduced variable #9477

Open

khuck-AMD and others added 4 commits August 26, 2026 13:24

Update core/src/HIP/Kokkos_HIP_Instance.hpp

3f2b51f

Co-authored-by: Damien L-G <dalg24+github@gmail.com> Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Making changes requested in review.

6f6f772

Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Addressing requests from code review

6c4725c

Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Fixing clang-format issues

9c95f22

Signed-off-by: Kevin Huck <kevin.huck@gmail.com>

dalg24 reviewed Aug 27, 2026

View reviewed changes

tretre91 reviewed Aug 27, 2026

View reviewed changes

Comment thread core/src/HIP/Kokkos_HIP_IsXnack.cpp Outdated

khuck-AMD and others added 6 commits August 27, 2026 08:45

Update core/src/HIP/Kokkos_HIP_ReduceScan.hpp

a6defc8

Co-authored-by: Damien L-G <dalg24+github@gmail.com> Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Update core/src/HIP/Kokkos_HIP_Instance.hpp

6d2af43

Co-authored-by: Damien L-G <dalg24+github@gmail.com> Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Update core/src/HIP/Kokkos_HIP_Instance.hpp

36ccda5

Co-authored-by: Damien L-G <dalg24+github@gmail.com> Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Update core/src/HIP/Kokkos_HIP_Instance.hpp

ba24c1d

Co-authored-by: Damien L-G <dalg24+github@gmail.com> Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Update core/src/HIP/Kokkos_HIP_IsXnack.cpp

1552ba2

Co-authored-by: Damien L-G <dalg24+github@gmail.com> Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Update core/src/HIP/Kokkos_HIP_IsXnack.cpp

aabbb11

Co-authored-by: Trévis Morvany <63788850+tretre91@users.noreply.github.com> Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

Addressing review comments and fixing config print of multi-gpu arch …

23bc9b4

…as well as amdgcnspirv special case. Signed-off-by: Kevin Huck <Kevin.Huck@amd.com>

tretre91 reviewed Sep 21, 2026

View reviewed changes

Conversation

khuck-AMD commented Jul 28, 2026

Related issues / PRs

Changelog Entry

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tretre91 left a comment

Choose a reason for hiding this comment

Uh oh!

tretre91 Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

tretre91 Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

masterleinad Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

khuck-AMD commented Aug 4, 2026

Uh oh!

dalg24 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dalg24 Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

khuck-AMD Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dalg24 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tretre91 left a comment

Choose a reason for hiding this comment

Uh oh!

tretre91 Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

tretre91 commented Sep 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants