Add an option to disable CUDA constant memory dispatch; add CUPTI-based tests to check that various expected launch types are used - #9542

Thank you for incorporating my change. I will close my one.

OK we need to talk about this a bit more. I am not sure that introducing an IMPL option makes too much sense. Clearly this is meant for downstream users to use, and we can't come back in the end "oh you have a problem with this, too bad this is just impl stuff".

Ah yeah, non-impl makes more sense. I "just" copied that convention from the PR this superseded and didn't think about it too much. I'll change that up tomorrow.

Also KOKKOS_IMPL_CUDA_USE_GRID_CONSTANT is now unused or? Probably can be removed (that btw seems like a change we could do either way).

It's still used here to specify launches with/without __grid_constant__; it's still defined as

It's CUDA 11.7+ (so relevant for all CUDA versions supported by Kokkos), but unfortunately it's only Volta+, which is why it needs those details, and why we need 4k vs 32k values flying around, for ex, lower down in Kokkos_Cuda_Instance.hpp. and redundantly in Kokkos_Cuda_KernelLaunch.hpp.

As for where all of the #ifdef KOKKOS_IMPL_USE_GRID_CONSTANT went in that file, it's just wrapped up (for ex) in this diff:

+  static constexpr CudaSpace::size_type KernelArgumentLimit =
#ifdef KOKKOS_IMPL_CUDA_USE_GRID_CONSTANT
-       0x008000; /* 32k bytes */
+  static constexpr bool GridConstantLaunchEnabled = true;
#else
-      0x001000; /*  4k bytes */
+  static constexpr bool GridConstantLaunchEnabled   = false;
#endif
+  static constexpr CudaSpace::size_type KernelArgumentLimit =
+     GridConstantLaunchEnabled ? 0x008000  /* 32k bytes */
+                                : 0x001000; /*  4k bytes */

Disentangling the diff, it becomes:

#ifdef KOKKOS_IMPL_CUDA_USE_GRID_CONSTANT
       0x008000; /* 32k bytes */
#else
      0x001000; /*  4k bytes */
#endif

->

static constexpr CudaSpace::size_type KernelArgumentLimit =
#ifdef KOKKOS_IMPL_CUDA_USE_GRID_CONSTANT
  static constexpr bool GridConstantLaunchEnabled = true;
#else
  static constexpr bool GridConstantLaunchEnabled   = false;
#endif
  static constexpr CudaSpace::size_type KernelArgumentLimit =
     GridConstantLaunchEnabled ? 0x008000  /* 32k bytes */
                                : 0x001000; /*  4k bytes */

Put differently, I created a static constexpr bool GridConstantLaunchEnabled higher up whose value is set to true/false depending on KOKKOS_IMPL_CUDA_USE_GRID_CONSTANT.

Creating GridConstantLaunchEnabled let me take #ifdef nightmares out of the dispatch logic, which is analogous to what I did with KOKKOS_ENABLE_IMPL_CUDA_CONSTANT_MEMORY. It's easy for me to roll this constexpr bool [...] = true/false; design decision back.

Hey @weinbe2 and @crtrott.

Seeing that this PR proposes adding a test that verifies CUDA API calls, I wanted to bring ReProspect to your attention.

ReProspect is a tool that @romintomasetti and I developed as part of Romin's PhD research, and that we recently published in JOSS. It's a Python package that allows the use of NVIDIA's tools for the analysis of CUDA programs to be scripted end-to-end in a concise Python script, up to programmatic test assertions. It covers CUDA API tracing through Nsight Systems, kernel profiling through Nsight Compute, and binary analysis through the CUDA binary utilities. We have used ReProspect to gain insight into various Kokkos PRs that we have worked on.

In ReProspect's documentation, we have a getting started with CUDA API tracing example that is quite similar in spirit to the test in this PR. In our example, a CUDA program launches 4 kernels with a diamond dependency pattern, and ReProspect is used to verify that the CUDA API calls are as expected. We also have an example that uses ReProspect to trace the CUDA API calls that occur when Kokkos allocates a View.

Looking at the test proposed in this PR, using ReProspect would lead to the test being structured as a pair of a Kokkos source file and a Python analysis script. The Kokkos source file would launch the kernels with the functors of the various sizes. The companion Python script would launch Nsight Systems on the executable, read the collected trace, and submit it to test assertions. The setup would likely involve running the test in a Docker container that provides Nsight Systems, Python, and ReProspect (as easy as pip install reprospect :)).

I am not asking for changes on this PR. But a test that verifies CUDA API behavior is a test of a new type in Kokkos. If interest in tests of this type grows, it may be worth comparing approaches at some point, e.g., comparing using CUPTI as done here with using ReProspect. Happy to discuss!