Skip to content

Split the omp model into omp-for and omp-target models to avoid making extra copies of buffers - #43

Closed
nicolas-bouton wants to merge 3 commits into
UoB-HPC:mainfrom
SiPearl:nb/split-omp-model-into-2-models
Closed

nicolas-bouton wants to merge 3 commits into
UoB-HPC:mainfrom
SiPearl:nb/split-omp-model-into-2-models

Conversation

@nicolas-bouton

Copy link
Copy Markdown

This MR turns the omp model into 2 models, omp-for and omp-target, avoids making extra copies of buffers for the omp-for model based on the std-range model, whereas the omp-target model is still making extra copies of buffers for the device.

The code from the omp model was used as the code for the omp-target model, since the kernel for this model was not modified, so it is clear in the git diff that the code for this model is not touched.

The Kokkos cmake file was also modified to let the user use an already compiled Kokkos library.

Remove occurences of `#pragma omp parallel for`. And doesn't change the behavior
of the cmake, same variables need to be passed to enable offloading.

* CMakeLists.txt,
src/main.cpp,
src/omp-target/fasten.hpp,
src/omp-target/model.cmake: Here.

Signed-off-by: Nicolas Bouton <nicolas.bouton@sipearl.com>
Provide a revised version of the `#pragma omp parallel for` implementation that
avoids making extra copies of buffers (like the std-ranges model), reducing
memory overhead and improving performance.

* CMakeLists.txt,
src/main.cpp,
src/omp-for/fasten.hpp,
src/omp-for/model.cmake: Here.

Signed-off-by: Nicolas Bouton <nicolas.bouton@sipearl.com>
* src/kokkos/model.cmake: Here.

Signed-off-by: Nicolas Bouton <nicolas.bouton@sipearl.com>
@tom91136

tom91136 commented Oct 4, 2026

Copy link
Copy Markdown
Member

Thanks for the PR. The copies are intentional for first-touch NUMA placement and occur outside the timed kernel (that said, the data movement is really small relative to the computation). Splitting the model would also duplicate the kernel, so I’d prefer to keep the current implementation.

I've address the Kokkos issue separately in 40254ff

@tom91136 tom91136 closed this Oct 4, 2026
@nicolas-bouton

nicolas-bouton commented Oct 5, 2026 •

Copy link
Copy Markdown
Author

[EDITED]

Hello @tom91136,

Thank you for reviewing the PR and for addressing the Kokkos issue separately in 40254ff.

I would appreciate it if you could reconsider this PR for the following reasons.

I understand that the copies in the current omp implementation are intended to support first-touch NUMA placement.

The additional copies discussed in this PR are at line 219 which are in fact not deep copies. And the intentional copies are effectively outside of the timed kernel as you pointed out.

However, on AArch64-based CPUs, my PR resulted in performance improvements of approximately 10%–20%. More broadly, the PR attempts to align the implementations with the kokkos and std-range models, making it easier to implement and maintain NUMA-aware behavior consistently across all three models.

Regarding kernel duplication, I understand the concern. The goal is not only to separate the omp-for and omp-target implementations, but also to make their coding styles consistent so that future fixes can be applied more easily. Several other mini-applications in the UoB-HPC group already separate these models. Since I am not yet familiar enough with the omp-target implementation, I have not attempted to modify that kernel in this PR.

Finally, I noticed that the Kokkos-related changes in 40254ff appear to overlap with the work included in this PR. Since these changes originated from this contribution, could you please credit me in the commit message with the Co-authored-by trailer?

Would you be willing to reconsider the PR, or suggest specific changes that would address these concerns?

@tom91136

Copy link
Copy Markdown
Member

Hi, so the Kokkos change (40254ff) does not in fact originate from your contribution, it was a direct copy of https://github.com/UoB-HPC/CloverLeaf/blob/9e67e5bcf264a259e7a873173da7f413fb44a146/kokkos/model.cmake that I made in 2023, you can check the diff yourself.

Can you elaborate on where the 10~20% came from in your change, and also which compiler on which hardware and at what PPWI do you observe this uplift?

@nicolas-bouton

Copy link
Copy Markdown
Author

Understood.

miniBUDE has been compiled and executed on aarch64 based CPUs (namely ARM Neoverse N1, ARM Neoverse V1, ARM Neoverse V2) with their appropriate architecture flags with -mcpu=neoverse-{n1,v1,v2} on GCC 15.

The testcase used is the following: --deck "../data/bm1" --ppwi 128 --iter 200 --rows 16 --poses 0.

Performance divergence occurs at high number of threads (between 40 to max cores).

Unfortunatly, after retesting it, the performance gain is about 5% which is lower than the performance gain claimed in my PR message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants