[ROCm] Fix foreach profiler check and cholesky_solve nondeterminism #170964
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/170964
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 742a068 with merge base 58fac80 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
ROCm's rocTracer has kernel name detection issues similar to NVIDIA's CUPTI on CUDA 12.6/12.8. The profiler may fail to detect the multi_tensor_apply_kernel even when it runs correctly. This follows the same pattern as the existing CUDA 12.6/12.8 skip, allowing the fastpath to run without attempting to verify via profiler. References: - CUDA profiler issue: pytorch#148681 - ROCm profiler issues: pytorch#97167 Enables 5+ foreach tests on ROCm that were failing due to profiler detection, not actual fastpath problems.
ROCm's hipSOLVER exhibits nondeterministic behavior in backward passes for complex types, causing gradcheck to fail with 'backward is not reentrant' errors. This adds GRADCHECK_NONDET_TOL (1e-12) to allow small numerical differences in gradient reentrance checks, following the pattern used by other ops with similar nondeterminism (addmm, mm, etc). Fixes: pytorch#164193
5c51272 to
742a068
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes two ROCm test failures by applying tolerances and skips that match existing patterns for similar issues.
Changes
1. Skip profiler check in foreach tests on ROCm
The foreach tests verify that
multi_tensor_apply_kernelruns by checking profiler output. On ROCm, rocTracer sometimes fails to detect the kernel even when it executes correctly—the same issue NVIDIA has with CUPTI on CUDA 12.6/12.8.We already skip this profiler check for those CUDA versions (#148681), so this extends that skip to ROCm (#97167).
2. Add nondeterminism tolerance for cholesky_solve gradcheck
hipSOLVER's backward pass for complex types has slight numerical variation between runs, which causes gradcheck's reentrant test to fail. Adding
GRADCHECK_NONDET_TOL(1e-12) fixes this—the same approach used foraddmm,mm, and other ops with similar behavior.Fixes #164193
Tests now passing
test_foreach_copy_with_multi_dtypes(5 dtype combinations)test_fn_gradgrad_cholesky_solve_cuda_complex128