[Caffe2] Missing CMAKE_CUDA_COMPILE_WHOLE_COMPILATION #18524
Comments
|
FYI I also got a similar error in microsoft/onnxruntime#663 |
|
Here's the interesting part. Switching off TRT will bypass this issue, and the GPU_ARCH message also gone. |
|
The new changes in |
|
Any update on this? |
For more details, refer to the discussion in pytorch/pytorch#18524
|
I'm seeing this issue as well, downgrading to 3.9.4 works due to this bit of code in third_party/onnx-tensorrt/CMakeLists.txt seems like things go bad when we go through the else leg and enable the CUDA language, as a test I changed CMAKE_VERSION_THRESHOLD to 3.15.0 and was able to compile fine with cmake version 3.14 so I agree, it doesn't seem to be a cmake issue, my guess is probably an issue with a mix of old style FindCUDA and the new style add_library/add_executable which supports cuda. WRT the message "GPU_ARCH is not defined. Generating CUDA code for default SMs.", it comes from a little further down in the file ( the message reads like an error instead of info, but it seems to do the right thing. |
For more details, refer to the discussion in pytorch/pytorch#18524
|
Also having this issue with CMake 3.14.5 |
|
confirm the issue with onnx-tensorrt with cmake 3.14.5 is solved with @rjknight solution.
Then there is no compile issue, no conflict with CUDA toolkit 9.x and 10.x and libtorch |
|
@rjknight @mikeseven I still have some CUB issue? /usr/local/cuda/include/cub/device/dispatch/dispatch_reduce.cuh(362): error: use the "typename" keyword to treat nontype "std::iterator_traits<_Iterator>::value_type [with _Iterator=InputIteratorT]" as a type in a dependent context
/usr/local/cuda/include/cub/device/dispatch/dispatch_reduce.cuh(363): error: use the "typename" keyword to treat nontype "std::iterator_traits<_Iterator>::value_type [with _Iterator=OutputIteratorT]" as a type in a dependent context
/usr/local/cuda/include/cub/device/dispatch/dispatch_reduce.cuh(683): error: use the "typename" keyword to treat nontype "std::iterator_traits<_Iterator>::value_type [with _Iterator=InputIteratorT]" as a type in a dependent context
/usr/local/cuda/include/cub/device/dispatch/dispatch_reduce.cuh(684): error: use the "typename" keyword to treat nontype "std::iterator_traits<_Iterator>::value_type [with _Iterator=OutputIteratorT]" as a type in a dependent context
/usr/local/cuda/include/cub/device/dispatch/dispatch_reduce.cuh(362): error: use the "typename" keyword to treat nontype "std::iterator_traits<_Iterator>::value_type [with _Iterator=InputIteratorT]" as a type in a dependent context
/usr/local/cuda/include/cub/device/dispatch/dispatch_reduce.cuh(363): error: use the "typename" keyword to treat nontype "std::iterator_traits<_Iterator>::value_type [with _Iterator=OutputIteratorT]" as a type in a dependent context
/usr/local/cuda/include/cub/device/dispatch/dispatch_reduce.cuh(683): error: use the "typename" keyword to treat nontype "std::iterator_traits<_Iterator>::value_type [with _Iterator=InputIteratorT]" as a type in a dependent context
/usr/local/cuda/include/cub/device/dispatch/dispatch_reduce.cuh(684): error: use the "typename" keyword to treat nontype "std::iterator_traits<_Iterator>::value_type [with _Iterator=OutputIteratorT]" as a type in a dependent context
4 errors detected in the compilation of "/tmp/tmpxft_00005fca_00000000-6_reduce.cpp1.ii".
-- Removing ....../pytorch/build/caffe2/CMakeFiles/torch.dir/utils/math/./torch_generated_reduce.cu.o
/usr/bin/cmake -E remove ....../pytorch/build/caffe2/CMakeFiles/torch.dir/utils/math/./torch_generated_reduce.cu.o
CMake Error at torch_generated_reduce.cu.o.Release.cmake:279 (message):
Error generating file
....../pytorch/build/caffe2/CMakeFiles/torch.dir/utils/math/./torch_generated_reduce.cu.o
make[2]: *** [caffe2/CMakeFiles/torch.dir/build.make:134500: caffe2/CMakeFiles/torch.dir/utils/math/torch_generated_reduce.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
4 errors detected in the compilation of "/tmp/tmpxft_00005fd2_00000000-6_math_gpu.cpp1.ii".
-- Removing ....../pytorch/build/caffe2/CMakeFiles/torch.dir/utils/./torch_generated_math_gpu.cu.o
/usr/bin/cmake -E remove ....../pytorch/build/caffe2/CMakeFiles/torch.dir/utils/./torch_generated_math_gpu.cu.o
CMake Error at torch_generated_math_gpu.cu.o.Release.cmake:279 (message):
Error generating file
....../pytorch/build/caffe2/CMakeFiles/torch.dir/utils/./torch_generated_math_gpu.cu.o
make[2]: *** [caffe2/CMakeFiles/torch.dir/build.make:136190: caffe2/CMakeFiles/torch.dir/utils/torch_generated_math_gpu.cu.o] Error 1
make[2]: Leaving directory '....../pytorch/build'
make[1]: *** [CMakeFiles/Makefile2:3175: caffe2/CMakeFiles/torch.dir/all] Error 2
make[1]: Leaving directory '....../pytorch/build'
make: *** [Makefile:166: all] Error 2
➜ build git:(master) ✗ cd ..My environments are (with a previously installed pytorch): ➜ utils git:(master) ✗ python collect_env.py
Collecting environment information...
PyTorch version: 1.2.0a0+d51bd21
Is debug build: No
CUDA used to build PyTorch: 10.1.168
OS: Ubuntu 19.04
GCC version: (Ubuntu 8.3.0-6ubuntu1) 8.3.0
CMake version: version 3.13.4
Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.1.168
GPU models and configuration: GPU 0: GeForce GTX 980M
Nvidia driver version: 430.40
cuDNN version: /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7.6.2
Versions of relevant libraries:
[pip3] numpy==1.16.4
[pip3] torch==1.2.0a0+d51bd21
[pip3] torchfile==0.1.0
[pip3] torchfusion==0.3.6
[pip3] torchfusion-utils==0.1.5
[pip3] torchtext==0.3.1
[pip3] torchvision==0.3.0a0+8a64dbc
[conda] Could not collectCan you help please? |
|
@jiapei100 This is not related to this issue so please open a new one for help. But my first suggestion is to try gcc-7 instead of 8. |
Summary: This PR makes Caffe2 compatible with TensorRT 6. To make sure it works well, new unit test is added. This test checks PyTorch->ONNX->TRT6 inference flow for all classification models from TorhchVision Zoo. Note on CMake changes: it has to be done in order to import onnx-tensorrt project. See #18524 for details. Pull Request resolved: #26426 Reviewed By: hl475 Differential Revision: D17495965 Pulled By: houseroad fbshipit-source-id: 3e8dbe8943f5a28a51368fd5686c8d6e86e7f693
Summary: This PR makes Caffe2 compatible with TensorRT 6. To make sure it works well, new unit test is added. This test checks PyTorch->ONNX->TRT6 inference flow for all classification models from TorhchVision Zoo. Note on CMake changes: it has to be done in order to import onnx-tensorrt project. See pytorch/pytorch#18524 for details. Pull Request resolved: pytorch/pytorch#26426 Reviewed By: hl475 Differential Revision: D17495965 Pulled By: houseroad fbshipit-source-id: 3e8dbe8943f5a28a51368fd5686c8d6e86e7f693
Summary: This PR makes Caffe2 compatible with TensorRT 6. To make sure it works well, new unit test is added. This test checks PyTorch->ONNX->TRT6 inference flow for all classification models from TorhchVision Zoo. Note on CMake changes: it has to be done in order to import onnx-tensorrt project. See pytorch#18524 for details. Pull Request resolved: pytorch#26426 Reviewed By: hl475 Differential Revision: D17495965 Pulled By: houseroad fbshipit-source-id: 3e8dbe8943f5a28a51368fd5686c8d6e86e7f693
|
Same issue occured with following environment.
Tried @rjknight workaround with higher version Luckily I've downgraded cmake version to Hope it helps. |
The above issue was a bug in the 3.23.0 release. Using 3.23.1 or newer will resolve the issue so you don't need to downgrade. FYI: any cmake that ends in |
Bug
This is probably a recent regression.
When building for CUDA, I got a lot of errors complaining about missing
CMAKE_CUDA_DEVICE_LINK_LIBRARYandCMAKE_CUDA_COMPILE_WHOLE_COMPILATION.Here's the build command and log when using cmake + make.
cmake + ninja will crash in a more violent way.
Environment
conda,pip, source): sourceAdditional context
The text was updated successfully, but these errors were encountered: