Repository navigation
Lock CUDNN descriptor cache lookups. - #1430
Conversation
Codecov Report
@@ Coverage Diff @@
## master #1430 +/- ##
==========================================
- Coverage 77.32% 76.62% -0.71%
==========================================
Files 121 121
Lines 9186 9188 +2
==========================================
- Hits 7103 7040 -63
- Misses 2083 2148 +65
Continue to review full report at Codecov.
|
|
Thank you! I'm going to try with the training now. I'm assuming the locking can mean multiple threads accessing the dictionaries now have to wait to get the lock? Do you think there would be a significant performance hit with that? |
Correct. I don't expect this to be very costly; if it is, we can always introduce a task-local cache. |
|
Seems like with this branch I get nested task error: Unhandled Task ERROR: CUDNNError: CUDNN_STATUS_EXECUTION_FAILED (code 8)
Stacktrace:
[1] throw_api_error(res::CUDA.CUDNN.cudnnStatus_t)
@ CUDA.CUDNN ~/.julia/packages/CUDA/qzBIx/lib/cudnn/error.jl:22
[2] macro expansion
@ ~/.julia/packages/CUDA/qzBIx/lib/cudnn/error.jl:35 [inlined]
[3] cudnnActivationForward(handle::Ptr{Nothing}, activationDesc::CUDA.CUDNN.cudnnActivationDescriptor, alpha::Base.RefValue{Floa
t32}, xDesc::CUDA.CUDNN.cudnnTensorDescriptor, x::CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, beta::Base.RefValue{Float32}, yDesc:
:CUDA.CUDNN.cudnnTensorDescriptor, y::CuArray{Float32, 4, CUDA.Mem.DeviceBuffer})
@ CUDA.CUDNN ~/.julia/packages/CUDA/qzBIx/lib/utils/call.jl:26
[4] #cudnnActivationForwardAD#1215
@ ~/.julia/packages/CUDA/qzBIx/lib/cudnn/activation.jl:48 [inlined]
[5] #cudnnActivationForwardWithDefaults#1214
@ ~/.julia/packages/CUDA/qzBIx/lib/cudnn/activation.jl:42 [inlined]
[6] #cudnnActivationForward!#1211
@ ~/.julia/packages/CUDA/qzBIx/lib/cudnn/activation.jl:22 [inlined]
[7] #67
@ ~/.julia/packages/NNlibCUDA/IeeBk/src/cudnn/activations.jl:13 [inlined]
[8] materialize(bc::Base.Broadcast.Broadcasted{CUDA.CuArrayStyle{4}, Nothing, typeof(relu), Tuple{CuArray{Float32, 4, CUDA.Mem.D
eviceBuffer}}})
@ NNlibCUDA ~/.julia/packages/NNlibCUDA/IeeBk/src/cudnn/activations.jl:30with ResNet-50. I haven't dug very deep into this yet. |
Fixes #1421 (comment).
Ideally the CUDNN wrappers should integrate with the HandleCache machinery we use for the other libraries, but this is a quick fix to get things working.
cc @DhairyaLGandhi @ToucheSir