The current implementation isn't configurable, and defaults to these semantics. Thoughts (especially @vchuravy)? With KA using explicit streams and not relying on the default stream to synchronize the world (note that device synchronization still works like that) I don't think there should be any issues.

Codecov Report

Merging #395 into master will decrease coverage by 0.12%.
The diff coverage is 15.38%.

@@            Coverage Diff             @@
##           master     #395      +/-   ##
==========================================
- Coverage   78.99%   78.86%   -0.13%     
==========================================
  Files         165      165              
  Lines        8731     8749      +18     
==========================================
+ Hits         6897     6900       +3     
- Misses       1834     1849      +15     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 892e649...fbedaaa. Read the comment docs.

I think from a KA perspective this fine, but we do currently synchronize against the CuDefaultStream() to synchronize against broadcast code.

target = CUDADevice()
kernel = mykernel!(target)

event = Event(target) # sync from above
event = kernel!(..., dependencies=event)

wait(target, event) # sync from below

where Event(CUDADevice) creates a event on the CuDefaultStream to and wait(CUDADevice, event) places asynchronization edge on the CuDefaultStream. So we would need a different way of doing that synchronization, probably we would synchronize with the per thread stream.

How do normal memory copies behave? Do they still act as a global sync?

So we would need a different way of doing that synchronization, probably we would synchronize with the per thread stream.

I don't think so, as cuStreamWaitEvent has a PTDS variant, so calling it on the default stream will synchronize the per-thread one.

How do normal memory copies behave? Do they still act as a global sync?

No, they only sync the thread-local default stream (which is a major part of the benefit here, since we can load data on a thread/task without disturbing others).

@jonathan-laurent This is probably also an interesting change for AlphaZero.jl (as a heavy user of GPU multi-threading/tasking), maybe you could give it a go to see if it improves performance and/or otherwise affects the programming model?

@ali-ramadhan and @glwagner, if you have time it would be good to try out Oceanigans on this, since last time we messed up synchronization the tests showed that relatively quickly.

This is probably also an interesting change for AlphaZero.jl (as a heavy user of GPU multi-threading/tasking), maybe you could give it a go to see if it improves performance and/or otherwise affects the programming model?

@maleadt This looks nice.
To be clear, though, only a single thread is using the GPU at any given time in AlphaZero.jl. Indeed, there is an inference server that runs on a separate task and that receives requests from other tasks. Is there still a chance that it can benefit from this PR?

Is there still a chance that it can benefit from this PR?

No, this only has an impact when multiple threads perform GPU computations.