Closes #242, closes #735.

With the recent stream-per-task execution model, we can start executing a whole bunch of operations asynchronously (because synchronization is cheaper now that it can yield back to the Julia scheduler). There was one exception though: memory copies. Basically, when transferring memory to or from the CPU, that memory should not be paged out. You can avoid that from happening by pinning or page-locking memory. If you don't, the operation will still execute synchronously, and worse, it will do so without yielding back to the Julia scheduler.

Demo for the rest of this PR:

using CUDA, LinearAlgebra

# an expensive computation
function compute(a,b,c)
    mul!(c, a, b)         # library call
    broadcast!(sin, c, c) # Julia kernel
    c
end

# one "iteration", computing in two tasks
# and comparing the output.
function iteration(a,b,c)
    results = Vector{Any}(undef, 2)
    NVTX.@range "computation" @sync begin
        @async begin
            results[1] = Array(compute(a,b,c))
        end
        @async begin
            results[2] = Array(compute(a,b,c))
        end
    end
    NVTX.@range "comparison" results[1] == results[2]
end

function main(N=1024)
    a = CUDA.rand(N,N)
    b = CUDA.rand(N,N)
    c = CUDA.rand(N,N)

    # make sure this data can be used by other tasks!
    synchronize()

    # warm-up
    iteration(a,b,c)
    GC.gc(true)

    CUDA.@profile begin
        NVTX.@range "run" iteration(a,b,c)
    end
end

isinteractive() || main()

Similar example to #662, but with one crucial difference: a memory copy is performed within the computational task (Array(compute(a,b,c))). That's bad, and results in the following profile:

The copies behave synchronously, which results in the first task blocking outside of the scheduler so that the second task can't get scheduled. One solution, as sneakily used in #662, is to synchronize() in each task such that we can start work on the other task before we perform a memory operation. That results in the following trace:

Much better. The void before each memory copy is because of allocating memory. A better solution is to pre-allocate that memory and pin it (so results[...] = Mem.pin(Array{eltype(c)}(undef, size(c))) in the main task, and a copyto! on the results instead of just calling the Array constructor), not only avoiding that 'blank space' but also making sure the memory operations can happen asynchronously. That gives a best-case profile trace:

Note that the copies themselves are also a lot faster. So you might think, why not eagerly pin every CPU buffer before a memory operation? Alas, pinning isn't free:

So, that brings me to the final design of this PR, where we try and figure out whether a CPU buffer is pinned before a memory operation, and if it isn't, eagerly synchronize execution. By doing so, we can at least finish the current work on the GPU concurrently to work queued up by other Julia tasks, even though we'll still end up synchronizing in the driver afterwards:

The overhead of these operations are minimal, couple of 10s of ns is barely visible on the profiler trace.

cc @staticfloat