I was seeing a bunch of sporadic INVALID_VALUE errors during plain memory copies, both here on CI and locally for a while now, when I read:

Memory regions requested must be either entirely registered with CUDA, or in the case of host pageable transfers, not registered at all. Memory regions spanning over allocations that are both registered and not registered with CUDA are not supported and will return CUDA_ERROR_INVALID_VALUE.

So I'm trying a revert of #233 here, since only @kshyatt is using that functionality and it was broken for her anyhow.

Codecov Report

Merging #258 into master will decrease coverage by 0.10%.
The diff coverage is 39.13%.

@@            Coverage Diff             @@
##           master     #258      +/-   ##
==========================================
- Coverage   80.53%   80.42%   -0.11%     
==========================================
  Files         154      154              
  Lines       10114    10231     +117     
==========================================
+ Hits         8145     8228      +83     
- Misses       1969     2003      +34     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a58ddf7...8553584. Read the comment docs.

Urgh, so we can't do set's but have to register the whole object? We are using the pinning with KA because pinning the host memory is necessary for asynchronous memory transfers

I'm not sure what the exact limitations are, but it seems like this does cause issues so let's revert for now.