[bugfix] multiple remotes with single conan upload - #7781
Conversation
…d command would fail
| self._upload_thread_pool = ThreadPool(1) | ||
|
|
||
| for remote, refs in refs_by_remote.items(): | ||
| self._upload_thread_pool = ThreadPool(8 if parallel_upload else 1) |
There was a problem hiding this comment.
Maybe this should use tools.cpu_count()?
There was a problem hiding this comment.
This doesn't look right either, as the self._upload_thread_pool variable will be overwritten, losing access to the other pool from the previous remote.
There was a problem hiding this comment.
Each iteration of this loop will complete before the next starts, since map() will block, then it will close the thread pool and start the next iteration.
There was a problem hiding this comment.
Wait, we can definitely reuse the thread pool here. Keeping it alive the whole time makes a lot more sense.
memsharded
left a comment
There was a problem hiding this comment.
Thanks for submitting this PR.
I would like to understand better why it is failing in the first case, I cannot see an evident reason, so I'd need to try it a bit first.
| self._upload_thread_pool = ThreadPool(1) | ||
|
|
||
| for remote, refs in refs_by_remote.items(): | ||
| self._upload_thread_pool = ThreadPool(8 if parallel_upload else 1) |
There was a problem hiding this comment.
This doesn't look right either, as the self._upload_thread_pool variable will be overwritten, losing access to the other pool from the previous remote.
memsharded
left a comment
There was a problem hiding this comment.
Great! Now I understand the issue, it is totally clear.
Thanks very much for contributing this.
Changelog: Bugfix: Fixed bug where uploading to multiple remotes in a single conan upload command would fail.
Docs: omit
Fixes #7780
This bug was due to closing and reusing a thread pool in a loop. Instead, a separate thread pool will be created at each loop iteration.
developbranch, documenting this one.Note: By default this PR will skip the slower tests and will use a limited set of python versions. Check here how to increase the testing level by writing some tags in the current PR body text.