Repository navigation
concurrent.futures.ProcessPoolExecutor pool deadlocks when submitting many tasks #105829
Description
Bug report
Submitting many tasks to a concurrent.futures.ProcessPoolExecutor pool
deadlocks with all three start methods.
When running the same example with multiprocessing.pool.Pool we have NOT been
able to cause a deadlock.
Different set of parameters affect how likely it is to get a deadlock
- All start methods
spawn,fork, andforkserverexhibit the deadlock
(the examples below are withspawnmethod) - It's possible to get a deadlock with num_processes 1-24
- As long as NUM_TASKS is high, TASK_DATA and TASK_SIZE can be low/removed and
still cause a hang. (see example script) - Set
DO_PRINT = Falsefor higher probability of hanging.
Example stack trace excerpts in hanged scenarios
-
reading the queue:
- 1 thread stuck at:
read (libpthread-2.27.so) recv_bytes (multiprocessing/connection.py:221) get (multiprocessing/queues.py:103) - other threads stuck at:
do_futex_wait.constprop.1 (libpthread-2.27.so) _multiprocessing_SemLock_acquire_impl (semaphore.c:355) get (multiprocessing/queues.py:102)
- 1 thread stuck at:
-
writing the queue:
- 1 thread stuck at:
write (libpthread-2.27.so) send_bytes (multiprocessing/connection.py:205) put (multiprocessing/queues.py:377) - other threads stuck at:
do_futex_wait.constprop.1 (libpthread-2.27.so) _multiprocessing_SemLock_acquire_impl (semaphore.c:355) put (multiprocessing/queues.py:376)
- 1 thread stuck at:
Example script exhibiting deadlock behavior
#!/usr/bin/env python3
""" Example that hangs with concurrent.futures.ProcessPoolExecutor """
import multiprocessing
import concurrent.futures
# Tweaking parameters
NUM_TASKS = 500000
TASK_DATA = 1
TASK_SIZE = 1
DO_PRINT = True # Set to false for almost guaranteed hang
START_METHOD = "spawn" # Does not seem to matter
NUM_PROCESSES = 4 # multiprocessing.cpu_count()
def main():
print("Starting pool")
ctx = multiprocessing.get_context(START_METHOD)
with concurrent.futures.ProcessPoolExecutor(max_workers=NUM_PROCESSES,
mp_context=ctx) as pool:
future_results = submit_to_pool(pool)
print("Collecting results")
assert False # Never reached
collect_results(future_results)
def collect_results(future_results):
return [r.result() for r in future_results]
def submit_to_pool(pool):
future_results = []
for task_idx in range(NUM_TASKS):
if DO_PRINT and task_idx % 20000 == 0:
# Too much printing here makes the hang to go away!!!
print("\nsubmit", task_idx)
task_name = f"task{task_idx}" * TASK_DATA
future_results.append(pool.submit(task, task_idx, task_name))
return future_results
def task(task_idx, task_name):
""" Do some dummy work """
s = ""
for i in range(TASK_SIZE):
s += str(i)
if DO_PRINT:
# Too much printing here makes the hang to go away!!!
print(".", end="", flush=True)
if __name__ == "__main__":
main()Environment
- My environment:
- Ubuntu 18.04.6 LTS (bionic)
- Python 3.10.5
- My colleagues environment:
- Ubuntu 22.04.2 LTS (jammy)
- Either:
- Python 3.10.5
- Python 3.11.0rc1
Details
Detailed stack traces in comments.
Linked PRs
- gh-105829: Fix concurrent.futures.ProcessPoolExecutor deadlock #108513
- [3.11] gh-105829: Fix concurrent.futures.ProcessPoolExecutor deadlock (GH-108513) #109783
- [3.12] gh-105829: Fix concurrent.futures.ProcessPoolExecutor deadlock (GH-108513) #109784
- gh-109917, gh-105829: Fix concurrent.futures _ThreadWakeup.wakeup() #110129
Metadata
Metadata
Assignees
Projects
- Status
Activity
kaddkaka commentedon Jun 15, 2023
Scenario 1
Running with 4 worker processes (this particular run hanged after submitting 380'000 tasks.
Scenario 2
Running with 4 worker processes (this particular run hanged after submitting 140'000 tasks.
Task modified to take no arguments:
Sending one
ctrl-clet's it continue until the end but stop with this Traceback:Scenario 3
Same as scenario 2 but other traceback.
Sending ctrl-c after the hang stops immediately with this Traceback:
elfstrom commentedon Jun 30, 2023
I think I have found the issue that is causing the deadlock. It seems
like the use of the shutdown_lock in the thread wake-up code is
responsible.
The shutdown_lock must be taken for both reads and writes of the
wake-up pipe. If a read or a write of the pipe blocks, the code will
deadlock. It looks like reads can't block (a poll() is done before
doing any reads) but writes have not protection against blocking.
What seems to happen in this case is that we send so many wake-up
messages that the pipe gets full and the write eventually blocks.
To confirm this I tried the following patch:
With this change I can no longer reproduce the deadlock. It is a bit
unclear to me if this change is actually valid. This could maybe lead
to new race conditions since we wont send a wake-up when the pipe is
full. I don't think that is the case however, since the reason for the
problem is that both reader and writer holds the
shutdown_lock. Everything is mutually exclusive here. That should
imply that we don't need to send anything if the pipe is full, the
reader can't be reading it concurrently, it will eventually wake up
and empty the pipe once we release the lock from the writer. This
might event be true without relying on the lock but that is much less
clear and looks scary to me.
I'm not sure if relying on the shutdown_lock for making the reader and
writer work correctly is a good solution? Another option would be to
split the shutdown lock into two separate locks, one for reading and
one for writing. Actual shutdown would need to hold both locks.
Something like this:
Again, not entirely clear if this is valid and preserves all required
invariants regarding the shutdown_lock. Was the mutual exclusivity
between reads and writes perhaps required? Since the lock is named
shutdown_lock and not wakeup_lock I would assume not but perhaps that
is wrong?
Removing the exclusivity could maybe also bring some minor performance
increase due to more concurrent execution but perhaps the GIL gets in
the way of that anyway?
Regardless, I think the problem is fairly clear but the solution needs
some careful thinking, preferably by someone who is very familiar
with this code (I have never seen it before looking into this issue).
kaddkaka commentedon Jul 12, 2023
Hi @pitrou, @brianquinlan and @gpshead, have you seen this?
gpshead commentedon Jul 13, 2023
Nice analysis and proposals @elfstrom. Indeed, it needs some thought to reason out.
I kind of like the non-blocking pipe write hack and agree that a full pipe shouldn't be a problem as presumably that implies that the reader is set to discover that and process things anyways, accomplishing the same goal.
But I'll need to dive into the code on this to understand the intents regardless.
On the multiple locks front, also plausible, though we should be strict about the acquisition ordering that must be used, otherwise it's more potential for deadlock. In addition to more comments describing the requirement, this could probably even be helped by abstracting them for normal use cases with some defensive code such that acquisition of the second lock via the abstraction will fail when the first one is already held by the same thread.
Another challenge with this kind of change is to come up with some form of easily executable likely deterministic regression test. i.e. something not too invasive that'd set up the otherwise infrequent state causing the existing code to fail that would no longer fail with the fix in place.
cjw296 commentedon Aug 18, 2023
Looks like both black and pylint are now tripping this bug in the right conditions, if I can find some time I'm going to have a look at the above...
cjw296 commentedon Aug 19, 2023
Looks like @vstinner introduced the usage of shutdown lock in this context, so tagging him here in case he has an opinion.
sharvil commentedon Aug 25, 2023
We consistently run into this issue in our large-scale data processing pipeline (Python 3.9, 3.10, and 3.11) and can never run it to completion. Thanks @elfstrom for the analysis and suggested fixes.
While I'm here, I should mention that
_ThreadWakeup.clearisn't guaranteed to clear the pipe.poll()can returnFalseon Linux when it receives a signal (man 2 poll). I understand it's a nonblocking poll, but AFAICT there's nothing in the man page that states that it can't error withEINTRwhen the timeout is zero.pythongh-105829: Add test to demonstrate deadlock
pythongh-105829: Fix concurrent.futures.ProcessPoolExecutor deadlock
elfstrom commentedon Aug 26, 2023
If I understand the code correctly then too many wakeups is fine so if clear sometimes leaves things in the pipe that should not be an issue. The thread will just wakeup, find nothing to do and go back to sleep again.
I also posted a proof of concept PR, hopefully it can get the ball rolling on a fix.
sharvil commentedon Aug 26, 2023
I agree that, given how the rest of the code is structured today, more wakeups is fine. A person poking at this code in the future may reasonably expect that
clearactually clears the pipe given the name of the function and a cursory look at the implementation.I'm not suggesting an invasive fix – but maybe a comment capturing the semantics / warning that
cleardoesn't always flush the pipe might help others looking at this down the line.Thanks for your work on this, and for pushing things along!
cjw296 commentedon Aug 30, 2023
@gpshead - do you know of any way to control the size of the pipe created? Feels like making the pipe small would make this more easily reproducible? But I can't find a way to do that...
24 remaining items
pythongh-105829: Fix concurrent.futures _ThreadWakeup.wakeup()
pythongh-109917, pythongh-105829: Fix concurrent.futures _ThreadWakeu…
pythongh-109917, pythongh-105829: Fix concurrent.futures _ThreadWakeu…
pythongh-109917, pythongh-105829: Fix concurrent.futures _ThreadWakeu…
[3.12] gh-105829: Fix concurrent.futures.ProcessPoolExecutor deadlock (…
[3.12] gh-105829: Fix concurrent.futures.ProcessPoolExecutor deadlock…
kaddkaka commentedon Oct 11, 2023
Which python versions will this fix be released in?
gpshead commentedon Oct 11, 2023
Yhg1s commentedon Oct 11, 2023
No, >=3.12.1.
[3.11] gh-105829: Fix concurrent.futures.ProcessPoolExecutor deadlock…
pythongh-105829: Fix concurrent.futures.ProcessPoolExecutor deadlock (p…
AlexTate commentedon Dec 11, 2025
@gpshead - sorry to resurrect this issue, but I'm wondering what options might exist for documenting this bug in Python versions which didn't receive the patch. For example, there are many projects running on 3.10 and 3.11 where this bug still exists but isn't documented (https://docs.python.org/3.11/library/concurrent.futures.html). The docs mention other ways to avoid deadlock, but nothing about this.
Sincerely,
A dev who arrived here after spending too much time investigating code which utilizes
ProcessPoolExecutorbecause at the time it seemed unlikely that a Python library was responsiblekaddkaka commentedon Dec 11, 2025
Other ways than documentation to make coders aware of the issue could be: