I've noticed some weird CI timeouts for a project I work on. The affected tests involve running a lot of eventlet-enabled subprocesses, and it seems that sometimes they hang during shutdown rather than exit properly.

I used this tool to get stacks for all greenlets and posix threads of one of these hung processes; I think the issue comes down to this posix-thread stack:

  File "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 310, in killall
    thr.join()
  File "/usr/lib64/python3.9/threading.py", line 1060, in join
    self._wait_for_tstate_lock()
  File "/usr/lib64/python3.9/threading.py", line 1080, in _wait_for_tstate_lock
    if lock.acquire(block, timeout):
  File "<string>", line 1, in <module>
  File "<string>", line 85, in <module>

I don't know exactly what's going on; I don't see anything else already holding the lock -- maybe there's something funny going on since we're in the midst of process shut-down?

FWIW, this is on CentOS Stream 9, Python 3.9.20, eventlet 0.37.0 -- I don't remember this being an issue back on CentOS Stream 8/Python 3.6.