Repository navigation
asyncio.TaskGroup does not close unawaited coroutines #115957
Description
Bug report
Bug description:
Consider the following code:
import asyncio
async def wait_and_raise():
await asyncio.sleep(0.5)
raise RuntimeError(1)
async def wait_and_start(tg):
try:
await asyncio.sleep(1)
finally:
try:
tg.create_task(asyncio.sleep(1))
except RuntimeError as e:
print(f"wait_and_start() caught {e!r}")
async def main():
try:
async with asyncio.TaskGroup() as tg:
tg.create_task(wait_and_start(tg))
tg.create_task(wait_and_raise())
except Exception as e:
print(f"main() caught {e!r}")
try:
tg.create_task(asyncio.sleep(1))
except RuntimeError as e:
print(f"main() caught {e!r}")
asyncio.run(main())This gives the following output
wait_and_start() caught RuntimeError('TaskGroup <TaskGroup tasks=1 errors=1 cancelling> is shutting down')
C:\code\taskgrouptest.py:16: RuntimeWarning: coroutine 'sleep' was never awaited
print(f"wait_and_start() caught {e!r}")
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
main() caught ExceptionGroup('unhandled errors in a TaskGroup', [RuntimeError(1)])
main() caught RuntimeError('TaskGroup <TaskGroup cancelling> is finished')
C:\code\taskgrouptest.py:29: RuntimeWarning: coroutine 'sleep' was never awaited
print(f"main() caught {e!r}")
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Arguably, when you call tg.create_task() on a task group that is shutting down or has finished, the calling code "knows" about the error because it gets a RuntimeError exception (as you can see above), so there is no need to get a warning about a coroutine that was not awaited. So, when a TaskGroup encounters this situation, it should close the coroutine before raising the error.
The other argument would be that this still represents a design mistake so should still get the warning. I can see both points of view but I'm raising this issue so a conscious decision can be made.
For comparison, when you do this on a Trio Nursery or AnyIO TaskGroup that has already closed, a coroutine never even gets created in the first place, because you use a different syntax (nursery.start_soon(foo, 1, 2) rather than tg.create_task(foo(1, 2))), so it's a lot like if asyncio were to close the coroutine. The situation is a bit different for a nursery that is shutting down: then it runs till the first (unshielded) await and is cancelled at that point, which is possible because they use level-based cancellation rather than edge-based cancellation.
CPython versions tested on:
3.12
Operating systems tested on:
Windows
Activity
arthur-tacca commentedon Feb 26, 2024
I'm told
loop.create_task()has the same problem if the loop is closed.gvanrossum commentedon Feb 27, 2024
I see. Reproduced and understood. The issue is in this code at the top of
TaskGroup.create_task():cpython/Lib/asyncio/taskgroups.py
Lines 156 to 161 in 6087315
If we inserted
coro.close()before each of the threeraisestatements there we would no longer get the warning. When using the common pattern, the coroutine would be closed without executing a single line of code.If the
coroutineargument is something else, who knows, but thecreate_task()argument is expected to conform to the complete protocol of coroutines, so if someone is duck typing this, they should already be prepared for this.It's also the case that if someone squirrels away a reference to the coroutine object, they can tell that it hasn't been closed (it's tricky, you basically have to try to call its
send()orthrow()method), but that also feels like a very remote possibility where someone is already breaking the contract forcreate_task().So I think this is a reasonable fix, but I wouldn't want to backport it to 3.12 or before.
Would you be interested in submitting a PR? The fix itself is trivial, but it would be nice if there was a test for the new behavior (this can be adapted from your example, using
coro.send(None)to test whether the coroutine has already been closed). A "versionchanged" directive in the docs forTaskGroup.create_task()would also be appreciated, as well as a brief mention in "What's new in 3.13". Let me know if you need more pointers.Regarding
loop.create_task(), the logic there is a bit trickier, and the use case feels weaker, so I'd leave that alone for now.pythongh-115957: Close coroutine if the TaskGroup has been closed
Jason-Y-Z commentedon Feb 27, 2024
Sorry for jumping in, but this looks quite relevant to what I'm working on, so I had a go at implementing the simple change, in #116009, and modified the tests accordingly.
pythongh-115957: Close coroutine if the TaskGroup has been closed
pythongh-115957: Close coroutine if the TaskGroup has been closed
pythongh-115957: Close coroutine if the TaskGroup is inactive
pythongh-115957: Close coroutine if the TaskGroup is inactive
pythongh-115957: Close coroutine if the TaskGroup is inactive
pythongh-115957: Close coroutine if the TaskGroup is inactive
gh-115957: Close coroutine if TaskGroup.create_task() raises an error (…
pythongh-115957: Close coroutine if TaskGroup.create_task() raises an…
pythongh-115957: Close coroutine if TaskGroup.create_task() raises an…