StopIteration in a generator in a thread hanging on asyncio

Bug report

Bug description:

import asyncio

def a():
    raise StopIteration

try:
    await asyncio.to_thread(a)
except:
    print(3.12.0)

other cases: agronholm/anyio#477

CPython versions tested on:

3.11, 3.12

Operating systems tested on:

Windows

Linked PRs

Please post working code.
print(3.12.0) is a SyntaxError.
With that fixed, 'await' outside an async function is also a syntax error.
With that fixed I still get an error.

import asyncio

def a():
    raise StopIteration

async def f():
    try:
        await ayncio.to_thread(a)
    except:
        print('done')

f()

results in RuntimeWarning: coroutine 'f' was never awaited.
(Afaik, without yield, a is not a generator function.)

That is the working code in notebook, except the 3.12.0 was trying to match the version number of the used python.
Using StopIteration instead of yield is for simplifying the code, and also report that this exception can also cause the issue.

Anyway, here are the working codes without notebook:

you will see it never exits.

import asyncio


def a():
    raise StopIteration


async def main():
    try:
        await asyncio.to_thread(a)
    except:
        print(3.12)

asyncio.run(main())

the yeild one

import asyncio


def a():
    for _ in [1, 2]:
        yield _


async def main():
    gen = a()
    while True:
        try:
            await asyncio.to_thread(next, gen)
        except:
            print(3.12)
            break

asyncio.run(main())

Also, in the related link, it has more example cases

I just noticed that my code had a spelling error masked by the overly broad except.
With the spelling and the call to f (or main), the following (in IDLE with Win 10)

Exception in callback _chain_future.<locals>._set_state(<Future pendi...ask_wakeup()]>, <Future at 0x...StopIteration>) at C:\Programs\Python313\Lib\asyncio\futures.py:381
handle: <Handle _chain_future.<locals>._set_state(<Future pendi...ask_wakeup()]>, <Future at 0x...StopIteration>) at C:\Programs\Python313\Lib\asyncio\futures.py:381>
Traceback (most recent call last):
  File "C:\Programs\Python313\Lib\asyncio\events.py", line 84, in _run
    self._context.run(self._callback, *self._args)
  File "C:\Programs\Python313\Lib\asyncio\futures.py", line 383, in _set_state
    _copy_future_state(other, future)
  File "C:\Programs\Python313\Lib\asyncio\futures.py", line 359, in _copy_future_state
    dest.set_exception(_convert_future_exc(exception))
TypeError: StopIteration interacts badly with generators and cannot be raised into a Future

and does indeed 'hang' (no >>> prompt appears). Hit ^C and

Traceback (most recent call last):
  File "C:\Programs\Python313\Lib\asyncio\runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "C:\Programs\Python313\Lib\asyncio\base_events.py", line 664, in run_until_complete
    return future.result()
  File "F:\dev\tem\tem2.py", line 8, in main
    await asyncio.to_thread(a)
  File "C:\Programs\Python313\Lib\asyncio\threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\dev\tem\tem2.py", line 12, in <module>
    asyncio.run(main())
  File "C:\Programs\Python313\Lib\asyncio\runners.py", line 194, in run
    return runner.run(main)
  File "C:\Programs\Python313\Lib\asyncio\runners.py", line 123, in run
    raise KeyboardInterrupt()
KeyboardInterrupt

and a prompt appears. I don't know enough about asyncio to say more, but I add the label to ping people who do.

My first response is "don't do that". That's why it prints such an elaborate warning.

Looking into the machinery involved, maybe there could be a try/except around the else block in _copy_future_state() (in asyncio/futures.py) that if the whole thing fails just calls dest.cancel(). That way the caller at least won't hang.

The failure should also be logged. Perhaps you can find the loop where to log it via dest._loop. Make sure that a failure to log doesn't crash.

So the PR should look like:

from .log import logger
def _copy_future_state(source, dest):
    """Internal helper to copy state from another Future.

    The other Future may be a concurrent.futures.Future.
    """
    assert source.done()
    if dest.cancelled():
        return
    assert not dest.done()
    if source.cancelled():
        dest.cancel()
    else:
        try:
            exception = source.exception()
            if exception is not None:
                dest.set_exception(_convert_future_exc(exception))
            else:
                result = source.result()
                dest.set_result(result)
        except:
            dest.cancel()
            # dest._loop.
            logger.error('Exception in default exception handler',
                        exc_info=True)

?

it seems that it's dest.set_exception(_convert_future_exc(exception)) raised error, and which is at

    def set_exception(self, exception):
        """Mark the future done and set an exception.

        If the future is already done when this method is called, raises
        InvalidStateError.
        """
        if self._state != _PENDING:
            raise exceptions.InvalidStateError(f'{self._state}: {self!r}')
        if isinstance(exception, type):
            exception = exception()
        if type(exception) is StopIteration:
            raise TypeError("StopIteration interacts badly with generators "
                            "and cannot be raised into a Future")
        self._exception = exception
        self._exception_tb = exception.__traceback__
        self._state = _FINISHED
        self.__schedule_callbacks()
        self.__log_traceback = True

why would set_exception() raise the error because of 'stopiteration' in python?

and what about changing the exception to runtimeerror like how the anyio does in its issue at the above first comment

This surprised me too. Maybe the git history of those lines has a clue.

It seems that this line exists after 3.5.2rc1 (not in 3..5.1),
and its commit is 1bd0307 and it referred to an invalid issue

That is a bpo issue: https://bugs.python.org/issue26221

FYI, I experimented with simply taking the checks for StopIteration out of the code. When using the pure Python version, this produces, surprise, surprise, a RuntimeError from the PEP 479 infrastructure. When using the C accelerator, this produces, even more surprising, a valid return value equal to the argument to StopIteration.

In futures.py, no code to replace StopIteration with RuntimeError exists, where does it produce a RuntimeError?

The simplest approach is probably what we were contemplating here: instead of raising TypeError, replace the exception with a RuntimeError (whose __cause__ is set to the original StopIteration instance).

So replacing StopIteration with RuntimeError in both .py and .c is the way to fix? What does the msg should be? Does PyErr_SetString() need to set __cause__?

Another approach would be to do the replacement on the receiving end: this would move the check for StopIteration to the result() method (raising RuntimeError there). That way a caller in the know could reconstruct the original StopIteration using fut.exception().

Is this approach to set the dest done in _copy_future_state() and had a RuntimeError exception?

In futures.py, no code to replace StopIteration with RuntimeError exists, where does it produce a RuntimeError?

That's the power of PEP 479. When a generator or async function raises StopIteration and the exception is about to bubble out of there, it is replaced by RuntimeError. The C code that does this lives here in intrinsics.c. (In older Python versions it was in genobject.c, but it's basically the same code.)

So replacing StopIteration with RuntimeError in both .py and .c is the way to fix? What does the msg should be? Does PyErr_SetString() need to set __cause__?

If we're replacing it in set_exception() I'd use the message that's used there currently for the TypeError. If we're doing it on the receiving end (in result()) I'd use the message from intrinsics.c above.

Is this approach to set the dest done in _copy_future_state() and had a RuntimeError exception?

I think you don't have to do anything in that function, regardless whether you replace it at the sending or the receiving end.

I think for the C code you can probably crib the code from intrinsics.c. For the Python code it's just err = RuntimeError(message); err.__cause__ = original_exception.

@NewUserHa

Hi! Saw that this issue seemed a little stale so thought I take a jab at it for my own learning, hope that's OK.

@NewUserHa let me know if you have something already in progress and I can close my PR.

@ordinary-jamie

@ordinary-jamie You should ask before you open your PR.

@NewUserHa

Ah sorry @NewUserHa; did you want me to close my PR then?

gh-112182: Replace StopIteration with RuntimeError for future (#113220)

pythongh-112182: Replace StopIteration with RuntimeError for future (p…

[3.12] gh-112182: Replace StopIteration with RuntimeError for future (G…

pythongh-112182: Replace StopIteration with RuntimeError for future (p…

Navigation Menu

Repository navigation

`StopIteration` in a generator in a thread hanging on asyncio #112182

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

9 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

StopIteration in a generator in a thread hanging on asyncio #112182

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

terryjreedy commented on Nov 17, 2023

NewUserHa commented on Nov 17, 2023

terryjreedy commented on Nov 17, 2023

gvanrossum commented on Nov 17, 2023

gvanrossum commented on Nov 17, 2023

NewUserHa commented on Nov 17, 2023

gvanrossum commented on Nov 17, 2023

NewUserHa commented on Nov 17, 2023

gvanrossum commented on Nov 18, 2023

9 remaining items

NewUserHa commented on Nov 19, 2023

gvanrossum commented on Nov 19, 2023

ordinary-jamie commented on Dec 16, 2023

NewUserHa commented on Dec 17, 2023

ordinary-jamie commented on Dec 17, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`StopIteration` in a generator in a thread hanging on asyncio #112182