Asynchronous comprehensions don't work in asyncio REPL

@asvetlov

BPO	39562
Nosy	@asvetlov, @ambv, @serhiy-storchaka, @Carreau, @asottile, @hroncok, @pablogsal, @miss-islington, @isidentical, @jack1142
PRs	bpo-39562: Allow executing asynchronous comprehensions in the asyncio REPL #18968 bpo-39965: Correctly raise SyntaxError if await is used outside async functions when PyCF_ALLOW_TOP_LEVEL_AWAIT is set #19010 [3.8] bpo-39562: Allow executing asynchronous comprehensions in the asyncio REPL (GH-18968) #19070 bpo-39562: Prevent collision of future and compiler flags #19230 [3.8] bpo-39562: Prevent collision of future and compiler flags (GH-19230) #19835 [3.8] bpo-39562: Correctly updated the version section in the what's new document #19838

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-07-06.20:18:05.903>
created_at = <Date 2020-02-05.17:28:25.689>
labels = ['type-bug', '3.8', '3.9', 'release-blocker', 'expert-asyncio']
title = "Asynchronous comprehensions don't work in asyncio REPL"
updated_at = <Date 2021-09-07.20:14:54.080>
user = 'https://github.com/jack1142'

bugs.python.org fields:

activity = <Date 2021-09-07.20:14:54.080>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-07-06.20:18:05.903>
closer = 'pablogsal'
components = ['asyncio']
creation = <Date 2020-02-05.17:28:25.689>
creator = 'jack1142'
dependencies = []
files = []
hgrepos = []
issue_num = 39562
keywords = ['patch']
message_count = 28.0
messages = ['361443', '361445', '364600', '364601', '365311', '365312', '365313', '365347', '365881', '366838', '366841', '366843', '366849', '366851', '367015', '367018', '367734', '367847', '367855', '367890', '367892', '367897', '368334', '368835', '373131', '373159', '373168', '400234']
nosy_count = 10.0
nosy_names = ['asvetlov', 'lukasz.langa', 'serhiy.storchaka', 'mbussonn', 'Anthony Sottile', 'hroncok', 'pablogsal', 'miss-islington', 'BTaskaya', 'jack1142']
pr_nums = ['18968', '19010', '19070', '19230', '19835', '19838']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39562'
versions = ['Python 3.8', 'Python 3.9']

asyncio REPL doesn't allow using asynchronous comprehensions outside of async func. Same behavior can also be observed when using ast.PyCF_ALLOW_TOP_LEVEL_AWAIT flag in compile()

Example with `async for`:
>>> async def async_gen():
...     for x in range(5):
...         yield await asyncio.sleep(1, x)
... 
>>> [x async for x in async_gen()]
  File "<console>", line 0
SyntaxError: asynchronous comprehension outside of an asynchronous function


Example with `await`:
>>> [await asyncio.sleep(1, x) for x in range(5)]
  File "<console>", line 0
SyntaxError: asynchronous comprehension outside of an asynchronous function

I also noticed that putting await before the async comprehension will make the code inside the comprehension run (though after it runs, it will fail on awaiting list):

>>> await [await asyncio.sleep(1, print(x)) for x in range(5)] 
0
1
2
3
4
Traceback (most recent call last):
  File "C:\Python38\lib\concurrent\futures\_base.py", line 439, in result
    return self.__get_result()
  File "C:\Python38\lib\concurrent\futures\_base.py", line 388, in __get_result
    raise self._exception
  File "<console>", line 1, in <module>
TypeError: object list can't be used in 'await' expression

New changeset 9052f7a by Batuhan Taşkaya in branch 'master':
bpo-39562: Allow executing asynchronous comprehensions in the asyncio REPL (GH-18968)
9052f7a

New changeset ec8a973 by Miss Islington (bot) in branch '3.8':
bpo-39562: Allow executing asynchronous comprehensions in the asyncio REPL (GH-18968)
ec8a973

This change caused python-gmpy2 tests to fail:
https://bugzilla.redhat.com/show_bug.cgi?id=1817710

Reproducer:
---

from __future__ import print_function, division
import doctest

filename = "doctest.txt"
with open(filename, "w") as fp:
    print("""
Test

====

    >>> all(x == 1 for x in [1, 1, 1])
    True
    """.strip(), file=fp)

result = doctest.testfile(filename, globs=globals())
print("result:", result)

This make non-await list comprehension coroutine-code-objects as well:

https://bugs.python.org/issue41218

    import ast
    import inspect
    cell = '[x for x in l]'
    code = compile(cell, "<>", "exec", flags=getattr(ast,'PyCF_ALLOW_TOP_LEVEL_AWAIT', 0x0))

inspect.CO_COROUTINE & code.co_flags == inspect.CO_COROUTINE  # this is now TRUE.

This leads to weird things in Jupyter/IPython when we try to detect wether a block of code is, or os not async.

Pablo, I see you reopened and marked as blocker,

As the changes here has been released maybe you want to reclose and marked https://bugs.python.org/issue41218 as blocker it should also be fixed by
#21357

As the changes here has been released maybe you want to reclose and marked https://bugs.python.org/issue41218 as blocker it should also be fixed by
#21357

Thanks for the heads up, Matthias!

Yes, I think is the best thing to do. For some reason, I thought this was not still released :(

this maybe shouldn't have been backported to 3.8.x as the change in compiler flags appears to break pyc files in subtle ways: https://stackoverflow.com/q/68910329/812183

Navigation Menu

Repository navigation

Asynchronous comprehensions don't work in asyncio REPL #83743

27 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Asynchronous comprehensions don't work in asyncio REPL #83743

Description

Activity

jack1142 commented on Feb 5, 2020

jack1142 commented on Feb 5, 2020

pablogsal commented on Mar 19, 2020

miss-islington commented on Mar 19, 2020

vstinner commented on Mar 30, 2020

27 remaining items

Carreau commented on Jul 6, 2020

Carreau commented on Jul 6, 2020

pablogsal commented on Jul 6, 2020

asottile commented on Aug 24, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions