Repository navigation
Make Executor.map work with infinite/large inputs correctly #74028
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2017-03-18.01:32:56.383>
labels = ['3.8', 'library']
title = 'Make Executor.map work with infinite/large inputs correctly'
updated_at = <Date 2021-05-05.00:42:51.731>
user = 'https://github.com/MojoVampire'bugs.python.org fields:
activity = <Date 2021-05-05.00:42:51.731>
actor = 'leezu'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-03-18.01:32:56.383>
creator = 'josh.r'
dependencies = []
files = []
hgrepos = []
issue_num = 29842
keywords = ['patch']
message_count = 8.0
messages = ['289789', '289790', '293677', '293708', '322390', '322391', '341565', '385067']
nosy_count = 7.0
nosy_names = ['bquinlan', 'ezio.melotti', 'max', 'josh.r', 'dlukes', 'Klamann', 'leezu']
pr_nums = ['707', '18566']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue29842'
versions = ['Python 3.8']Linked PRs
- gh-74028: Introduce a prefetch parameter to Executor.map to handle large iterators #114975
- gh-74028:
concurrent.futures.Executor.map: introducebuffersizeparam for lazier behavior #125663 - gh-74028: update
whatsnew/3.14.rstpost gh-125663 #131214 - gh-74028:
concurrent.futures.Executor.map: avoid reference cycles when an exception is raised #131701
Activity
VannTen commentedon Aug 18, 2022
Chiming in as I recently used as_completed assuming it was consuming lazily. Reading it's code, it seems to me it could be adapted directly rather than on top of this (using islice and "re prefetching" the amount of yielded futures on each iteration). Does the author have an opinion ?
Jason-Y-Z commentedon Feb 18, 2024
Hey all, I had a go at resolving this based on the previous attempt in #114975. Please take a look if of interest @vstinner @VannTen
[IMP] snippets.convert_html_columns: a batch processing story
[IMP] snippets.convert_html_columns: a batch processing story
ebonnal commentedon Oct 17, 2024
Hi,
I also think a lot of codebases could leverage the elegance of a lazier
Executor.map.Here is a follow-up PR that introduces an optional
buffersizeparam and doesn't alter the default behavior: #125663.I would greatly appreciate it if you could take a look 🙏🏻.
Note: It’s very similar to @MojoVampire's proposal (followed up by @graingert and @Jason-Y-Z).
cc: @vstinner @VannTen
gh-74028: add `buffersize` parameter to `concurrent.futures.Executor.…
3.14.rst: move the pythongh-74028 resolution in the right section `co…
gh-74028: update `whatsnew/3.14.rst` post gh-125663 (#131214)
pythongh-74028: add `buffersize` parameter to `concurrent.futures.Exe…
pythongh-74028: update `whatsnew/3.14.rst` post pythongh-125663 (pyth…
Fix CI: break exception ref cycle in _MapResultIterator.__next__ (pyt…
Fix CI: break exception ref cycle in _MapResultIterator.__next__ (pyt…
gh-74028: `concurrent.futures.Executor.map`: avoid reference cycles w…