Repository navigation
STOP_ITERATION monitoring event is not triggered by unspecialized bytecode #116090
Description
Bug report
Bug description:
Repro: Check out this commit, build, and run python -mtest test_monitoring -mtest_implicit_stop_iteration.
Context: The unspecialized implementation of FOR_ITER has some built-in opcode fusion, in that it skips the following END_FOR and POP_TOP when tp_iternext returns NULL. When the iterator is a generator, as it is in ExceptionMonitoringTest.test_implicit_stop_iteration(), that means we skip the monitor_stop_iteration() call in INSTRUMENTED_END_FOR.
test_implicit_stop_iteration() passes as written, though, because the FOR_ITER in the helper function has been specialized to FOR_ITER_GEN by the time the generator finishes, and FOR_ITER_GEN's implementation doesn't perform the same fusion as FOR_ITER.
This bug can be exposed by making this change to test_implicit_stop_iteration(), so the FOR_ITER stays unspecialized while iterating over the generator in question.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Activity
swtaarrs commentedon Feb 29, 2024
It should be easy to remove the opcode fusion from
FOR_ITER, which would let theEND_FORhappen naturally. That could hurt performance a little, but maybe it's OK since common cases should be specialized anyway?gvanrossum commentedon Feb 29, 2024
For @markshannon.
markshannon commentedon Jun 26, 2024
FTR, this is expected behavior. You either get a
STOP_ITERATIONevent or aRAISEevent with aStopIterationexception (and correspondingEXCEPTION_HANDLEDevent).The reason for this slightly odd behavior is as follows:
StopIterationbeing raised as it isn't a "real" exception.StopIterationevery time a generator exhausts.StopIterationevents that would otherwise be optimized away, we need a way to provide those events without impacting performance for everyone else. That's theSTOP_ITERATIONevent.markshannon commentedon Jun 26, 2024
That would result in both the
RAISEevent and theSTOP_ITERATIONevent for unoptimized exits from generators.markshannon commentedon Jun 26, 2024
In hindsight, we should probably tried to have done the following:
STOP_ITERATIONto be disabled (for consistency withRAISEevents)STOP_ITERATIONinstead of aRAISEevent withStopIterationwhen a generator or coroutine is exhausted.EXCEPTION_HANDLEDevents for thoseRAISEevents that are converted toSTOP_ITERATIONevents.Maybe we could do this for 3.14
GH-116090: Fix test and clarify behavior for exception events when ex…
GH-116090: Fire RAISE events from _FOR_ITER_TIER_TWO (GH-122413)
pythonGH-116090: Fire RAISE events from _FOR_ITER_TIER_TWO (pythonGH-…
[3.13] pythonGH-116090: Fire RAISE events from _FOR_ITER_TIER_TWO (py…
[3.13] pythonGH-116090: Fire RAISE events from _FOR_ITER_TIER_TWO (py…
[3.13] GH-116090: Fire RAISE events from _FOR_ITER_TIER_TWO (GH-122419)
markshannon commentedon Jan 8, 2025
The current behavior might be a bit more complicated than it could be, but it works and any changes risk breaking backwards compatibility.