Repository navigation
JIT: Use-after-free in stop_tracing_and_jit #143604
Description
Crash report
What happened?
It's possible to make ASan detect a use-after-free in stop_tracing_and_jit by running the code below in a patched build. The interpreter must be called from a venv in which lafleur is installed as editable, likely due to that priming the GC into a state suitable for the crash.
This is the most elusive crash I've ever encountered, and this convoluted repro took a long time to figure out.
Necessary diff (couldn't make it work with the latest support for env var policy):
diff --git a/Include/internal/pycore_backoff.h b/Include/internal/pycore_backoff.h
index fadd11f04ec..3e779a141af 100644
--- a/Include/internal/pycore_backoff.h
+++ b/Include/internal/pycore_backoff.h
@@ -125,7 +125,7 @@ trigger_backoff_counter(void)
// For example, 4095 does not work for the nqueens benchmark on pyperformance
// as we always end up tracing the loop iteration's
// exhaustion iteration. Which aborts our current tracer.
-#define JUMP_BACKWARD_INITIAL_VALUE 4000
+#define JUMP_BACKWARD_INITIAL_VALUE 63
#define JUMP_BACKWARD_INITIAL_BACKOFF 6
static inline _Py_BackoffCounter
initial_jump_backoff_counter(_PyPolicy *policy)
@@ -139,7 +139,7 @@ initial_jump_backoff_counter(_PyPolicy *policy)
* Must be larger than ADAPTIVE_COOLDOWN_VALUE,
* otherwise when a side exit warms up we may construct
* a new trace before the Tier 1 code has properly re-specialized. */
-#define SIDE_EXIT_INITIAL_VALUE 4000
+#define SIDE_EXIT_INITIAL_VALUE 63
#define SIDE_EXIT_INITIAL_BACKOFF 6
static inline _Py_BackoffCounterTo install lafleur as editable, first create a venv and activate it, then clone the lafleur repo and run pip install -e /path/to/lafleur/clone.
This MRE must be invoked like this:
/path/to/venv/with/editable/lafleur/install/bin/python mre.py
MRE
source1 = """
import random
def uop_harness_f1():
class _Chameot:
def _(self):
r""
def _h_(self):
super()
class Base_super_1324:
def _(self):
ser = 0
def f(self):
ser += 1
class Sub_super_1324(Base_super_1324):
try:
_1324.f()
except Exception:
pass
Sub_super_1324.__bases__ = (object,)
class Base_super_8139:
def _(self):
ser = 0
class Sub_super_8139(Base_super_8139):
def _(self):
super()
def f(self):
super().f()
Sub_super_8139._(
object,
)
instance_super_8139 = Sub_super_8139()
class Base_super_4784:
def _(self):
ser = 0
def f(self): ...
class Sub_super_4784(Base_super_4784):
def _(self):
super()
def f(self):
Sub_super_4784._(
object,
)
Sub_super_4784.__bases__ = (object,)
for _ in range(10):
try:
_ = instance_super_8139.f()
except AttributeError:
pass
class Base_super_2371:
def _(self):
ser = 0
def f(self): ...
class Sub_super_2371(Base_super_2371):
def _(self):
super()
instance_super_2371 = Sub_super_2371.__bases__ = (object,)
for _ in range(10):
try:
_ = instance_super_2371.f()
except AttributeError:
pass
for i_f1 in range(300):
uop_harness_f1()
"""
source2 = """
def f1():
class C(int):
def _(self): pass
class Base1:
def ___(self):
ser = 0
def f(self):
ser += 1
class Sub1(Base1):
def ___(self):
super()
def f(self):
super().f()
if e:
Sub1.___(object)
instance1 = Sub1()
Sub1.__bases__ = (object,)
for _ in range(100):
try:
_ = instance1.f()
except AttributeError: pass
class Base2:
def ___(self):
ser = 0
def f(self):
ser += 1
class Sub2(Base2):
def ___(self):
super()
def f(self):
Sub2.___(
object,
)
class Base3:
def ___(self):
ser = 0
def f(self):
ser += 1
class Sub3(Base3):
def ___(self):
super()
def f(self):
super()
class Base4:
def ___(self):
ser = 0
def f(self):
ser += 1
class Sub4(Base4):
def ___(self):
super()
def f(self):
super()
for _ in range(300):
f1()
"""
import argparse
import ctypes
import json
import sys
from pathlib import Path
def run(files):
namespace = {}
for sourcename in [source1, source2]:
code = compile(sourcename, "code", "exec")
exec(code, namespace)
def f():
parser = argparse.ArgumentParser()
parser.add_argument("files", nargs="?")
args = parser.parse_args()
run(args.files)
f()ASan output
=================================================================
==3031457==ERROR: AddressSanitizer: heap-use-after-free on address 0x7d2eb305a936 at pc 0x5e651cdbe309 bp 0x7ffe01f85140 sp 0x7ffe01f85138
WRITE of size 2 at 0x7d2eb305a936 thread T0
#0 0x5e651cdbe308 in stop_tracing_and_jit /home/danzin/projects/jit_cpython/Python/ceval.c:1483:33
#1 0x5e651cd7b242 in _PyEval_EvalFrameDefault /home/danzin/projects/jit_cpython/Python/generated_cases.c.h:11823:27
#2 0x5e651cd56e77 in _PyEval_EvalFrame /home/danzin/projects/jit_cpython/./Include/internal/pycore_ceval.h:118:16
#3 0x5e651cd56e77 in _PyEval_Vector /home/danzin/projects/jit_cpython/Python/ceval.c:2539:12
#4 0x5e651cd56894 in PyEval_EvalCode /home/danzin/projects/jit_cpython/Python/ceval.c:1005:21
#5 0x5e651cd49d06 in builtin_exec_impl /home/danzin/projects/jit_cpython/Python/bltinmodule.c:1193:17
#6 0x5e651cd49d06 in builtin_exec /home/danzin/projects/jit_cpython/Python/clinic/bltinmodule.c.h:579:20
#7 0x5e651cae42eb in cfunction_vectorcall_FASTCALL_KEYWORDS /home/danzin/projects/jit_cpython/Objects/methodobject.c:465:24
#8 0x5e651c9c508f in _PyObject_VectorcallTstate /home/danzin/projects/jit_cpython/./Include/internal/pycore_call.h:136:11
#9 0x5e651cd57fdc in _Py_VectorCallInstrumentation_StackRefSteal /home/danzin/projects/jit_cpython/Python/ceval.c:1094:11
#10 0x5e651cd93ebe in _PyEval_EvalFrameDefault /home/danzin/projects/jit_cpython/Python/generated_cases.c.h:1777:35
#11 0x5e651cd56e77 in _PyEval_EvalFrame /home/danzin/projects/jit_cpython/./Include/internal/pycore_ceval.h:118:16
#12 0x5e651cd56e77 in _PyEval_Vector /home/danzin/projects/jit_cpython/Python/ceval.c:2539:12
#13 0x5e651cd56894 in PyEval_EvalCode /home/danzin/projects/jit_cpython/Python/ceval.c:1005:21
#14 0x5e651d37d26e in run_eval_code_obj /home/danzin/projects/jit_cpython/Python/pythonrun.c:1366:12
#15 0x5e651d37c43b in run_mod /home/danzin/projects/jit_cpython/Python/pythonrun.c:1469:19
#16 0x5e651d376a3c in pyrun_file /home/danzin/projects/jit_cpython/Python/pythonrun.c:1294:15
#17 0x5e651d37459c in _PyRun_SimpleFileObject /home/danzin/projects/jit_cpython/Python/pythonrun.c:518:13
#18 0x5e651d37390d in _PyRun_AnyFileObject /home/danzin/projects/jit_cpython/Python/pythonrun.c:81:15
#19 0x5e651d3f017a in pymain_run_file_obj /home/danzin/projects/jit_cpython/Modules/main.c:410:15
#20 0x5e651d3f017a in pymain_run_file /home/danzin/projects/jit_cpython/Modules/main.c:429:15
#21 0x5e651d3ee243 in pymain_run_python /home/danzin/projects/jit_cpython/Modules/main.c:691:21
#22 0x5e651d3ee243 in Py_RunMain /home/danzin/projects/jit_cpython/Modules/main.c:772:5
#23 0x5e651d3ef146 in pymain_main /home/danzin/projects/jit_cpython/Modules/main.c:802:12
#24 0x5e651d3ef2b7 in Py_BytesMain /home/danzin/projects/jit_cpython/Modules/main.c:826:12
#25 0x7f7eb3e2a574 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#26 0x7f7eb3e2a627 in __libc_start_main csu/../csu/libc-start.c:360:3
#27 0x5e651c7194f4 in _start (/home/danzin/projects/jit_cpython/python+0x2e04f4) (BuildId: d44f0b1b57ec0bcbb2ebba729e428f82744768b0)
0x7d2eb305a936 is located 182 bytes inside of 1496-byte region [0x7d2eb305a880,0x7d2eb305ae58)
freed by thread T0 here:
#0 0x5e651c7be7fa in free (/home/danzin/projects/jit_cpython/python+0x3857fa) (BuildId: d44f0b1b57ec0bcbb2ebba729e428f82744768b0)
#1 0x5e651d2e325f in _PyExecutor_Free /home/danzin/projects/jit_cpython/Python/optimizer.c:261:5
#2 0x5e651d2e325f in _Py_ClearExecutorDeletionList /home/danzin/projects/jit_cpython/Python/optimizer.c:301:13
#3 0x5e651ce7731d in _Py_HandlePending /home/danzin/projects/jit_cpython/Python/ceval_gil.c:1401:9
#4 0x5e651cd86dd3 in check_periodics /home/danzin/projects/jit_cpython/Python/ceval_macros.h:499:16
#5 0x5e651cd86dd3 in _PyEval_EvalFrameDefault /home/danzin/projects/jit_cpython/Python/generated_cases.c.h:2395:27
#6 0x5e651cd56e77 in _PyEval_EvalFrame /home/danzin/projects/jit_cpython/./Include/internal/pycore_ceval.h:118:16
#7 0x5e651cd56e77 in _PyEval_Vector /home/danzin/projects/jit_cpython/Python/ceval.c:2539:12
#8 0x5e651cd56894 in PyEval_EvalCode /home/danzin/projects/jit_cpython/Python/ceval.c:1005:21
#9 0x5e651cd49d06 in builtin_exec_impl /home/danzin/projects/jit_cpython/Python/bltinmodule.c:1193:17
#10 0x5e651cd49d06 in builtin_exec /home/danzin/projects/jit_cpython/Python/clinic/bltinmodule.c.h:579:20
#11 0x5e651cae42eb in cfunction_vectorcall_FASTCALL_KEYWORDS /home/danzin/projects/jit_cpython/Objects/methodobject.c:465:24
#12 0x5e651c9c508f in _PyObject_VectorcallTstate /home/danzin/projects/jit_cpython/./Include/internal/pycore_call.h:136:11
#13 0x5e651cd57fdc in _Py_VectorCallInstrumentation_StackRefSteal /home/danzin/projects/jit_cpython/Python/ceval.c:1094:11
#14 0x5e651cd93ebe in _PyEval_EvalFrameDefault /home/danzin/projects/jit_cpython/Python/generated_cases.c.h:1777:35
#15 0x5e651cd56e77 in _PyEval_EvalFrame /home/danzin/projects/jit_cpython/./Include/internal/pycore_ceval.h:118:16
#16 0x5e651cd56e77 in _PyEval_Vector /home/danzin/projects/jit_cpython/Python/ceval.c:2539:12
#17 0x5e651cd56894 in PyEval_EvalCode /home/danzin/projects/jit_cpython/Python/ceval.c:1005:21
#18 0x5e651d37d26e in run_eval_code_obj /home/danzin/projects/jit_cpython/Python/pythonrun.c:1366:12
#19 0x5e651d37c43b in run_mod /home/danzin/projects/jit_cpython/Python/pythonrun.c:1469:19
#20 0x5e651d376a3c in pyrun_file /home/danzin/projects/jit_cpython/Python/pythonrun.c:1294:15
#21 0x5e651d37459c in _PyRun_SimpleFileObject /home/danzin/projects/jit_cpython/Python/pythonrun.c:518:13
#22 0x5e651d37390d in _PyRun_AnyFileObject /home/danzin/projects/jit_cpython/Python/pythonrun.c:81:15
#23 0x5e651d3f017a in pymain_run_file_obj /home/danzin/projects/jit_cpython/Modules/main.c:410:15
#24 0x5e651d3f017a in pymain_run_file /home/danzin/projects/jit_cpython/Modules/main.c:429:15
#25 0x5e651d3ee243 in pymain_run_python /home/danzin/projects/jit_cpython/Modules/main.c:691:21
#26 0x5e651d3ee243 in Py_RunMain /home/danzin/projects/jit_cpython/Modules/main.c:772:5
#27 0x5e651d3ef146 in pymain_main /home/danzin/projects/jit_cpython/Modules/main.c:802:12
#28 0x5e651d3ef2b7 in Py_BytesMain /home/danzin/projects/jit_cpython/Modules/main.c:826:12
#29 0x7f7eb3e2a574 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#30 0x7f7eb3e2a627 in __libc_start_main csu/../csu/libc-start.c:360:3
#31 0x5e651c7194f4 in _start (/home/danzin/projects/jit_cpython/python+0x2e04f4) (BuildId: d44f0b1b57ec0bcbb2ebba729e428f82744768b0)
previously allocated by thread T0 here:
#0 0x5e651c7bea98 in malloc (/home/danzin/projects/jit_cpython/python+0x385a98) (BuildId: d44f0b1b57ec0bcbb2ebba729e428f82744768b0)
#1 0x5e651cb326a1 in _PyMem_DebugRawAlloc /home/danzin/projects/jit_cpython/Objects/obmalloc.c:2926:24
#2 0x5e651cb326a1 in _PyMem_DebugRawMalloc /home/danzin/projects/jit_cpython/Objects/obmalloc.c:2959:12
#3 0x5e651cb326a1 in _PyMem_DebugMalloc /home/danzin/projects/jit_cpython/Objects/obmalloc.c:3124:12
#4 0x5e651ce5df00 in _PyObject_MallocWithType /home/danzin/projects/jit_cpython/./Include/internal/pycore_object_alloc.h:46:17
#5 0x5e651ce5df00 in gc_alloc /home/danzin/projects/jit_cpython/Python/gc.c:2352:17
#6 0x5e651ce5df00 in _PyObject_GC_NewVar /home/danzin/projects/jit_cpython/Python/gc.c:2394:25
#7 0x5e651d2e05c0 in allocate_executor /home/danzin/projects/jit_cpython/Python/optimizer.c:1240:30
#8 0x5e651d2e05c0 in make_executor_from_uops /home/danzin/projects/jit_cpython/Python/optimizer.c:1327:35
#9 0x5e651d2e05c0 in uop_optimize /home/danzin/projects/jit_cpython/Python/optimizer.c:1506:35
#10 0x5e651d2e05c0 in _PyOptimizer_Optimize /home/danzin/projects/jit_cpython/Python/optimizer.c:164:15
#11 0x5e651cdbe243 in stop_tracing_and_jit /home/danzin/projects/jit_cpython/Python/ceval.c:1461:15
#12 0x5e651cd7b242 in _PyEval_EvalFrameDefault /home/danzin/projects/jit_cpython/Python/generated_cases.c.h:11823:27
#13 0x5e651cd56e77 in _PyEval_EvalFrame /home/danzin/projects/jit_cpython/./Include/internal/pycore_ceval.h:118:16
#14 0x5e651cd56e77 in _PyEval_Vector /home/danzin/projects/jit_cpython/Python/ceval.c:2539:12
#15 0x5e651cd56894 in PyEval_EvalCode /home/danzin/projects/jit_cpython/Python/ceval.c:1005:21
#16 0x5e651cd49d06 in builtin_exec_impl /home/danzin/projects/jit_cpython/Python/bltinmodule.c:1193:17
#17 0x5e651cd49d06 in builtin_exec /home/danzin/projects/jit_cpython/Python/clinic/bltinmodule.c.h:579:20
#18 0x5e651cae42eb in cfunction_vectorcall_FASTCALL_KEYWORDS /home/danzin/projects/jit_cpython/Objects/methodobject.c:465:24
#19 0x5e651c9c508f in _PyObject_VectorcallTstate /home/danzin/projects/jit_cpython/./Include/internal/pycore_call.h:136:11
#20 0x5e651cd57fdc in _Py_VectorCallInstrumentation_StackRefSteal /home/danzin/projects/jit_cpython/Python/ceval.c:1094:11
#21 0x5e651cd93ebe in _PyEval_EvalFrameDefault /home/danzin/projects/jit_cpython/Python/generated_cases.c.h:1777:35
#22 0x5e651cd56e77 in _PyEval_EvalFrame /home/danzin/projects/jit_cpython/./Include/internal/pycore_ceval.h:118:16
#23 0x5e651cd56e77 in _PyEval_Vector /home/danzin/projects/jit_cpython/Python/ceval.c:2539:12
#24 0x5e651cd56894 in PyEval_EvalCode /home/danzin/projects/jit_cpython/Python/ceval.c:1005:21
#25 0x5e651d37d26e in run_eval_code_obj /home/danzin/projects/jit_cpython/Python/pythonrun.c:1366:12
#26 0x5e651d37c43b in run_mod /home/danzin/projects/jit_cpython/Python/pythonrun.c:1469:19
#27 0x5e651d376a3c in pyrun_file /home/danzin/projects/jit_cpython/Python/pythonrun.c:1294:15
#28 0x5e651d37459c in _PyRun_SimpleFileObject /home/danzin/projects/jit_cpython/Python/pythonrun.c:518:13
#29 0x5e651d37390d in _PyRun_AnyFileObject /home/danzin/projects/jit_cpython/Python/pythonrun.c:81:15
#30 0x5e651d3f017a in pymain_run_file_obj /home/danzin/projects/jit_cpython/Modules/main.c:410:15
#31 0x5e651d3f017a in pymain_run_file /home/danzin/projects/jit_cpython/Modules/main.c:429:15
#32 0x5e651d3ee243 in pymain_run_python /home/danzin/projects/jit_cpython/Modules/main.c:691:21
#33 0x5e651d3ee243 in Py_RunMain /home/danzin/projects/jit_cpython/Modules/main.c:772:5
#34 0x5e651d3ef146 in pymain_main /home/danzin/projects/jit_cpython/Modules/main.c:802:12
#35 0x5e651d3ef2b7 in Py_BytesMain /home/danzin/projects/jit_cpython/Modules/main.c:826:12
#36 0x7f7eb3e2a574 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#37 0x7f7eb3e2a627 in __libc_start_main csu/../csu/libc-start.c:360:3
#38 0x5e651c7194f4 in _start (/home/danzin/projects/jit_cpython/python+0x2e04f4) (BuildId: d44f0b1b57ec0bcbb2ebba729e428f82744768b0)
SUMMARY: AddressSanitizer: heap-use-after-free /home/danzin/projects/jit_cpython/Python/ceval.c:1483:33 in stop_tracing_and_jit
Shadow bytes around the buggy address:
0x7d2eb305a680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x7d2eb305a700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x7d2eb305a780: 00 fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x7d2eb305a800: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x7d2eb305a880: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x7d2eb305a900: fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd
0x7d2eb305a980: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x7d2eb305aa00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x7d2eb305aa80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x7d2eb305ab00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x7d2eb305ab80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==3031457==ABORTING
Output from running with PYTHON_LLTRACE=4 PYTHON_OPT_DEBUG=4:
193517_uaf_lltrace_opt_debug.txt
Found using lafleur.
CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
Output from running 'python -VV' on the command line:
Python 3.15.0a3+ (heads/main-dirty:6d54b6ac7d5, Jan 9 2026, 09:11:42) [Clang 21.1.2 (2ubuntu6)]
Activity
[-]Use-after-free in `stop_tracing_and_jit`[/-][+]JIT: Use-after-free in `stop_tracing_and_jit`[/+]Fidget-Spinner commentedon Jan 9, 2026
Oh it seems that the comment about holding a strong reference to the previous exit executor is wrong! I actually forgot to Py_XNewRef and store the previous executor in
_Py_InitializeTracing.pythongh-143604: Hold strong reference to executor during JIT tracing
gh-143604: Hold strong reference to executor during JIT tracing (GH-1…
Fidget-Spinner commentedon Jan 10, 2026
@devdanzin should be fixed on main now. Can you please verify?
devdanzin commentedon Jan 10, 2026
Verified, it's fixed, thanks!
pythongh-143604: Hold strong reference to executor during JIT tracing (…
pythongh-143604: Hold strong reference to executor during JIT tracing (…