Skip to content

profiling SEGV with --mode=gil with freethreading #140739

Closed
Closed
profiling SEGV with --mode=gil with freethreading#140739
Labels
3.15bugs and security fixesstdlibStandard Library Python modules in the Lib/ directorytopic-profilingtype-crashA hard crash of the interpreter, possibly with a core dump

Description

@dimaqq
dimaqq
opened on Oct 29, 2025

Bug report

Bug description:

Command:
sudo -E time env PYTHONPATH=src python -m profiling.sampling -a --sort-tottime --flamegraph --mode=gil -d 999 test/test_unit.py 100 100

Version:
Python 3.15.0a1 free-threading build (main, Oct 14 2025, 21:34:02) [Clang 20.1.4 ] on darwin

macOS:
Darwin kot.local 24.6.0 Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:51 PDT 2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T8112 arm64

Workload summary:

with concurrent.futures.ThreadPoolExecutor() as executor:
    ...
    executor.map(some_func, [arg1s], [arg2s], ...)

Workload needs to be large enough and run long enough for the crash to happen.

  • large .map() arity (no issue with 10, crashes with 100)
  • more than several seconds long

The profiling "controller" process appears to crash, leaving the "workload" process running:

Output:

Resetting... [....................................................................................................]
Reset [0...................................................................................................]
0: [0001110001010101111110001110010101010101001010101000111101010111000111000101010111111000111001010101] --> 0
0: [0...................................................................................................]
0: [0011010101001100010111010000010111000101010000000100000100010100010101100000110011000101011001000001] --> 1
1: [0...................................................................................................]
1: [0011101000000000100110001000010001000001010000000100000010010000010100100001010011000001001100000010] --> 2
2: [0...................................................................................................]
2: [0001010000000010000000000000110000000011000000001100000010100000010110110000011000000011101100000000] --> 3
time: command terminated abnormally
        2.56 real         1.39 user         1.05 sys
fish: Job 1, 'sudo -E time env PYTHONPATH=src…' terminated by signal SIGSEGV (Address boundary error)
(jgol-peer) 🦐/c/j/peer (main) [SIGSEGV]> 3: [0...................................................................................................]
3: [0000100000000010000000000000110000000100000000000000000010000000000001000000000100000010100000000100] --> 4
4: [0...................................................................................................]
4: [0000000000000000000000000000100000000010000000000000000000000000000000000000001100000001100000000100] --> 5
5: [0...................................................................................................]
5: [0000000000000000000000000000000000000000000000000000000000000000000000000000011100000001010000000110] --> 6
6: [0...................................................................................................]
6: [0000000000000000000000000000000000000000000000000000000000000000000010000000010100000010010000000110] --> 7
7: [0...................................................................................................]
7: [0000000000000000000000000000000000000000000000000000000000000000000010000000010100000010010000000110] --> 8
8: [0...................................................................................................]
(jgol-peer) 🦐/c/j/peer (main) [SIGSEGV]> sudo8: [0000000000000000000000000000000000000000000000000000000000000000000010000000010100000010010000000110] --> 9test_unit.py 100 100
(jgol-peer) 🦐/c/j/peer (main) [SIGSEGV]> sudo ki9: [0...................................................................................................]
(jgol-peer) 🦐/c/j/peer (main) [SIGSEGV]> sudo killa9: [0000000000000000000000000000000000000000000000000000000000000000000010000000010100000010010000000110] --> 10
(jgol-peer) 🦐/c/j/peer (main) [SIGSEGV]> sudo killall 10: [0...................................................................................................]
(jgol-peer) 🦐/c/j/peer (main) [SIGSEGV]> sudo killall -10: [0000000000000000000000000000000000000000000000000000000000000000000010000000010100000010010000000110] --> 11
(jgol-peer) 🦐/c/j/peer (main) [SIGSEGV]> sudo killall -9 py11: [0...................................................................................................]
(jgol-peer) 🦐/c/j/peer (main) [SIGSEGV]> sudo killall -9 python
(jgol-peer) 🦐/c/j/peer (main)>

P.S. I'm sorry I don't have a compact reproducer at this point

CPython versions tested on:

3.15

Operating systems tested on:

macOS

Linked PRs

Activity

  1. dimaqq
    added
    type-bugAn unexpected behavior, bug, or error
    on Oct 29, 2025
  2. dimaqq
    changed the title [-]profiling SEGV with --model=gil with freethreading[/-] [+]profiling SEGV with --mode=gil with freethreading[/+] on Oct 29, 2025
  3. sergey-miryanov
    added
    stdlibStandard Library Python modules in the Lib/ directory
    on Oct 29, 2025
  4. efimov-mikhail
    added
    type-crashA hard crash of the interpreter, possibly with a core dump
    3.15bugs and security fixes
    on Oct 29, 2025
  5. StanFromIreland
    added
    topic-profiling
    and removed
    type-bugAn unexpected behavior, bug, or error
    on Oct 29, 2025

  6. pablogsal commented on Dec 26, 2025

    @pablogsal
    pablogsal
    on Dec 26, 2025
    Member

    I cannot repro anymore with the current main. I used this script:

    import concurrent.futures
import sys
import time


def cpu_bound_work(n, iterations):
    result = 0
    for _ in range(iterations):
        for i in range(1000):
            result += i * n
    return result


def main():
    arity = int(sys.argv[1]) if len(sys.argv) > 1 else 100
    iterations = int(sys.argv[2]) if len(sys.argv) > 2 else 100

    print(f"Running with arity={arity}, iterations={iterations}")
    print(f"Python: {sys.version}")
    print()

    arg1s = list(range(arity))
    arg2s = [iterations] * arity

    round_num = 0
    while True:
        print(f"Round {round_num}...")
        with concurrent.futures.ThreadPoolExecutor() as executor:
            results = list(executor.map(cpu_bound_work, arg1s, arg2s))
        print(f"  Completed {len(results)} tasks")
        round_num += 1
        time.sleep(0.1)


if __name__ == "__main__":
    main()

  7. pablogsal commented on Dec 26, 2025

    @pablogsal
    pablogsal
    on Dec 26, 2025
    Member

    Let's reopen if you see it again

  8. pablogsal
    closed this as not plannedon Dec 26, 2025

  9. dimaqq commented on Dec 26, 2025

    @dimaqq
    dimaqq
    on Dec 26, 2025
    ContributorAuthor

    I can't repro with your code either.
    Let me dig up the original workload, and I'll make a separate repo for the reporoducer.

  10. dimaqq commented on Dec 26, 2025

    @dimaqq
    dimaqq
    on Dec 26, 2025
    ContributorAuthor

    Hmmm... this is getting more interesting: I get a crash in my existing venv, but it's harder to get a crash, and I'm getting potentially different crash with fresh versions of dependencies:

    (jgol-peer) 🦐/c/j/peer-test-me-relock (main)> sudo -H -E time env PYTHONPATH=src uvx -p 3.15t --with 'ops[testing]==3.3.1,pytest,pip' python -m profiling.sampling -a --sort-tottime --flamegraph --mode=gil -d 999 test/test_unit.py 100 100
      Built pyyaml==6.0.3
Installed 14 packages in 13ms
Resetting... [....................................................................................................]
Reset [0...................................................................................................]
Reset [0001110001010101111110001110010101010101001010101000111101010111000111000101010111111000111001010101]
0: [0001110001010101111110001110010101010101001010101000111101010111000111000101010111111000111001010101] --> 0
0: [0...................................................................................................]
0: [0011010101001100010111010000010111000101010000000100000100010100010101100000110011000101011001000001] --> 1
       22.81 real         6.05 user         2.26 sys
(jgol-peer) 🦐/c/j/peer-test-me-relock (main) [SIGBUS]>

    That's newer Python 3.15.0a2 free-threading build (main, Dec 5 2025, 20:59:15) [Clang 21.1.4 ] on darwin
    The shell shows that last command died SIGBUS, also the test is not supposed to exit so quickly, should have continued until --> 100 rounds.

    Here are all the dependencies used:

    importlib_metadata==8.7.1
iniconfig==2.3.0
opentelemetry-api==1.39.1
ops==3.3.1
ops-scenario==8.3.1
packaging==25.0
pluggy==1.6.0
Pygments==2.19.2
pytest==9.0.2
PyYAML==6.0.3
typing_extensions==4.15.0
websocket-client==1.9.0
zipp==3.23.0

    I think that only PyYAML is a compiled dep, and everything else is pure Python.

    I'll try to make a smaller MRE.

  11. pablogsal commented on Dec 26, 2025

    @pablogsal
    pablogsal
    on Dec 26, 2025
    Member

    Thanks! It will help immensely if you could give us a trace with lldb. Also if you could test with main that would help too. 3.15.0a2 is quite outdated at this point

  12. pablogsal commented on Dec 26, 2025

    @pablogsal
    pablogsal
    on Dec 26, 2025
    Member

    Can you paste here the original workload even if is huge?

  13. pablogsal
    reopened this on Dec 26, 2025

  14. pablogsal commented on Dec 26, 2025

    @pablogsal
    pablogsal
    on Dec 26, 2025
    Member

    Ah I think I found the problem

  15. dimaqq commented on Dec 26, 2025

    @dimaqq
    dimaqq
    on Dec 26, 2025
    ContributorAuthor

    MRE https://github.com/dimaqq/repro-cpython-140739

  16. pablogsal commented on Dec 26, 2025

    @pablogsal
    pablogsal
    on Dec 26, 2025
    Member

    Thanks a lot! Yeah, this indeed confirms my theory. Will work on a fix! This was very useful thanks for all the help

  17. dimaqq commented on Dec 26, 2025

    @dimaqq
    dimaqq
    on Dec 26, 2025
    ContributorAuthor

    No worries, thank you for you hard work during the holiday season ❤️

  18. pablogsal
    added 2 commits that reference this issue on Dec 26, 2025

    pythongh-140739: Fix crashes from corrupted remote memory

    8dc7a48

    gh-140739: Fix crashes from corrupted remote memory (#143190)

    d3d4cf9

  19. pablogsal commented on Dec 26, 2025

    @pablogsal
    pablogsal
    on Dec 26, 2025
    Member

    @dimaqq can you check again with the current main? :)

  20. pablogsal
    closed this as completedon Dec 26, 2025
  21. pablogsal
    added 2 commits that reference this issue on Dec 27, 2025

    pythongh-140739: Fix missing exception on allocation failure in Binar…

    c65d0f6

    gh-140739: Fix missing exception on allocation failure in BinaryWriter (

    5436289

  22. dimaqq commented on Dec 29, 2025

    @dimaqq
    dimaqq
    on Dec 29, 2025
    ContributorAuthor

    I'll have to get back to this in 2026.
    (couldn't build the main branch, PEBKAC; and off to holidays)

  23. thunder-coding
    added 2 commits that reference this issue on Feb 15, 2026

    pythongh-140739: Fix crashes from corrupted remote memory (python#143190

    8dce99f

    pythongh-140739: Fix missing exception on allocation failure in Binar…

    fd69a27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.15bugs and security fixesstdlibStandard Library Python modules in the Lib/ directorytopic-profilingtype-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions