Skip to content

Conversation

@ZeroIntensity
Copy link
Member

@ZeroIntensity ZeroIntensity commented Nov 6, 2025

What does this PR do?

Adds support for Python 3.14 by running the test suite until it all passes.

Why was it initiated? Any relevant Issues?

PR Checklist

  • Correct base branch selected? Should be develop branch.
  • Enabled commit hook or executed ./bin/autoformat-nuitka-source.
  • All tests still pass. Check the Developer Manual about Running the Tests. There are GitHub
    Actions tests that cover the most important things however, and you are welcome to rely on those,
    but they might not cover enough.
  • Ideally new features or fixed regressions ought to be covered via new tests.
  • Ideally new or changed features have documentation updates.

Summary by Sourcery

Add compatibility for Python 3.14 by updating generated C runtime integration, annotation handling in compiled functions, build scripts, and tests.

Enhancements:

  • Expand compiled generator integration and opcode deoptimization mapping to support Python 3.14's new interpreter frame API
  • Introduce deferred annotation evaluation via a new annotate mechanism and update compiled function annotation getters/setters
  • Refine dict unhashable key error formatting and reverse context manager exit/enter lookup order for Python 3.14

Build:

  • Adjust SCons build script to skip hacl library linkage on Python 3.14

Tests:

  • Update async test to use asyncio.new_event_loop() for compatibility with Python 3.14

Copy link

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We failed to fetch the diff for pull request #3662

You can try again by commenting this pull request with @sourcery-ai review, or contact us for help.

In 3.14, support for calling asyncio.get_event_loop() without a running
event loop was removed. Instead, use asyncio.new_event_loop().
We might be able to use asyncio.run(), but I won't touch that for now.
@ZeroIntensity ZeroIntensity marked this pull request as ready for review November 7, 2025 17:01
@sourcery-ai
Copy link

sourcery-ai bot commented Nov 7, 2025

Reviewer's Guide

This PR implements Python 3.14 support by updating generator internals and opcode mappings for the new interpreter frame API, introducing a deferred annotation mechanism with annotate hooks in both AST transforms and runtime, adjusting built-in error messages and lookup orders to match CPython 3.14 behavior, refining the build configuration for the Python 3.14 HACL library, and updating tests to use the new asyncio event loop API.

Sequence diagram for annotate deferred annotation retrieval (Python 3.14+)

sequenceDiagram
    participant User
    participant Nuitka_FunctionObject
    participant DeferredAnnotateFunction
    User->>Nuitka_FunctionObject: Access __annotations__
    alt m_annotate is set
        Nuitka_FunctionObject->>DeferredAnnotateFunction: Call __annotate__(format=1)
        DeferredAnnotateFunction-->>Nuitka_FunctionObject: Return dict of annotations
        Nuitka_FunctionObject-->>User: Return dict
    else m_annotate is not set
        Nuitka_FunctionObject-->>User: Return empty dict
    end
Loading

Sequence diagram for context manager protocol lookup order (Python 3.14+)

sequenceDiagram
    participant Nuitka
    participant ContextManagerObject
    Nuitka->>ContextManagerObject: Lookup __exit__
    alt __exit__ found
        Nuitka->>ContextManagerObject: Lookup __enter__
        alt __enter__ found
            Nuitka-->>Nuitka: Proceed with context manager
        else __enter__ missing
            Nuitka-->>Nuitka: Raise error (missing __enter__)
        end
    else __exit__ missing
        Nuitka-->>Nuitka: Raise error (missing __exit__)
    end
Loading

Class diagram for deferred annotation mechanism in Nuitka_FunctionObject (Python 3.14+)

classDiagram
    class Nuitka_FunctionObject {
        PyObject *m_annotations
        PyObject *m_annotate
        PyObject *m_qualname
        ...
    }
    Nuitka_FunctionObject : +get_annotations()
    Nuitka_FunctionObject : +set_annotations(value)
    Nuitka_FunctionObject : +get_annotate()
    Nuitka_FunctionObject : +set_annotate(value)
    Nuitka_FunctionObject : +clone()
    Nuitka_FunctionObject : +tp_dealloc()
    Nuitka_FunctionObject <|-- DeferredAnnotateFunction
    class DeferredAnnotateFunction {
        +__call__(format)
        returns dict or raises NotImplementedError
    }
Loading

Class diagram for AST transform: deferred annotation function creation

classDiagram
    class ExpressionFunctionBody {
        provider
        name
        code_object
        flags
        parameters
        ...
    }
    class ExpressionFunctionRef {
        function_body
        source_ref
    }
    class ExpressionFunctionCreation {
        function_ref
        defaults
        kw_defaults
        annotations
        source_ref
    }
    ExpressionFunctionCreation --> ExpressionFunctionRef
    ExpressionFunctionRef --> ExpressionFunctionBody
Loading

File-Level Changes

Change Details Files
Update generator integration and opcode mapping for Python 3.14
  • Remove upper version guard (<3e0) and enable uncompiled generator throw integration for all Python 3
  • Add new opcode deoptimization entries under PYTHON_VERSION >= 0x3e0
  • Adapt generator send/close/throw functions to use the new _PyInterpreterFrame API and updated stack push/pop
  • Adjust yield-from and frame state handling for the new interpreter frame layout
nuitka/build/static_src/CompiledGeneratorTypeUncompiledIntegration.c
Introduce deferred annotation mechanism in AST transforms
  • Add deferredAnnotateBody and makeDeferredAnnotateFunction helpers
  • Switch buildParameterAnnotations to return a deferred function on Python >= 3.14
  • Import InternalModule helper for annotation support
nuitka/tree/ReformulationFunctionStatements.py
Support dynamic annotate hook in compiled functions
  • Add m_annotate field to Nuitka_FunctionObject struct
  • Implement get/set methods for annotate property under Python >= 3.14
  • Adjust function new, clone, dealloc and get_annotations to use m_annotate for deferred annotations
nuitka/build/static_src/CompiledFunctionType.c
nuitka/build/include/nuitka/compiled_function.h
Update unhashable dict key error message for Python 3.14
  • Change TypeError template to “cannot use '%s' as a dict key (unhashable type: '%s')” on Python >= 3.14
  • Preserve original message and side-effect wrapping logic for older versions
nuitka/nodes/DictionaryNodes.py
Adjust with-statement lookup order for exit/enter
  • Reverse attribute assignment order for synchronous with on 3.6–<3.14 to reflect CPython’s lookup order change in 3.14
nuitka/tree/ReformulationWithStatements.py
Enhance context manager error reporting
  • Add missing enter special-case exception message in LOOKUP_SPECIAL
nuitka/build/static_src/HelpersAttributes.c
Update build configuration for Python 3.14 HACL library
  • Disable HACL linkage for Python >= 3.14 by setting hacl_version to None
nuitka/build/SconsPythonBuild.py
Update test to use new asyncio event loop API
  • Replace asyncio.get_event_loop() with asyncio.new_event_loop() in run_until_complete call
tests/basics/ReferencingTest35.py

Possibly linked issues

  • Code Cleanup #1: The PR adds Python 3.14 support by modifying version checks and annotation handling, directly addressing the issue's request.

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

Copy link

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey there - I've reviewed your changes and they look great!

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location> `nuitka/build/include/nuitka/compiled_function.h:69-71` </location>
<code_context>
     PyObject *m_annotations;
 #endif

+#if PYTHON_VERSION >= 0x300
+    PyObject *m_annotate;
+#endif
+
</code_context>

<issue_to_address>
**suggestion:** m_annotate is defined for all Python 3, but only used for 3.14+.

Restrict the definition of m_annotate to Python 3.14+ to improve clarity and prevent confusion in earlier versions.

```suggestion
#if PYTHON_VERSION >= 0x30E0
    PyObject *m_annotate;
#endif
```
</issue_to_address>

### Comment 2
<location> `nuitka/nodes/DictionaryNodes.py:236-245` </location>
<code_context>
+                if python_version >= 0x3E0:
</code_context>

<issue_to_address>
**issue:** Error message for unhashable dict keys updated for Python 3.14+, but side effects handling is inconsistent.

Please ensure side effect handling is consistent for both error message branches to avoid potential loss of side effects.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment on lines +236 to +245
if python_version >= 0x3E0:
result = makeRaiseExceptionExpressionFromTemplate(
exception_type="TypeError",
template="cannot use '%s' as a dict key (unhashable type: '%s')",
template_args=(
makeExpressionAttributeLookup(
expression=key.getTypeValue(),
attribute_name="__name__",
source_ref=key.source_ref,
),
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: Error message for unhashable dict keys updated for Python 3.14+, but side effects handling is inconsistent.

Please ensure side effect handling is consistent for both error message branches to avoid potential loss of side effects.

Copy link
Member

@kayhayen kayhayen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The __annotations__ change for the function level could be useful, I need to check it out, for classes it will be a lot more annoying to make.

}
function->m_annotations = CALL_FUNCTION_WITH_SINGLE_ARG(tstate, function->m_annotate, _PyLong_GetOne());
if (function->m_annotations == NULL) {
return NULL;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will that function be executed each time when it throws an error over and over? Is that compatible?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems so, you can try this script:

def test(x):
    pass

def evil(format):
    raise RuntimeError("evil")

test.__annotate__ = evil

for _ in range(3):
    try:
        print(test.__annotations__)
    except RuntimeError as error:
        print(error)

// For simplicity's sake, the annotations parameter doubles as the __annotate__
// parameter on 3.14+
assert(annotations == NULL || PyCallable_Check(annotations));
result->m_annotations = NULL;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, m_annotations seems now unused with 3.14?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it's used as a cache. Once __annotate__ is called, the result is stored in __annotations__, so the annotations don't have to be re-evaluated every time.

const uint8_t Nuitka_PyOpcode_Deopt[256] = {
#if PYTHON_VERSION >= 0x3d0
#if PYTHON_VERSION >= 0x3e0
[121] = 121,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious, why are there plain numbers used in CPython now?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unknown opcodes need to be marked as de-opting to themselves to support some third-party JIT compilers. See python/cpython#128045. It's probably fine to omit it for Nuitka, but I don't know what will happen if we do.

)
# On 3.14+, annotations are deferred by default.
if python_version >= 0x3E0:
return makeDeferredAnnotateFunction(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you saying, that all this is doing, is to create a function that returns the dictionary, just later? I thought the actual code of the annotations is to be executed delayed.

def f():
   x : something()

f()
f.__annotations__

I was assuming this will not crash when calling f().

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, that's ignored in all versions, but this is showing it:

class C():
   x : something()

print(C.__annotations__)

With 3.14, this crashes when .__annotations__ is looked up, and with 3.13, it does not.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With a function, it shows like this:

def f(a : something()):
   pass

print(f.__annotations__)

The re-formulation needs to use the nodes and create the updates to the __annotations__ as a function result. You need to check all uses of __annotations__ in the building code.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On second thought, that might be already correct for functions if "keys" and "values" are the ones from the function definition, not sure now, but it definitely doesn't do classes yet. And for empty annotations, I think it ought not to create a function, but instead end up passing a value. Or we check if that functions becomes a constant returner, in which case we ought to just pass its value or something, but that is optimization, which should for now maybe only result in TODOs. But a new function object per function, we don't want to hurt performance that much if we can help it in any way.

@kayhayen
Copy link
Member

kayhayen commented Nov 8, 2025

The current build is also failing for pre-3.14, I wouldn't add 3.14 to the matrix yet, but in the end when it all works, otherwise it's hiding those kinds of errors.

@ZeroIntensity ZeroIntensity marked this pull request as draft November 24, 2025 22:56
ZeroIntensity and others added 6 commits November 27, 2025 13:11
In 3.14, support for calling asyncio.get_event_loop() without a running
event loop was removed. Instead, use asyncio.new_event_loop().
We might be able to use asyncio.run(), but I won't touch that for now.
This will allow us to populate co_positions()
This reverts commit c30a008.
Nevermind we can just use the existing lnotab thing.
* Skipped this during rebase for simplicity.
Copy link
Member

@kayhayen kayhayen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job on this, I merged parts already to develop, leaving only the ones I have questions about here.

* a bad CPython release apparently and between 3.7.3 and 3.7.4 these have
* become runtime incompatible.
*
* On Python 3.14.0, the introduction of _Py_TriggerGC() also broke this.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please elaborae some more, not having these is very problematic for performance I believe

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In 3.14, _PyObject_GC_TRACK now has a call to _Py_TriggerGC, which is not exported.


def simpleFunction10():
asyncio.get_event_loop().run_until_complete(run())
asyncio.new_event_loop().run_until_complete(run())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's being tried there is to do minimal work under test, creating a new event loop (and releasing it?) is by definition not that. If we were to create our own on the outside and use that, no problem though.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, this raises an exception now, since get_event_loop no longer creates an event loop if one isn't running. It might work once I fix the exceptions in makeDiffable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, but we actually mean to succeed in using it.

new_node=result,
)
)
# For some reason, calling wrapExpressionWithSideEffects
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not keeping them (the side effects) is not acceptable though, I would need to see what you actually tried there to tell, but side effects need to be of course that of the dictionaries key/values, not the dictionary building itself.

return outer_body


def makeDeferredAnnotateFunction(provider, keys, values, source_ref):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this may actually be correct, and execute indeed delayed, and as such then be correct.

Does this pass any kinds of tests for it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this passes the tests with annotations in tests/basics.

@kayhayen
Copy link
Member

So, my goal is to get the merge-ready bits sorted out after I make a new pre-release out of factory later today.

@kayhayen
Copy link
Member

Seems the event loop taking is already leaking on 3.13, which is maybe kind of expected, I will try and move that to the outside the function, so it happens only once, max. I think we want to cover "run()" as a coroutine and not that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants