-
-
Notifications
You must be signed in to change notification settings - Fork 749
Python 3.14 support #3662
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Python 3.14 support #3662
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We failed to fetch the diff for pull request #3662
You can try again by commenting this pull request with @sourcery-ai review, or contact us for help.
In 3.14, support for calling asyncio.get_event_loop() without a running event loop was removed. Instead, use asyncio.new_event_loop(). We might be able to use asyncio.run(), but I won't touch that for now.
Reviewer's GuideThis PR implements Python 3.14 support by updating generator internals and opcode mappings for the new interpreter frame API, introducing a deferred annotation mechanism with annotate hooks in both AST transforms and runtime, adjusting built-in error messages and lookup orders to match CPython 3.14 behavior, refining the build configuration for the Python 3.14 HACL library, and updating tests to use the new asyncio event loop API. Sequence diagram for annotate deferred annotation retrieval (Python 3.14+)sequenceDiagram
participant User
participant Nuitka_FunctionObject
participant DeferredAnnotateFunction
User->>Nuitka_FunctionObject: Access __annotations__
alt m_annotate is set
Nuitka_FunctionObject->>DeferredAnnotateFunction: Call __annotate__(format=1)
DeferredAnnotateFunction-->>Nuitka_FunctionObject: Return dict of annotations
Nuitka_FunctionObject-->>User: Return dict
else m_annotate is not set
Nuitka_FunctionObject-->>User: Return empty dict
end
Sequence diagram for context manager protocol lookup order (Python 3.14+)sequenceDiagram
participant Nuitka
participant ContextManagerObject
Nuitka->>ContextManagerObject: Lookup __exit__
alt __exit__ found
Nuitka->>ContextManagerObject: Lookup __enter__
alt __enter__ found
Nuitka-->>Nuitka: Proceed with context manager
else __enter__ missing
Nuitka-->>Nuitka: Raise error (missing __enter__)
end
else __exit__ missing
Nuitka-->>Nuitka: Raise error (missing __exit__)
end
Class diagram for deferred annotation mechanism in Nuitka_FunctionObject (Python 3.14+)classDiagram
class Nuitka_FunctionObject {
PyObject *m_annotations
PyObject *m_annotate
PyObject *m_qualname
...
}
Nuitka_FunctionObject : +get_annotations()
Nuitka_FunctionObject : +set_annotations(value)
Nuitka_FunctionObject : +get_annotate()
Nuitka_FunctionObject : +set_annotate(value)
Nuitka_FunctionObject : +clone()
Nuitka_FunctionObject : +tp_dealloc()
Nuitka_FunctionObject <|-- DeferredAnnotateFunction
class DeferredAnnotateFunction {
+__call__(format)
returns dict or raises NotImplementedError
}
Class diagram for AST transform: deferred annotation function creationclassDiagram
class ExpressionFunctionBody {
provider
name
code_object
flags
parameters
...
}
class ExpressionFunctionRef {
function_body
source_ref
}
class ExpressionFunctionCreation {
function_ref
defaults
kw_defaults
annotations
source_ref
}
ExpressionFunctionCreation --> ExpressionFunctionRef
ExpressionFunctionRef --> ExpressionFunctionBody
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey there - I've reviewed your changes and they look great!
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location> `nuitka/build/include/nuitka/compiled_function.h:69-71` </location>
<code_context>
PyObject *m_annotations;
#endif
+#if PYTHON_VERSION >= 0x300
+ PyObject *m_annotate;
+#endif
+
</code_context>
<issue_to_address>
**suggestion:** m_annotate is defined for all Python 3, but only used for 3.14+.
Restrict the definition of m_annotate to Python 3.14+ to improve clarity and prevent confusion in earlier versions.
```suggestion
#if PYTHON_VERSION >= 0x30E0
PyObject *m_annotate;
#endif
```
</issue_to_address>
### Comment 2
<location> `nuitka/nodes/DictionaryNodes.py:236-245` </location>
<code_context>
+ if python_version >= 0x3E0:
</code_context>
<issue_to_address>
**issue:** Error message for unhashable dict keys updated for Python 3.14+, but side effects handling is inconsistent.
Please ensure side effect handling is consistent for both error message branches to avoid potential loss of side effects.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| if python_version >= 0x3E0: | ||
| result = makeRaiseExceptionExpressionFromTemplate( | ||
| exception_type="TypeError", | ||
| template="cannot use '%s' as a dict key (unhashable type: '%s')", | ||
| template_args=( | ||
| makeExpressionAttributeLookup( | ||
| expression=key.getTypeValue(), | ||
| attribute_name="__name__", | ||
| source_ref=key.source_ref, | ||
| ), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
issue: Error message for unhashable dict keys updated for Python 3.14+, but side effects handling is inconsistent.
Please ensure side effect handling is consistent for both error message branches to avoid potential loss of side effects.
kayhayen
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The __annotations__ change for the function level could be useful, I need to check it out, for classes it will be a lot more annoying to make.
| } | ||
| function->m_annotations = CALL_FUNCTION_WITH_SINGLE_ARG(tstate, function->m_annotate, _PyLong_GetOne()); | ||
| if (function->m_annotations == NULL) { | ||
| return NULL; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will that function be executed each time when it throws an error over and over? Is that compatible?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems so, you can try this script:
def test(x):
pass
def evil(format):
raise RuntimeError("evil")
test.__annotate__ = evil
for _ in range(3):
try:
print(test.__annotations__)
except RuntimeError as error:
print(error)| // For simplicity's sake, the annotations parameter doubles as the __annotate__ | ||
| // parameter on 3.14+ | ||
| assert(annotations == NULL || PyCallable_Check(annotations)); | ||
| result->m_annotations = NULL; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
However, m_annotations seems now unused with 3.14?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, it's used as a cache. Once __annotate__ is called, the result is stored in __annotations__, so the annotations don't have to be re-evaluated every time.
| const uint8_t Nuitka_PyOpcode_Deopt[256] = { | ||
| #if PYTHON_VERSION >= 0x3d0 | ||
| #if PYTHON_VERSION >= 0x3e0 | ||
| [121] = 121, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Curious, why are there plain numbers used in CPython now?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unknown opcodes need to be marked as de-opting to themselves to support some third-party JIT compilers. See python/cpython#128045. It's probably fine to omit it for Nuitka, but I don't know what will happen if we do.
| ) | ||
| # On 3.14+, annotations are deferred by default. | ||
| if python_version >= 0x3E0: | ||
| return makeDeferredAnnotateFunction( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are you saying, that all this is doing, is to create a function that returns the dictionary, just later? I thought the actual code of the annotations is to be executed delayed.
def f():
x : something()
f()
f.__annotations__
I was assuming this will not crash when calling f().
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, that's ignored in all versions, but this is showing it:
class C():
x : something()
print(C.__annotations__)
With 3.14, this crashes when .__annotations__ is looked up, and with 3.13, it does not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With a function, it shows like this:
def f(a : something()):
pass
print(f.__annotations__)
The re-formulation needs to use the nodes and create the updates to the __annotations__ as a function result. You need to check all uses of __annotations__ in the building code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On second thought, that might be already correct for functions if "keys" and "values" are the ones from the function definition, not sure now, but it definitely doesn't do classes yet. And for empty annotations, I think it ought not to create a function, but instead end up passing a value. Or we check if that functions becomes a constant returner, in which case we ought to just pass its value or something, but that is optimization, which should for now maybe only result in TODOs. But a new function object per function, we don't want to hurt performance that much if we can help it in any way.
|
The current build is also failing for pre-3.14, I wouldn't add 3.14 to the matrix yet, but in the end when it all works, otherwise it's hiding those kinds of errors. |
This reverts commit c30a008. Nevermind we can just use the existing lnotab thing.
In 3.14, support for calling asyncio.get_event_loop() without a running event loop was removed. Instead, use asyncio.new_event_loop(). We might be able to use asyncio.run(), but I won't touch that for now.
This will allow us to populate co_positions()
This reverts commit c30a008. Nevermind we can just use the existing lnotab thing.
* Skipped this during rebase for simplicity.
kayhayen
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good job on this, I merged parts already to develop, leaving only the ones I have questions about here.
| * a bad CPython release apparently and between 3.7.3 and 3.7.4 these have | ||
| * become runtime incompatible. | ||
| * | ||
| * On Python 3.14.0, the introduction of _Py_TriggerGC() also broke this. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please elaborae some more, not having these is very problematic for performance I believe
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In 3.14, _PyObject_GC_TRACK now has a call to _Py_TriggerGC, which is not exported.
|
|
||
| def simpleFunction10(): | ||
| asyncio.get_event_loop().run_until_complete(run()) | ||
| asyncio.new_event_loop().run_until_complete(run()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's being tried there is to do minimal work under test, creating a new event loop (and releasing it?) is by definition not that. If we were to create our own on the outside and use that, no problem though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unfortunately, this raises an exception now, since get_event_loop no longer creates an event loop if one isn't running. It might work once I fix the exceptions in makeDiffable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, but we actually mean to succeed in using it.
| new_node=result, | ||
| ) | ||
| ) | ||
| # For some reason, calling wrapExpressionWithSideEffects |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not keeping them (the side effects) is not acceptable though, I would need to see what you actually tried there to tell, but side effects need to be of course that of the dictionaries key/values, not the dictionary building itself.
| return outer_body | ||
|
|
||
|
|
||
| def makeDeferredAnnotateFunction(provider, keys, values, source_ref): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe this may actually be correct, and execute indeed delayed, and as such then be correct.
Does this pass any kinds of tests for it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, this passes the tests with annotations in tests/basics.
|
So, my goal is to get the merge-ready bits sorted out after I make a new pre-release out of factory later today. |
|
Seems the event loop taking is already leaking on 3.13, which is maybe kind of expected, I will try and move that to the outside the function, so it happens only once, max. I think we want to cover "run()" as a coroutine and not that. |
Also fix ordering of __firstlineno__ assignment.
What does this PR do?
Adds support for Python 3.14 by running the test suite until it all passes.
Why was it initiated? Any relevant Issues?
PR Checklist
developbranch../bin/autoformat-nuitka-source.Running the Tests. There are GitHubActions tests that cover the most important things however, and you are welcome to rely on those,
but they might not cover enough.
Summary by Sourcery
Add compatibility for Python 3.14 by updating generated C runtime integration, annotation handling in compiled functions, build scripts, and tests.
Enhancements:
Build:
Tests: