PEP 563 (i.e., `from future import annotations`) Is Deprecated: "Ding Dong, the Witchy PEP Is Dead." #299

leycec · 2023-11-01T05:21:48Z

leycec
Nov 1, 2023
Maintainer

Spooky Halloween update intensifies. This is an official message of the Beartype Broadcasting Service. Trust no other.

Long has @leycec warned ye of the ill tidings of PEP 563, the standard previously known as from __future__ import annotations. Yet long have ye ignored @leycec.

Verily, for did @leycec not counsel all the corporations and academic institutions of the land that some future version of Python to come would deprecate PEP 563 and therefore thy entire fragile house of codebase cards built on the leaning back of PEP 563? Verily, that ill-fated day of fetid sorrow has come. Shudder as one, ye who reap what ye have sown! </cackles>

Yay. For @leycec quoths the ravenous PEP 649: ^"Nevermore."

If accepted, this PEP would supersede PEP 563, and PEP 563’s behavior would be deprecated and eventually removed.

Guess ye what? That PEP was accepted.

Can We Pretend It's Not Halloween for One Hot Minute?

Fine, fine. You're not wrong. Even I got tired of playing the biblical PEP prophet Leycecjah there. 😅

Here's the skinny: ^{...spooky idioms, thy name is English Language}

Python 3.13 officially accepted PEP 649.
PEP 649 officially supercedes PEP 563.
PEP 563 is therefore officially deprecated.
PEP 563 will therefore be eventually removed.
But from __future__ import annotations unconditionally enables PEP 563.
Then either:
- The entirety of from __future__ import annotations is officially deprecated and will be eventually removed or...
- The subset of from __future__ import annotations that specifically pertains to PEP 563 is officially deprecated and will be eventually removed.

It's unclear which of these is the case. What is clear is that from __future__ import annotations is now a volatile and highly explosive compound.

EDIT: The following paragraph is all lies and best ignored. Still, it made me chuckle quietly and I miss it already. Regardless of whether from __future__ import annotations is going away in entirety or merely in part, backward compatibility will be broken. Something you currently rely upon is about to go on an all-you-can-drink bender in Tokyo's infamous Kabukicho red light district. (Been there, done that, can't remember a thing.) If your codebase or a codebase you love is currently prefacing literally every Python module with from __future__ import annotations, that codebase is now primed to viciously explode under Python ≥ 3.13.

Do the right thing today, tomorrow, or next year at the latest. Just say "Let's pretend this never happened" to from __future__ import annotations. GitHubbers don't let GitHubbers from __future__ import annotations. Not even once.

B-b-but You're Breakin' Mah Heart, Here!

@beartype feels your pain, but fails to sympathize. It's Halloween, after all. No sympathy for the devil.

Alright. Alright already! Although devilish, you may now be wondering: "But what are we supposed to do now?" You have two choices – both of which suck:

Do absolutely nothing. Although new codebases should avoid from __future__ import annotations for all of the voluminous reasons enumerated below, existing codebases like PyTorch that have already gone all-in on from __future__ import annotations should avoid breaking their own backs. Your code is fine and will continue to work as is in perpetuity, because Python devs are less dumber than we think. To wit, those voluminous reasons for avoiding from __future__ import annotations in new codebases are:
- It's extremely inefficient. See below.
- It fundamentally breaks the Python parser. See below.
- It fundamentally breaks everything that is not @beartype. See below.
Require Python ≥ 3.12 and use PEP 695-compliant type syntax everywhere. type is the new typing Bad Boy. It's PEP 563, except it actually works. Why? Because PEP 695 explicitly limits the scope of PEP 563-style behaviour to just a single statement (rather than the entire friggin' module). That statement is the new type syntax. This is PEP 695. This is type:

# "type" syntax implicitly instantiates type variables. "T" does *NOT* need to be explicitly declared!
type ListOrSet[T] = list[T] | set[T]

# "type" syntax explicitly permits forward references to undeclared types. "Animal" has yet to be declared!
type AnimalOrVegetable = Animal | "Vegetable"

# "type" syntax explicitly permits recursive type hints, too. Behold! Mega-optic-recursion!
type RecursiveList[T] = T | list[RecursiveList[T]]

Support Python < 3.12 by using... strings. Yeah. Strings. I know, I know. I weep as well. But if your userbase is unwilling to follow you into the pure golden land of Python ≥ 3.12, then you have no recourse but to explicitly stringify all of the forward references you previously let PEP 563 stringify for you. Repeat after Pappa Leycec, everybody: "Explicit is better than implicit:" ^{Pretty sure Pappa Leycec is a jerk.}

from typing import TypeVar, Union

# Python < 3.12 sure sucks in hindsight, doesn't it? I blame someone who is not me.
AnimalOrVegetable = Union["Animal", "Vegetable"]

# Python < 3.12: The Suckage Continues.
T = TypeVar('T')
RecursiveList[T] = Union[T, list['[RecursiveList[T]']]

To Those Who Are About to Cry, @beartype Pities You

Relevant downstream consumers severely exposed to this deprecation include:

@justinchuby: Nevermind, bro. You're all good. Carry on! @beartype still got your back. PyTorch. The Gods weep! @justinchuby, I regretfully summon you to inform you that a badness is on the horizon. I've grepped over 100 instances of from __future__ import annotations throughout the PyTorch codebase. Unwinding that is not going to prove trivial. On the contrary, it's gonna be boring; it's gonna take forever; it's gonna cause premature balding. In short, it's gonna hurt. Let us all be suddenly thankful we are not the fall guy for that one.
@posita, because memes are happening.
@wesselb, for similar reasons.
@eli-schwartz, because of this comment here:

I'm not entirely sure I understand the problem here -- the concept behind from __future__ import annotations is essentially "pretend as if every single annotation, bar none, was quoted, but without actually quoting them". Why is it bad to use this...

...because that is now officially deprecated is what I am saying. Even if PEP 563 was still alive, however, enabling PEP 563 via from __future__ import annotations would still be the wrong thing to do. Why? Because PEP 563:

Is extremely inefficient. But efficiency was the whole point of PEP 563! So PEP 563 fails at even its core mandate. PEP 563 forces the Python interpreter to parse every type hint from a string into an abstract syntax tree (AST) node and then unparse that same node back into a string, which then forces everything at runtime to then re-parse that same string back into a facsimile of the original AST node. It's absurd, frankly – but somebody who hates Python thought that was a really great idea. PEP 563 was never about making forward references look "syntactically clean," although that ended up being its primary use case. Horrifying.
Fundamentally breaks the Python parser. Seriously. It does. I actually have a swath of PEP 563 unit tests I disable under both CPython ≥ 3.10 and PyPy ≥ 3.8. Why? Because those unit tests only pass under older CPython versions. Newer CPython and PyPy versions actually broke PEP 563 in various edge cases that somebody somewhere cares about (e.g., closures, nested callables).
Fundamentally breaks everything that is not @beartype. Seriously. It does. I am pretty sure that @beartype is the only Python package that fully complies with PEP 563. In hindsight, I kinda regret spending so much time on that. PEP 563 is dead now and I want my wasted man-hours back. Weep for me, emoji! 😭

... but perfectly fine to manually quote them?

...because that will never be deprecated. @leycec guarantees it. ^{← never say something like this} 😬 🤯 💥

eli-schwartz · 2023-11-01T11:21:41Z

eli-schwartz
Nov 1, 2023

Then either:
* The entirety of `from __future__ import annotations` is **officially deprecated and will be eventually removed** _or_...
* The subset of `from __future__ import annotations` that specifically pertains to PEP 563 is **officially deprecated and will be eventually removed.**
It's unclear which of these is the case. What is clear is that from __future__ import annotations is now a volatile and highly explosive compound. Regardless of whether from __future__ import annotations is going away in entirety or merely in part, backward compatibility will be broken.

I'm highly skeptical that the Python Core Developers will break backwards compatibility "just like that". And at the very least, the either-or in this can be clarified: it is against policy for an object located in __future__ to be deleted, so importing it will never break. The effects of importing it resulting in automatic stringization of annotations will, however, be deprecated and "eventually" removed (not in 3.13, apparently) which means that from __future__ import annotations will become dead code that does nothing.

The new approach to annotations is supposed to be more or less backwards compatible. The details are discussed at https://peps.python.org/pep-0649/#backwards-compatibility-with-pep-563-semantics

Is extremely inefficient. But efficiency was the whole point of PEP 563! So PEP 563 fails at even its core mandate. PEP 563 forces the Python interpreter to parse every type hint from a string into an abstract syntax tree (AST) node and then unparse that same node back into a string, which then forces everything at runtime to then re-parse that same string back into a facsimile of the original AST node.

Do you have a citation for that? Because it really should not be creating code objects -- it is documented to not do so -- so if it were really being inefficient like that then that would be a CPython 3.7 - 3.12 bug and I'd have expected someone to report it by now (possibly you).

0 replies

eli-schwartz · 2023-11-01T11:59:56Z

eli-schwartz
Nov 1, 2023

Alright. Alright already! Although devilish, you may now be wondering: "But what are we supposed to do now?" You have two choices – both of which suck:

Require Python ≥ 3.12 and use PEP 695-compliant type syntax everywhere. type is the new typing Bad Boy. It's PEP 563, except it actually works. Why? Because PEP 695 explicitly limits the scope of PEP 563-style behaviour to just a single statement (rather than the entire friggin' module). That statement is the new type syntax. This is PEP 695. This is type:

I'm unsure how you formed this conclusion. It is totally unrelated to forward annotations. The new type syntax does for the typing module contents TypeVar, Generic, TypeAlias what PEP 585: Type Hinting Generics In Standard Collections, did for List, Dict, Tuple, Set, FrozenSet, Type. That is all. Before and after PEP 695, you may use stringified annotations or not, as you choose, but you need to do them both via importing the typing module rather than via syntax. After the PEP, you may use it with builtins that look prettier and shorter and don't require an import. Semantically the same results.

Support Python < 3.12 by using... strings. Yeah. Strings. I know, I know. I weep as well. But if your userbase is unwilling to follow you into the pure golden land of Python ≥ 3.12, then you have no recourse but to explicitly stringify all of the forward references you previously let PEP 563 stringify for you. Repeat after Pappa Leycec, everybody: "Explicit is better than implicit:" ^{Pretty sure Pappa Leycec is a jerk.}

The recourse is to simply not change anything, which continues to do precisely what it always did on python < 3.12 -- and also for that matter python 3.12 itself. It will either continue to do what it used to do, or simply do nothing and leave you with PEP 649 deferred annotations by default on versions of python that dropped all support for PEP 563 but do support 649.

One could of course argue that explicit is better than implicit, but this is dogma, not an implementation requirement.

0 replies

leycec · 2023-11-02T06:35:51Z

leycec
Nov 2, 2023
Maintainer Author

I'm highly skeptical that the Python Core Developers will break backwards compatibility "just like that".

No one is breaking backwards compatibility "just like that." Not even @beartype does that – and we play fast and loose around here. This is how it works:

First, a thing is deprecated.
Then, that thing is removed.

That's how deprecations work. That's what deprecations mean. You get a warning that backward compatibility will be broken. Then you get the error, which is backward compatibility being broken.

PEP 649 has warned you that PEP 563 has been officially deprecated. You are, of course, free to ignore that deprecation. Ignoring deprecations is obviously unwise. Nonetheless, freedom means the freedom to faceplant into the yawning cliff that is now looming in front you. I exercise that freedom by avoiding the cliff. But you do you, you know?

I'm not here to argue with reality, @eli-schwartz. I'm here to describe reality. It's what I do. It's an Aspy thing.

And at the very least, the either-or in this can be clarified: it is against policy for an object located in __future__ to be deleted, so importing it will never break.

They should break the import. They're breaking backward compatibility. So, they should explicitly inform everyone they're breaking your code by removing from __future__ import annotations entirely. Explicit is better than implicit.

That said... I don't particularly care. I'm unaffected either way. I never used from __future__ import annotations – not even once. That was obviously always a bad idea. I'm not exposed to this in any of the codebases I maintain. I have no strong opinions here. CPython devs will be CPython devs. </shrug>

The new approach to annotations is supposed to be more or less backwards compatible.

Bro, it's not. You seem really argumentative. Why are you really argumentative?

PEP 649 explicitly told you that PEP 563 is now officially deprecated. The new approach to annotations thus fundamentally breaks backwards compatibility with from __future__ import annotations. That's a good thing. That's also not up for debate. Seriously. Can we move on to the part of the useful conversation where you accept that PEP 563 has been deprecated? Because it has. 😮‍💨

Do you have a citation for that? Because it really should not be creating code objects -- it is documented to not do so -- so if it were really being inefficient like that then that would be a CPython 3.7 - 3.12 bug and I'd have expected someone to report it by now (possibly you).

Wow. So we have confirmed you are here to argue. Kinda disappointed, to be honest. Flame wars are strictly verboten here. We all got better things to do.

Likewise, nobody said nuffin' about "creating code objects." But... thanks for putting words in my digital mouth that hopefully has less broken teeth than my physical mouth, I guess? I said that "PEP 563 forces the Python interpreter to parse every type hint from a string into an abstract syntax tree (AST) node and then unparse that same node back into a string," which is exactly what Python does when you enable PEP 563 via from __future__ with annotations.

Seriously. Python parses and then unparses itself under PEP 563. That's well-known. That's how it works. That's well-documented everywhere. See also Python's official developer handbook:

Python/ast_unparse.c will need changes to unparse AST involved with the grammar change (“unparsing” is used to turn annotations into strings per PEP 563).

How did you think any of that crusty black magic worked, anyway? Actually, belay that question. I increasingly don't want to know – because you are here to argue about everything. It saddens us all.

I'm unsure how you formed this conclusion. It is totally unrelated to forward annotations.

Seriously. Do you really have any idea what you are talking about or do you just spout random inanities as if that makes them true? PEP 695-compliant type syntax creates a PEP 649-compliant annotation scope, which then enables declaration of unquoted forward references. It's literally in the friggin' PEP, bro. This is in PEP 695:

# A type alias that includes a forward reference
type AnimalOrVegetable = Animal | "Vegetable"

Verily, and I yet again quote PEP 695:

The value of the type alias is evaluated lazily (see Lazy Evaluation below).

Do you know what "evaluated lazily" means here? It means that the value of the type alias created by a type statement is deferred until the __value__ dunder attribute of that alias is accessed at runtime by a runtime type-checker. Verily, and I yet again quote PEP 695:

This PEP introduces three new contexts where expressions may occur that represent static types: TypeVar bounds, TypeVar constraints, and the value of type aliases. These expressions may contain references to names that are not yet defined.

What do you think a "reference to a name that is not yet defined" is? That's right. It's a forward reference.

If you want an unquoted forward reference in Python ≥ 3.12 without exposing yourself to deprecations, you have to use PEP 695. There is no other way. PEP 563 is officially deprecated. Can we please move on? Arguing with reality isn't a good look. Please at least read the PEP you are arguing about before arguing about that PEP.

I sigh with XML yet again. </double_sigh>

The recourse is to simply not change anything...

Code that contains implicit forward references (i.e., type hints referencing undeclared objects under from __future__ import annotations) will break. When? I dunno. Sometime. Do I look like a PEP genie? ^{...don't answer that.}

This means code like this:

from __future__ import annotations
def muh_func() -> MuhClass: pass  # <-- dis about to break, yo
class MuhClass(object): pass

That will break. "Simply not changing anything" is not an option. Do nothing and all code with implicit forward references will die. Ignore the deprecation of PEP 563 and all code with implicit forward references will die. This is a hard-and-fast guarantee.

Ignoring a bad thing coming does not prevent that bad thing from coming. Trainwreck emoji, I summon thee to save @eli-schwartz from himself! 🚋 💣 💥

0 replies

leycec · 2023-11-02T07:13:11Z

leycec
Nov 2, 2023
Maintainer Author

Wait. I get it. I get what you are saying, @eli-schwartz – and you're absolutely right about PEP 649 preserving backward compatibility in the way that matters most. I was blinded by a growing sense of futility and incoherent frustration at your argumentative communication style, which could really use some work. Seriously. Please meditate.

But... you're almost certainly right about this:

from __future__ import annotations
def muh_func() -> MuhClass: pass
class MuhClass(object): pass

That will always work – even after support for PEP 563 is officially removed. How? Because, exactly as you suggest:

from __future__ import annotations will simply reduce to a silent noop.
Annotations will be unconditionally evaluated under a PEP 695- and 649-compliant annotation scope. Ergo, all type hints will globally support unquoted forward references – including type hints declared inside a type statement as well as type hints annotating classes and callables.

Gotcha. That makes total sense. That is actually fairly clever. And that is really the way it absolutely must be for sanity to prevail across the Python ecosystem, right? Right. I'll amend my original post to admit that doing absolutely nothing is both permissible and advisable for codebases that have already committed to PEP 563.

Phew. You dodged a gruesome bullet there, PyTorch. We can all breath easily now. AI has been preserved! 🤖

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

PEP 563 (i.e., `from future import annotations`) Is Deprecated: "Ding Dong, the Witchy PEP Is Dead." #299

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 4 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Uh oh!

PEP 563 (i.e., from __future__ import annotations) Is Deprecated: "Ding Dong, the Witchy PEP Is Dead." #299

Uh oh!

Uh oh!

leycec Nov 1, 2023 Maintainer

Can We Pretend It's Not Halloween for One Hot Minute?

B-b-but You're Breakin' Mah Heart, Here!

To Those Who Are About to Cry, @beartype Pities You

Replies: 4 comments

Uh oh!

eli-schwartz Nov 1, 2023

Uh oh!

Uh oh!

eli-schwartz Nov 1, 2023

Uh oh!

Uh oh!

leycec Nov 2, 2023 Maintainer Author

Uh oh!

Uh oh!

leycec Nov 2, 2023 Maintainer Author

PEP 563 (i.e., `from future import annotations`) Is Deprecated: "Ding Dong, the Witchy PEP Is Dead." #299

leycec
Nov 1, 2023
Maintainer

eli-schwartz
Nov 1, 2023

eli-schwartz
Nov 1, 2023

leycec
Nov 2, 2023
Maintainer Author

leycec
Nov 2, 2023
Maintainer Author