Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Generic TypeAlias (PEP 585) and Union Type (PEP 604) in generated .pyi stub files #129420

Open
wants to merge 61 commits into
base: gh/XuehaiPan/72/base
Choose a base branch
from

Conversation

XuehaiPan
Copy link
Collaborator

@XuehaiPan XuehaiPan commented Jun 24, 2024

#129001 (comment) is the motivation for the whole stack of PRs. In torch/__init__.py, torch._C.Type shadows from typing import Type, and there is no type stub for torch._C.Type in torch/_C/__init__.pyi. So we need to use from typing import Type as _Type. After enabling Generic TypeAlias (PEP 585) in the .pyi type stub files, we can use type instead of typing.Type or from typing import Type as _Type.


Stack from ghstack (oldest at bottom):


  • Generic TypeAlias (PEP 585): e.g. typing.List[T] -> list[T], typing.Dict[KT, VT] -> dict[KT, VT], typing.Type[T] -> type[T].
  • Union Type (PEP 604): e.g. Union[X, Y] -> X | Y, Optional[X] -> X | None, Optional[Union[X, Y]] -> X | Y | None.

Note that in .pyi stub files, we do not need from __future__ import annotations. So this PR does not violate issue #117449:

cc @ezyang @malfet @xuzhao9 @gramster @albanD @Skylion007

[ghstack-poisoned]
Copy link

pytorch-bot bot commented Jun 24, 2024

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129420

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 1 Unrelated Failure

As of commit 9d6f11e with merge base dbd6ada (image):

NEW FAILURES - The following jobs have failed:

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@@ -1088,26 +1087,26 @@ def replace_special_case(hint: str) -> str:
"def __init__(self, other: Tensor) -> None: ...",
f"def __init__(self, size: _size, *, {DEVICE_PARAM}) -> None: ...",
],
"as_subclass": ["def as_subclass(self, cls: _Type[S]) -> S: ..."],
"as_subclass": ["def as_subclass(self, cls: type[S]) -> S: ..."],
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use type[S] instead of from typing import Type as _Type and _Type[S]. Using _Type is hard to maintain while one can use Type[S] which is torch._C.Type[S].

@@ -3,25 +3,33 @@
# mypy: allow-untyped-defs

import builtins
from types import EllipsisType
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ellipsis in builtins is deprecated. Import EllipsisType = type(Ellipsis) explicitly.

SupportsIndex,
Tuple,
Type as _Type,
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See commnet for type[S] and _Type[S] above.

@@ -128,44 +115,40 @@ class Stream:
device: _device # The device of the stream

@overload
def __new__(self, device: Optional[DeviceLikeType] = None, *, priority: _int = 0) -> Stream: ...
def __new__(cls, device: DeviceLikeType | None = None, *, priority: _int = 0) -> Stream: ...
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self -> cls is auto fixed by ruff.

def _set_dispatch_mode(mode: TorchDispatchMode) -> None: ...
def _get_dispatch_stack_at(idx: _int) -> Any: ...
def _len_torch_dispatch_stack() -> _int: ...
def _activate_gpu_trace() -> None: ...

class _DisableTorchDispatch:
def __init__(self): ...
def __init__(self) -> None: ...
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

__init__(self, ...) -> None: is auto fixed by ruff.

@@ -42,7 +42,7 @@ def lookup(v: str) -> object:

def indent_lines(indent: str, v: Sequence[object]) -> str:
return "".join(
[indent + l + "\n" for e in v for l in str(e).splitlines()]
[(indent + l).rstrip() + "\n" for e in v for l in str(e).splitlines()]
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removes white spaces in empty lines.

tools/pyi/gen_pyi.py Outdated Show resolved Hide resolved
@XuehaiPan XuehaiPan added the suppress-bc-linter Suppresses the failures of API backward-compatibility linter (Lint/bc_linter) label Jun 24, 2024
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
XuehaiPan added a commit to XuehaiPan/pytorch that referenced this pull request Jul 20, 2024
… `.pyi` stub files

ghstack-source-id: e3f41c9fb6be223bbd43762b9b948af82a045f83
Pull Request resolved: pytorch#129420
[ghstack-poisoned]
XuehaiPan added a commit to XuehaiPan/pytorch that referenced this pull request Aug 1, 2024
… `.pyi` stub files

ghstack-source-id: e6675f1506bd806e9593539d656022af6878a6ad
Pull Request resolved: pytorch#129420
XuehaiPan added a commit to XuehaiPan/pytorch that referenced this pull request Aug 2, 2024
… `.pyi` stub files

ghstack-source-id: e6675f1506bd806e9593539d656022af6878a6ad
Pull Request resolved: pytorch#129420
[ghstack-poisoned]
XuehaiPan added a commit to XuehaiPan/pytorch that referenced this pull request Aug 2, 2024
… `.pyi` stub files

ghstack-source-id: a9a7750c6e6a5a26f744bc935ecdc008057c390d
Pull Request resolved: pytorch#129420
[ghstack-poisoned]
XuehaiPan added a commit to XuehaiPan/pytorch that referenced this pull request Aug 3, 2024
… `.pyi` stub files

ghstack-source-id: f0695b605d936f8a34731aebf684a3f7871964ef
Pull Request resolved: pytorch#129420
XuehaiPan added a commit to XuehaiPan/pytorch that referenced this pull request Aug 3, 2024
… `.pyi` stub files

ghstack-source-id: f0695b605d936f8a34731aebf684a3f7871964ef
Pull Request resolved: pytorch#129420
XuehaiPan added a commit to XuehaiPan/pytorch that referenced this pull request Aug 4, 2024
… `.pyi` stub files

ghstack-source-id: f0695b605d936f8a34731aebf684a3f7871964ef
Pull Request resolved: pytorch#129420
XuehaiPan added a commit to XuehaiPan/pytorch that referenced this pull request Aug 4, 2024
… `.pyi` stub files

ghstack-source-id: f0695b605d936f8a34731aebf684a3f7871964ef
Pull Request resolved: pytorch#129420
XuehaiPan added a commit to XuehaiPan/pytorch that referenced this pull request Aug 4, 2024
… `.pyi` stub files

ghstack-source-id: f0695b605d936f8a34731aebf684a3f7871964ef
Pull Request resolved: pytorch#129420
XuehaiPan added a commit to XuehaiPan/pytorch that referenced this pull request Aug 12, 2024
… `.pyi` stub files

ghstack-source-id: f0695b605d936f8a34731aebf684a3f7871964ef
Pull Request resolved: pytorch#129420
[ghstack-poisoned]
XuehaiPan added a commit to XuehaiPan/pytorch that referenced this pull request Aug 12, 2024
… `.pyi` stub files

ghstack-source-id: 3806e9533634a37bab9aa41666cf496083c29ee5
Pull Request resolved: pytorch#129420
XuehaiPan added a commit to XuehaiPan/pytorch that referenced this pull request Oct 11, 2024
… `.pyi` stub files

ghstack-source-id: 3806e9533634a37bab9aa41666cf496083c29ee5
Pull Request resolved: pytorch#129420
[ghstack-poisoned]
XuehaiPan added a commit to XuehaiPan/pytorch that referenced this pull request Oct 11, 2024
… `.pyi` stub files

ghstack-source-id: 42e720e3a04c30d4c9c4ede0ad3fa633888617e9
Pull Request resolved: pytorch#129420
[ghstack-poisoned]
XuehaiPan added a commit to XuehaiPan/pytorch that referenced this pull request Oct 11, 2024
… `.pyi` stub files

ghstack-source-id: 903346beee6f507c0a086c3a48cc5cd09e1e4133
Pull Request resolved: pytorch#129420
[ghstack-poisoned]
XuehaiPan added a commit to XuehaiPan/pytorch that referenced this pull request Oct 24, 2024
… `.pyi` stub files

ghstack-source-id: e971c2fc63604da189c8293ae9b7e6b956fd0cd2
Pull Request resolved: pytorch#129420
XuehaiPan added a commit to XuehaiPan/pytorch that referenced this pull request Oct 24, 2024
… `.pyi` stub files

ghstack-source-id: e971c2fc63604da189c8293ae9b7e6b956fd0cd2
Pull Request resolved: pytorch#129420
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
better-engineering Relatively self-contained tasks for better engineering contributors ciflow/inductor ciflow/trunk Trigger trunk jobs on your pull request module: typing Related to mypy type annotations open source release notes: dataloader release notes category skip-pr-sanity-checks suppress-bc-linter Suppresses the failures of API backward-compatibility linter (Lint/bc_linter) topic: not user facing topic category
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants