Repository navigation
Expand file tree
/
Copy pathpep-0661.rst
More file actions
582 lines (417 loc) · 23.4 KB
/
Copy pathpep-0661.rst
File metadata and controls
582 lines (417 loc) · 23.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
PEP: 661
Title: Sentinel Values
Author: Tal Einat <tal@python.org>, Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-661-sentinel-values/9126
Status: Final
Type: Standards Track
Created: 06-Jun-2021
Python-Version: 3.15
Post-History: `20-May-2021 <https://discuss.python.org/t/sentinel-values-in-the-stdlib/8810>`__, `06-Jun-2021 <https://discuss.python.org/t/pep-661-sentinel-values/9126>`__
Resolution: `23-Apr-2026 <https://discuss.python.org/t/pep-661-sentinel-values/9126/337>`__
.. canonical-doc:: :external+py3.15:class:`sentinel`
Abstract
========
Unique placeholder values, commonly known as "sentinel values", are common in
programming. They have many uses, such as for:
* Default values for function arguments, for when a value was not given::
def foo(value=None):
...
* Return values from functions when something is not found or unavailable::
>>> "abc".find("d")
-1
* Missing data, such as NULL in relational databases or "N/A" ("not
available") in spreadsheets
Python has the special value ``None``, which is intended to be used as such
a sentinel value in most cases. However, sometimes an alternative sentinel
value is needed, usually when it needs to be distinct from ``None`` since
``None`` is a valid value in that context. Such cases are common enough that
several idioms for implementing such sentinels have arisen over the years, but
uncommon enough that there hasn't been a clear need for standardization.
However, the common implementations, including some in the stdlib, suffer from
several significant drawbacks.
This PEP proposes adding a built-in class for defining sentinel values, to
be used in the stdlib and made publicly available to all Python code.
Sentinels can be defined in Python with the ``sentinel()`` built-in class,
and in C with the ``PySentinel_New()`` C API function.
Note: Changing all existing sentinels in the stdlib to be implemented this
way is not deemed necessary, and whether to do so is left to the discretion
of the maintainers.
Motivation
==========
In May 2021, a question was brought up on the python-dev mailing list
[1]_ about how to better implement a sentinel value for
``traceback.print_exception``. The existing implementation used the
following common idiom::
_sentinel = object()
However, this object has an uninformative and overly verbose repr, causing the
function's signature to be overly long and hard to read::
>>> help(traceback.print_exception)
Help on function print_exception in module traceback:
print_exception(exc, /, value=<object object at
0x000002825DF09650>, tb=<object object at 0x000002825DF09650>,
limit=None, file=None, chain=True)
Additionally, two other drawbacks of many existing sentinels were brought up
in the discussion:
1. Some do not have a distinct type, hence it is impossible to define clear
type signatures for functions with such sentinels as default values.
2. They behave unexpectedly after being copied, due to a separate instance
being created and thus comparisons using ``is`` failing. Some common
sentinel idioms have similar problems after being pickled and unpickled.
In the ensuing discussion, Victor Stinner supplied a list of currently used
sentinel values in the Python standard library [2]_. This showed that the
need for sentinels is fairly common, that there are various implementation
methods used even within the stdlib, and that many of these suffer from at
least one of the three above drawbacks.
The discussion did not lead to any clear consensus on whether a standard
implementation method is needed or desirable, whether the drawbacks mentioned
are significant, nor which kind of implementation would be good. The author
of this PEP created an issue on bugs.python.org (now a GitHub issue [3]_)
suggesting options for improvement, but that focused on only a single
problematic aspect of a few cases, and failed to gather any support.
A poll [4]_ was created on discuss.python.org to get a clearer sense of
the community's opinions. After nearly two weeks, significant further,
discussion, and 39 votes, the poll's results were not conclusive. 40% had
voted for "The status-quo is fine / there’s no need for consistency in
this", but most voters had voted for one or more standardized solutions.
Specifically, 37% of the voters chose "Consistent use of a new, dedicated
sentinel factory / class / meta-class, also made publicly available in the
stdlib".
With such mixed opinions, this PEP was created to facilitate making a decision
on the subject.
While working on this PEP, iterating on various options and implementations
and continuing discussions, the author has come to the opinion that a simple,
good implementation available in the standard library would be worth having,
both for use in the standard library itself and elsewhere.
Rationale
=========
The criteria guiding the chosen implementation were:
1. The sentinel objects should behave as expected by a sentinel object: When
compared using the ``is`` operator, it should always be considered
identical to itself but never to any other object.
2. Creating a sentinel object should be a simple, straightforward one-liner.
3. It should be simple to define as many distinct sentinel values as needed.
4. The sentinel objects should have a clear and short repr.
5. It should be possible to use clear type signatures for sentinels.
6. The sentinel objects should behave correctly after copying, and sentinels
should have predictable behavior when pickled and unpickled.
7. Such sentinels should work when using CPython 3.x and PyPy3, and ideally
also with other implementations of Python.
8. As simple and straightforward as possible, in implementation and especially
in use. Avoid this becoming one more special thing to learn when learning
Python. It should be easy to find and use when needed, and obvious enough
when reading code that one would normally not feel a need to look up its
documentation.
With so many uses in the Python standard library [2]_, it would be useful to
have an implementation available to the standard library, since the stdlib
cannot use implementations of sentinel objects available elsewhere (such as
the ``sentinels`` [5]_ or ``sentinel`` [6]_ PyPI packages).
After researching existing idioms and implementations, and going through many
different possible implementations, the design below was chosen to meet these
criteria while keeping the API and implementation small (see
`Reference Implementation`_).
To support common use cases, we add two customization points:
- The ``repr`` argument can be used to customize the ``repr()`` of sentinels. Many
pre-existing sentinels in the standard library use a custom repr like ``<missing>``,
and this argument allows us to convert these sentinels to the new API while
preserving this behavior.
- The ``__module__`` attribute of sentinels is writable, allowing users to
control the module used for pickling in some unusual cases, such as sentinels
created through ``exec()``. This aligns with the behavior of other built-in
types with a ``__module__`` attribute, including classes, functions, ``TypeVar``
objects, and (as of Python 3.15) type aliases.
Specification
=============
A new built-in callable named ``sentinel`` will be added.
>>> MISSING = sentinel('MISSING')
>>> MISSING
MISSING
``sentinel()`` takes a single required positional-only argument, ``name``, which must
be a ``str``, and an optional keyword-only argument, ``repr``.
Passing a non-string as the ``name`` raises ``TypeError``. The name is used as
the sentinel's name. The value of the ``repr`` argument is used for the ``str()`` and ``repr()``
of the sentinel object, if given. If not given, the name is used instead.
Sentinel objects have two public attributes:
* ``__name__`` is the sentinel's name.
* ``__module__`` is the name of the module where ``sentinel()`` was called.
This attribute is writable.
``sentinel`` may not be subclassed.
Each call to ``sentinel(name)`` returns a new sentinel object. If a sentinel
is needed in more than one place, it should be assigned to a variable and that
same object should be reused explicitly, just as with the common
``MISSING = object()`` idiom::
MISSING = sentinel('MISSING')
def read_value(default=MISSING):
...
Checking if a value is such a sentinel *should* be done using the ``is``
operator, as is recommended for ``None``. Equality checks using ``==`` will
also work as expected, returning ``True`` only when the object is compared
with itself. Identity checks such as ``if value is MISSING:`` should usually
be used rather than boolean checks such as ``if value:`` or ``if not value:``.
Sentinel objects are "truthy", i.e. boolean evaluation will result in
``True``. This parallels the default for arbitrary classes, as well as the
boolean value of ``Ellipsis``. This is unlike ``None``, which is "falsy".
Creating a copy of a sentinel object, such as by using ``copy.copy()`` or by
``copy.deepcopy()``, will return the same object.
Sentinels importable from their defining module by name preserve their identity
when pickled and unpickled, using the standard pickle mechanism for named
singletons. When ``sentinel()`` creates a sentinel, it records the calling
module as the sentinel's ``__module__`` attribute. Pickling records the
sentinel by module and name. Unpickling then imports the module and retrieves
the sentinel by name, so the following round trip preserves identity::
MISSING = sentinel('MISSING')
assert pickle.loads(pickle.dumps(MISSING)) is MISSING
Sentinels that are not importable by module and name, such as sentinels
created in a local scope and not assigned to a matching module global or class
attribute, are not picklable.
The repr of a sentinel object is the ``name`` passed to ``sentinel()``. No
implicit module qualification is added. If a qualified repr is desired, the
qualified name should be passed explicitly::
>>> MyClass_NotGiven = sentinel('MyClass.NotGiven')
>>> MyClass_NotGiven
MyClass.NotGiven
Ordering comparisons are undefined for sentinel objects. Sentinels do not
support weakrefs.
Typing
------
To make usage of sentinels clear and simple in typed Python code, we propose to
amend the type system with a special case for sentinel objects.
Sentinel objects may be used in
:term:`type expressions <typing:type expression>`, representing themselves.
This is similar to how ``None`` is handled in the existing type system. For
example::
MISSING = sentinel('MISSING')
def foo(value: int | MISSING = MISSING) -> int:
...
More formally, type checkers should recognize sentinel creations of the form
``NAME = sentinel('NAME')`` as creating a new sentinel object. If the name
passed to ``sentinel()`` does not match the name the object is
assigned to, type checkers should emit an error.
Sentinels defined using this syntax may be used in
:term:`type expressions <typing:type expression>`. They
represent a :term:`fully static type <typing:fully static type>` that has a
single member, the sentinel object itself.
Type checkers should support narrowing union types involving sentinels
using the ``is`` and ``is not`` operators::
from typing import assert_type
MISSING = sentinel('MISSING')
def foo(value: int | MISSING) -> None:
if value is MISSING:
assert_type(value, MISSING)
else:
assert_type(value, int)
To support usage in type expressions, the runtime implementation
of sentinel objects should have the ``__or__`` and ``__ror__``
methods, returning :py:class:`typing.Union` objects.
The Typing Council `supports <https://github.com/python/steering-council/issues/258#issuecomment-2417524379>`_
this part of the proposal.
C API
-----
Sentinels can also be useful in C extensions. We propose two new
C API functions:
* ``PyObject *PySentinel_New(const char *name, const char *module_name, const char *repr)``
creates a new sentinel object. ``repr`` may be ``NULL``, in which case the sentinel's
repr will be the same as its name.
* ``bool PySentinel_Check(PyObject *obj)`` checks if an object is a sentinel.
C code can use the ``==`` operator to check if an object is a
specific sentinel.
Backwards Compatibility
=======================
Adding a new builtin means that code which currently relies on the bare name
``sentinel`` raising ``NameError`` will instead see the new builtin. This is
the usual compatibility consideration for new builtins. Existing local,
global, and imported names called ``sentinel`` are unaffected.
Code that already uses the name ``sentinel`` will have to be adapted to use
the new builtin and may receive new linter errors from linters that warn
about collisions with builtin names.
How to Teach This
=================
The normal types of documentation of new builtins and features, namely
docstrings, library docs and a section in "What's New", should suffice.
Security Implications
=====================
This proposal should have no security implications.
Reference Implementation
========================
A reference implementation is available as a CPython pull request [10]_. A
previous reference implementation is found in a dedicated GitHub repo [7]_.
A sketch of the intended behavior follows::
class sentinel:
"""Unique sentinel values."""
__slots__ = ("__name__", "__module__", "_repr")
def __init_subclass__(cls):
raise TypeError("type 'sentinel' is not an acceptable base type")
def __init__(self, name, /, repr=None):
if not isinstance(name, str):
raise TypeError("sentinel name must be a string")
self.__name__ = name
self.__module__ = sys._getframemodulename(1)
self._repr = repr if repr is not None else name
def __repr__(self):
return self._repr
def __reduce__(self):
return self.__name__
def __copy__(self):
return self
def __deepcopy__(self, memo):
return self
def __or__(self, other):
return typing.Union[self, other]
def __ror__(self, other):
return typing.Union[other, self]
A backport `exists <https://typing-extensions.readthedocs.io/en/latest/#typing_extensions.Sentinel>`_
in the :pypi:`typing-extensions` module, though its behavior does not precisely
match the current iteration of this PEP.
Rejected Ideas
==============
Use ``NotGiven = object()``
---------------------------
This suffers from all of the drawbacks mentioned in the `Rationale`_ section.
Add a single new sentinel value, such as ``MISSING`` or ``Sentinel``
--------------------------------------------------------------------
Since such a value could be used for various things in various places, one
could not always be confident that it would never be a valid value in some use
cases. On the other hand, a dedicated and distinct sentinel value can be used
with confidence without needing to consider potential edge-cases.
Additionally, it is useful to be able to provide a meaningful name and repr
for a sentinel value, specific to the context where it is used.
Finally, this was a very unpopular option in the poll [4]_, with only 12%
of the votes voting for it.
Use the existing ``Ellipsis`` sentinel value
--------------------------------------------
This is not the original intended use of Ellipsis, though it has become
increasingly common to use it to define empty class or function blocks instead
of using ``pass``.
Also, similar to a potential new single sentinel value, ``Ellipsis`` can't be
as confidently used in all cases, unlike a dedicated, distinct value.
Use a single-valued enum
------------------------
The suggested idiom is::
class NotGivenType(Enum):
NotGiven = 'NotGiven'
NotGiven = NotGivenType.NotGiven
Besides the excessive repetition, the repr is overly long:
``<NotGivenType.NotGiven: 'NotGiven'>``. A shorter repr can be defined, at
the expense of a bit more code and yet more repetition.
Finally, this option was the least popular among the nine options in the
poll [4]_, being the only option to receive no votes.
A sentinel class decorator
--------------------------
The suggested idiom is::
@sentinel
class NotGivenType: pass
NotGiven = NotGivenType()
While this allows for a very simple and clear implementation of the decorator,
the idiom is too verbose, repetitive, and difficult to remember.
Using class objects
-------------------
Since classes are inherently singletons, using a class as a sentinel value
makes sense and allows for a simple implementation.
The simplest version of this is::
class NotGiven: pass
To have a clear repr, one would need to use a meta-class::
class NotGiven(metaclass=SentinelMeta): pass
... or a class decorator::
@Sentinel
class NotGiven: pass
Using classes this way is unusual and could be confusing. The intention of
code would be hard to understand without comments. It would also cause
such sentinels to have some unexpected and undesirable behavior, such as
being callable.
Define a recommended "standard" idiom, without supplying an implementation
--------------------------------------------------------------------------
Most common existing idioms have significant drawbacks. So far, no idiom
has been found that is clear and concise while avoiding these drawbacks.
Also, in the poll [4]_ on this subject, the options for recommending an
idiom were unpopular, with the highest-voted option being voted for by only
25% of the voters.
Use a new standard library module
---------------------------------
Earlier drafts proposed adding a ``Sentinel`` class to a new ``sentinels`` or
``sentinellib`` module. However, adding a new module for a single public
callable is unnecessary, and using a module makes the feature less convenient
than the existing ``object()`` idiom. The Steering Council also specifically
encouraged making the feature a builtin so that it is at least as easy to use
as ``object()``.
Using the name ``sentinels`` would also conflict with an existing, actively
used PyPI package. While other module names are possible, making the feature a
builtin avoids the naming problem entirely.
Use a registry of per-module sentinel names
-------------------------------------------
Earlier drafts proposed making sentinel names unique within each module. Under
that design, repeated calls such as ``sentinel("MISSING")`` from the same
module would return the same object, using a process-global registry keyed by
module name and sentinel name.
This was rejected because the behavior is too implicit. Code that needs a
shared sentinel can define one explicitly and reuse it by name, just as code
already does with ``MISSING = object()``. Code in a local scope may also want
a fresh sentinel for each call or iteration, and repeated calls to
``sentinel(name)`` should behave like repeated calls to ``object()`` by
creating distinct objects.
Removing the registry also keeps the implementation and mental model simpler:
``sentinel(name)`` creates a new unique object whose repr is ``name``.
Automatically discover or pass a module name
--------------------------------------------
Earlier drafts proposed an optional ``module_name`` argument to support the
registry-based design.
With the registry removed, a public ``module_name`` argument is no longer
needed for the core proposal. The implementation still records the calling
module internally, as ``TypeVar`` and similar helpers do, so that pickle can
serialize importable sentinels by module and name. This internal module name
does not affect the sentinel's repr. If users want a repr that includes a
module or class name, they can include it in the single ``name`` argument
explicitly, e.g. ``sentinel("mymodule.MISSING")``.
Allowing customization of boolean evaluation
--------------------------------------------
Discussions considered allowing sentinels to be explicitly truthy, falsy, or
not convertible to ``bool``. Some existing third-party sentinels expose falsy
behavior as part of their public API, and several participants argued that
raising in boolean contexts would better enforce identity checks.
This PEP keeps the initial proposal simpler by giving sentinels the default
truthy behavior of ordinary objects and by recommending identity checks.
Custom boolean behavior may be considered later if the added API and typing
complexity is judged worthwhile.
Using ``typing.Literal`` in type annotations
--------------------------------------------
This was suggested by several people in discussions and is what this PEP
first went with. However, it was pointed out that this would cause potential
confusion, due to e.g. ``Literal["MISSING"]`` referring to the string value
``"MISSING"`` rather than being a forward-reference to a sentinel value
``MISSING``. Using the bare name was also suggested often in discussions.
This follows the precedent and well-known pattern set by ``None``, and has the
advantages of not requiring an import and being much shorter.
Additional Notes
================
* For sentinels defined in a class scope, to avoid potential name clashes,
or when a qualified repr would be clearer, one should pass the desired
qualified name explicitly. For example::
>>> class MyClass:
... NotGiven = sentinel('MyClass.NotGiven')
>>> MyClass.NotGiven
MyClass.NotGiven
* Creating sentinels in a function or method is allowed. Each call to
``sentinel()`` creates a distinct object, so a sentinel created in a local
scope behaves like one created by calling ``object()`` in that scope.
* The boolean value of ``NotImplemented`` is ``True``, but using this is
deprecated since Python 3.9 (doing so generates a deprecation warning.)
This deprecation is due to issues specific to ``NotImplemented``, as
described in bpo-35712 [8]_.
* To define multiple, related sentinel values, possibly with a defined
ordering among them, one should instead use ``Enum`` or something similar.
* There was a discussion on the typing-sig mailing list [9]_ about the typing
for these sentinels, where different options were discussed.
Footnotes
=========
.. [1] Python-Dev mailing list: `The repr of a sentinel <https://mail.python.org/archives/list/python-dev@python.org/thread/ZLVPD2OISI7M4POMTR2FCQTE6TPMPTO3/>`_
.. [2] Python-Dev mailing list: `"The stdlib contains tons of sentinels" <https://mail.python.org/archives/list/python-dev@python.org/message/JBYXQH3NV3YBF7P2HLHB5CD6V3GVTY55/>`_
.. [3] `bpo-44123: Make function parameter sentinel values true singletons <https://github.com/python/cpython/issues/88289>`_
.. [4] discuss.python.org Poll: `Sentinel Values in the Stdlib <https://discuss.python.org/t/sentinel-values-in-the-stdlib/8810/>`_
.. [5] `The "sentinels" package on PyPI <https://pypi.org/project/sentinels/>`_
.. [6] `The "sentinel" package on PyPI <https://pypi.org/project/sentinel/>`_
.. [7] `Reference implementation at the taleinat/python-stdlib-sentinels GitHub repo <https://github.com/taleinat/python-stdlib-sentinels>`_
.. [8] `bpo-35712: Make NotImplemented unusable in boolean context <https://github.com/python/cpython/issues/79893>`_
.. [9] `Discussion thread about type signatures for these sentinels on the typing-sig mailing list <https://mail.python.org/archives/list/typing-sig@python.org/thread/NDEJ7UCDPINP634GXWDARVMTGDVSNBKV/#LVCPTY26JQJW7NKGKGAZXHQKWVW7GOGL>`_
.. [10] `CPython reference implementation <https://github.com/JelleZijlstra/cpython/pull/4>`_
Copyright
=========
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.