****************************

What's New In Python 3.2

****************************

:Author: Raymond Hettinger

.. $Id$

Rules for maintenance:

* Anyone can add text to this document. Do not spend very much time

on the wording of your changes, because your text will probably

get rewritten. (Note, during release candidate phase or just before

a beta release, please use the tracker instead -- this helps avoid

merge conflicts. If you must add a suggested entry directly,

please put it in an XXX comment and the maintainer will take notice).

* The maintainer will go through Misc/NEWS periodically and add

changes; it's therefore more important to add your changes to

Misc/NEWS than to this file.

* This is not a complete list of every single change; completeness

is the purpose of Misc/NEWS. Some changes I consider too small

or esoteric to include. If such a change is added to the text,

I'll just remove it. (This is another reason you shouldn't spend

too much time on writing your addition.)

* If you want to draw your new text to the attention of the

maintainer, add 'XXX' to the beginning of the paragraph or

section.

* It's OK to just add a fragmentary note about a change. For

example: "XXX Describe the transmogrify() function added to the

socket module." The maintainer will research the change and

write the necessary text.

* You can comment out your additions if you like, but it's not

necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is

sufficient; the e-mail address isn't necessary. It's helpful to

add the issue number:

XXX Describe the transmogrify() function added to the socket

module.

(Contributed by P.Y. Developer; :issue:`12345`.)

This saves the maintainer the effort of going through the SVN log

when researching a change.

This article explains the new features in Python 3.2 as compared to 3.1.

Python 3.2 was released on February 20, 2011. It

focuses on a few highlights and gives a few examples. For full details, see the

`Misc/NEWS

<https://github.com/python/cpython/blob/076ca6c3c8df3030307e548d9be792ce3c1c6eea/Misc/NEWS>`__

file.

.. seealso::

:pep:`392` - Python 3.2 Release Schedule

PEP 384: Defining a Stable ABI

==============================

In the past, extension modules built for one Python version were often

not usable with other Python versions. Particularly on Windows, every

feature release of Python required rebuilding all extension modules that

one wanted to use. This requirement was the result of the free access to

Python interpreter internals that extension modules could use.

With Python 3.2, an alternative approach becomes available: extension

modules which restrict themselves to a limited API (by defining

Py_LIMITED_API) cannot use many of the internals, but are constrained

to a set of API functions that are promised to be stable for several

releases. As a consequence, extension modules built for 3.2 in that

mode will also work with 3.3, 3.4, and so on. Extension modules that

make use of details of memory structures can still be built, but will

need to be recompiled for every feature release.

.. seealso::

:pep:`384` - Defining a Stable ABI

PEP written by Martin von Löwis.

PEP 389: Argparse Command Line Parsing Module

=============================================

A new module for command line parsing, :mod:`argparse`, was introduced to

overcome the limitations of :mod:`optparse` which did not provide support for

positional arguments (not just options), subcommands, required options and other

common patterns of specifying and validating options.

This module has already had widespread success in the community as a

third-party module. Being more fully featured than its predecessor, the

:mod:`argparse` module is now the preferred module for command-line processing.

The older module is still being kept available because of the substantial amount

of legacy code that depends on it.

Here's an annotated example parser showing features like limiting results to a

set of choices, specifying a *metavar* in the help screen, validating that one

or more positional arguments is present, and making a required option::

import argparse

parser = argparse.ArgumentParser(

description = 'Manage servers', # main description for help

epilog = 'Tested on Solaris and Linux') # displayed after help

parser.add_argument('action', # argument name

choices = ['deploy', 'start', 'stop'], # three allowed values

help = 'action on each target') # help msg

parser.add_argument('targets',

metavar = 'HOSTNAME', # var name used in help msg

nargs = '+', # require one or more targets

help = 'url for target machines') # help msg explanation

parser.add_argument('-u', '--user', # -u or --user option

required = True, # make it a required argument

help = 'login as user')

Example of calling the parser on a command string::

>>> cmd = 'deploy sneezy.example.com sleepy.example.com -u skycaptain'

>>> result = parser.parse_args(cmd.split())

>>> result.action

'deploy'

>>> result.targets

['sneezy.example.com', 'sleepy.example.com']

>>> result.user

'skycaptain'

Example of the parser's automatically generated help::

>>> parser.parse_args('-h'.split())

usage: manage_cloud.py [-h] -u USER

{deploy,start,stop} HOSTNAME [HOSTNAME ...]

Manage servers

positional arguments:

{deploy,start,stop} action on each target

HOSTNAME url for target machines

optional arguments:

-h, --help show this help message and exit

-u USER, --user USER login as user

Tested on Solaris and Linux

An especially nice :mod:`argparse` feature is the ability to define subparsers,

each with their own argument patterns and help displays::

import argparse

parser = argparse.ArgumentParser(prog='HELM')

subparsers = parser.add_subparsers()

parser_l = subparsers.add_parser('launch', help='Launch Control') # first subgroup

parser_l.add_argument('-m', '--missiles', action='store_true')

parser_l.add_argument('-t', '--torpedos', action='store_true')

parser_m = subparsers.add_parser('move', help='Move Vessel', # second subgroup

aliases=('steer', 'turn')) # equivalent names

parser_m.add_argument('-c', '--course', type=int, required=True)

parser_m.add_argument('-s', '--speed', type=int, default=0)

.. code-block:: shell-session

$ ./helm.py --help # top level help (launch and move)

$ ./helm.py launch --help # help for launch options

$ ./helm.py launch --missiles # set missiles=True and torpedos=False

$ ./helm.py steer --course 180 --speed 5 # set movement parameters

.. seealso::

:pep:`389` - New Command Line Parsing Module

PEP written by Steven Bethard.

:ref:`upgrading-optparse-code` for details on the differences from :mod:`optparse`.

PEP 391: Dictionary Based Configuration for Logging

====================================================

The :mod:`logging` module provided two kinds of configuration, one style with

function calls for each option or another style driven by an external file saved

in a :mod:`configparser` format. Those options did not provide the flexibility

to create configurations from JSON or YAML files, nor did they support

incremental configuration, which is needed for specifying logger options from a

command line.

To support a more flexible style, the module now offers

:func:`logging.config.dictConfig` for specifying logging configuration with

plain Python dictionaries. The configuration options include formatters,

handlers, filters, and loggers. Here's a working example of a configuration

dictionary::

{"version": 1,

"formatters": {"brief": {"format": "%(levelname)-8s: %(name)-15s: %(message)s"},

"full": {"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"}

},

"handlers": {"console": {

"class": "logging.StreamHandler",

"formatter": "brief",

"level": "INFO",

"stream": "ext://sys.stdout"},

"console_priority": {

"class": "logging.StreamHandler",

"formatter": "full",

"level": "ERROR",

"stream": "ext://sys.stderr"}

},

"root": {"level": "DEBUG", "handlers": ["console", "console_priority"]}}

If that dictionary is stored in a file called :file:`conf.json`, it can be

loaded and called with code like this::

>>> import json, logging.config

>>> with open('conf.json') as f:

... conf = json.load(f)

...

>>> logging.config.dictConfig(conf)

>>> logging.info("Transaction completed normally")

INFO : root : Transaction completed normally

>>> logging.critical("Abnormal termination")

2011-02-17 11:14:36,694 root CRITICAL Abnormal termination

.. seealso::

:pep:`391` - Dictionary Based Configuration for Logging

PEP written by Vinay Sajip.

PEP 3148: The ``concurrent.futures`` module

============================================

Code for creating and managing concurrency is being collected in a new top-level

namespace, *concurrent*. Its first member is a *futures* package which provides

a uniform high-level interface for managing threads and processes.

The design for :mod:`concurrent.futures` was inspired by the

*java.util.concurrent* package. In that model, a running call and its result

are represented by a :class:`~concurrent.futures.Future` object that abstracts

features common to threads, processes, and remote procedure calls. That object

supports status checks (running or done), timeouts, cancellations, adding

callbacks, and access to results or exceptions.

The primary offering of the new module is a pair of executor classes for

launching and managing calls. The goal of the executors is to make it easier to

use existing tools for making parallel calls. They save the effort needed to

setup a pool of resources, launch the calls, create a results queue, add

time-out handling, and limit the total number of threads, processes, or remote

procedure calls.

Ideally, each application should share a single executor across multiple

components so that process and thread limits can be centrally managed. This

solves the design challenge that arises when each component has its own

competing strategy for resource management.

Both classes share a common interface with three methods:

:meth:`~concurrent.futures.Executor.submit` for scheduling a callable and

returning a :class:`~concurrent.futures.Future` object;

:meth:`~concurrent.futures.Executor.map` for scheduling many asynchronous calls

at a time, and :meth:`~concurrent.futures.Executor.shutdown` for freeing

resources. The class is a :term:`context manager` and can be used in a

:keyword:`with` statement to assure that resources are automatically released

when currently pending futures are done executing.

A simple of example of :class:`~concurrent.futures.ThreadPoolExecutor` is a

launch of four parallel threads for copying files::

import concurrent.futures, shutil

with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e:

e.submit(shutil.copy, 'src1.txt', 'dest1.txt')

e.submit(shutil.copy, 'src2.txt', 'dest2.txt')

e.submit(shutil.copy, 'src3.txt', 'dest3.txt')

e.submit(shutil.copy, 'src3.txt', 'dest4.txt')

.. seealso::

:pep:`3148` - Futures -- Execute Computations Asynchronously

PEP written by Brian Quinlan.

:ref:`Code for Threaded Parallel URL reads<threadpoolexecutor-example>`, an

example using threads to fetch multiple web pages in parallel.

:ref:`Code for computing prime numbers in

parallel<processpoolexecutor-example>`, an example demonstrating

:class:`~concurrent.futures.ProcessPoolExecutor`.

PEP 3147: PYC Repository Directories

=====================================

Python's scheme for caching bytecode in *.pyc* files did not work well in

environments with multiple Python interpreters. If one interpreter encountered

a cached file created by another interpreter, it would recompile the source and

overwrite the cached file, thus losing the benefits of caching.

The issue of "pyc fights" has become more pronounced as it has become

commonplace for Linux distributions to ship with multiple versions of Python.

These conflicts also arise with CPython alternatives such as Unladen Swallow.

To solve this problem, Python's import machinery has been extended to use

distinct filenames for each interpreter. Instead of Python 3.2 and Python 3.3 and

Unladen Swallow each competing for a file called "mymodule.pyc", they will now

look for "mymodule.cpython-32.pyc", "mymodule.cpython-33.pyc", and

"mymodule.unladen10.pyc". And to prevent all of these new files from

cluttering source directories, the *pyc* files are now collected in a

"__pycache__" directory stored under the package directory.

Aside from the filenames and target directories, the new scheme has a few

aspects that are visible to the programmer:

* Imported modules now have a ``__cached__`` attribute which stores

the name of the actual file that was imported:

>>> import collections

>>> collections.__cached__ # doctest: +SKIP

'c:/py32/lib/__pycache__/collections.cpython-32.pyc'

* The tag that is unique to each interpreter is accessible from the :mod:`!imp`

module:

>>> import imp # doctest: +SKIP

>>> imp.get_tag() # doctest: +SKIP

'cpython-32'

* Scripts that try to deduce source filename from the imported file now need to

be smarter. It is no longer sufficient to simply strip the "c" from a ".pyc"

filename. Instead, use the new functions in the :mod:`!imp` module:

>>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc') # doctest: +SKIP

'c:/py32/lib/collections.py'

>>> imp.cache_from_source('c:/py32/lib/collections.py') # doctest: +SKIP

'c:/py32/lib/__pycache__/collections.cpython-32.pyc'

* The :mod:`py_compile` and :mod:`compileall` modules have been updated to

reflect the new naming convention and target directory. The command-line

invocation of *compileall* has new options: ``-i`` for

specifying a list of files and directories to compile and ``-b`` which causes

bytecode files to be written to their legacy location rather than

*__pycache__*.

* The :mod:`importlib.abc` module has been updated with new :term:`abstract base

classes <abstract base class>` for loading bytecode files. The obsolete

ABCs, :class:`!PyLoader` and

:class:`!PyPycLoader`, have been deprecated (instructions on how

to stay Python 3.1 compatible are included with the documentation).

.. seealso::

:pep:`3147` - PYC Repository Directories

PEP written by Barry Warsaw.

PEP 3149: ABI Version Tagged .so Files

======================================

The PYC repository directory allows multiple bytecode cache files to be

co-located. This PEP implements a similar mechanism for shared object files by

giving them a common directory and distinct names for each version.

The common directory is "pyshared" and the file names are made distinct by

identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the

major and minor version numbers, and optional build flags (such as "d" for

debug, "m" for pymalloc, "u" for wide-unicode). For an arbitrary package "foo",

you may see these files when the distribution package is installed::

/usr/share/pyshared/foo.cpython-32m.so

/usr/share/pyshared/foo.cpython-33md.so

In Python itself, the tags are accessible from functions in the :mod:`sysconfig`

module::

>>> import sysconfig

>>> sysconfig.get_config_var('SOABI') # find the version tag

'cpython-32mu'

>>> sysconfig.get_config_var('EXT_SUFFIX') # find the full filename extension

'.cpython-32mu.so'

.. seealso::

:pep:`3149` - ABI Version Tagged .so Files

PEP written by Barry Warsaw.

PEP 3333: Python Web Server Gateway Interface v1.0.1

=====================================================

This informational PEP clarifies how bytes/text issues are to be handled by the

WSGI protocol. The challenge is that string handling in Python 3 is most

conveniently handled with the :class:`str` type even though the HTTP protocol

is itself bytes oriented.

The PEP differentiates so-called *native strings* that are used for

request/response headers and metadata versus *byte strings* which are used for

the bodies of requests and responses.

The *native strings* are always of type :class:`str` but are restricted to code

points between *U+0000* through *U+00FF* which are translatable to bytes using

*Latin-1* encoding. These strings are used for the keys and values in the

environment dictionary and for response headers and statuses in the

:func:`!start_response` function. They must follow :rfc:`2616` with respect to

encoding. That is, they must either be *ISO-8859-1* characters or use

:rfc:`2047` MIME encoding.

For developers porting WSGI applications from Python 2, here are the salient

points:

* If the app already used strings for headers in Python 2, no change is needed.

* If instead, the app encoded output headers or decoded input headers, then the

headers will need to be re-encoded to Latin-1. For example, an output header

encoded in utf-8 was using ``h.encode('utf-8')`` now needs to convert from

bytes to native strings using ``h.encode('utf-8').decode('latin-1')``.

* Values yielded by an application or sent using the :meth:`!write` method

must be byte strings. The :func:`!start_response` function and environ

must use native strings. The two cannot be mixed.

For server implementers writing CGI-to-WSGI pathways or other CGI-style

protocols, the users must to be able access the environment using native strings

even though the underlying platform may have a different convention. To bridge

this gap, the :mod:`wsgiref` module has a new function,

:func:`wsgiref.handlers.read_environ` for transcoding CGI variables from

:data:`os.environ` into native strings and returning a new dictionary.

.. seealso::

:pep:`3333` - Python Web Server Gateway Interface v1.0.1

PEP written by Phillip Eby.

Other Language Changes

======================

Some smaller changes made to the core Python language are:

* String formatting for :func:`format` and :meth:`str.format` gained new

capabilities for the format character **#**. Previously, for integers in

binary, octal, or hexadecimal, it caused the output to be prefixed with '0b',

'0o', or '0x' respectively. Now it can also handle floats, complex, and

Decimal, causing the output to always have a decimal point even when no digits

follow it.

>>> format(20, '#o')

'0o24'

>>> format(12.34, '#5.0f')

' 12.'

(Suggested by Mark Dickinson and implemented by Eric Smith in :issue:`7094`.)

* There is also a new :meth:`str.format_map` method that extends the

capabilities of the existing :meth:`str.format` method by accepting arbitrary

:term:`mapping` objects. This new method makes it possible to use string

formatting with any of Python's many dictionary-like objects such as

:class:`~collections.defaultdict`, :class:`~shelve.Shelf`,

:class:`~configparser.ConfigParser`, or :mod:`dbm`. It is also useful with

custom :class:`dict` subclasses that normalize keys before look-up or that

supply a :meth:`~object.__missing__` method for unknown keys::

>>> import shelve

>>> d = shelve.open('tmp.shl')

>>> 'The {project_name} status is {status} as of {date}'.format_map(d)

'The testing project status is green as of February 15, 2011'

>>> class LowerCasedDict(dict):

... def __getitem__(self, key):

... return dict.__getitem__(self, key.lower())

...

>>> lcd = LowerCasedDict(part='widgets', quantity=10)

>>> 'There are {QUANTITY} {Part} in stock'.format_map(lcd)

'There are 10 widgets in stock'

>>> class PlaceholderDict(dict):

... def __missing__(self, key):

... return '<{}>'.format(key)

...

>>> 'Hello {name}, welcome to {location}'.format_map(PlaceholderDict())

'Hello <name>, welcome to <location>'

(Suggested by Raymond Hettinger and implemented by Eric Smith in

:issue:`6081`.)

* The interpreter can now be started with a quiet option, ``-q``, to prevent

the copyright and version information from being displayed in the interactive

mode. The option can be introspected using the :data:`sys.flags` attribute:

.. code-block:: shell-session

$ python -q

>>> sys.flags

sys.flags(debug=0, division_warning=0, inspect=0, interactive=0,

optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0,

ignore_environment=0, verbose=0, bytes_warning=0, quiet=1)

(Contributed by Marcin Wojdyr in :issue:`1772833`).

* The :func:`hasattr` function works by calling :func:`getattr` and detecting

whether an exception is raised. This technique allows it to detect methods

created dynamically by :meth:`~object.__getattr__` or :meth:`~object.__getattribute__` which

would otherwise be absent from the class dictionary. Formerly, *hasattr*

would catch any exception, possibly masking genuine errors. Now, *hasattr*

has been tightened to only catch :exc:`AttributeError` and let other

exceptions pass through::

>>> class A:

... @property

... def f(self):

... return 1 // 0

...

>>> a = A()

>>> hasattr(a, 'f')

Traceback (most recent call last):

...

ZeroDivisionError: integer division or modulo by zero

(Discovered by Yury Selivanov and fixed by Benjamin Peterson; :issue:`9666`.)

* The :func:`str` of a float or complex number is now the same as its

:func:`repr`. Previously, the :func:`str` form was shorter but that just

caused confusion and is no longer needed now that the shortest possible

:func:`repr` is displayed by default:

>>> import math

>>> repr(math.pi)

'3.141592653589793'

>>> str(math.pi)

'3.141592653589793'

(Proposed and implemented by Mark Dickinson; :issue:`9337`.)

* :class:`memoryview` objects now have a :meth:`~memoryview.release` method

and they also now support the context management protocol. This allows timely

release of any resources that were acquired when requesting a buffer from the

original object.

>>> with memoryview(b'abcdefgh') as v:

... print(v.tolist())

[97, 98, 99, 100, 101, 102, 103, 104]

(Added by Antoine Pitrou; :issue:`9757`.)

* Previously it was illegal to delete a name from the local namespace if it

occurs as a free variable in a nested block::

def outer(x):

def inner():

return x

inner()

del x

This is now allowed. Remember that the target of an :keyword:`except` clause

is cleared, so this code which used to work with Python 2.6, raised a

:exc:`SyntaxError` with Python 3.1 and now works again::

def f():

def print_error():

print(e)

try:

something

except Exception as e:

print_error()

# implicit "del e" here

(See :issue:`4617`.)

* :ref:`Struct sequence types <struct-sequence-objects>` are now subclasses of tuple.

This means that C structures like those returned by :func:`os.stat`,

:func:`time.gmtime`, and :data:`sys.version_info` now work like a

:term:`named tuple` and now work with functions and methods that

expect a tuple as an argument. This is a big step forward in making the C

structures as flexible as their pure Python counterparts:

>>> import sys

>>> isinstance(sys.version_info, tuple)

True

>>> 'Version %d.%d.%d %s(%d)' % sys.version_info # doctest: +SKIP

'Version 3.2.0 final(0)'

(Suggested by Arfrever Frehtes Taifersar Arahesis and implemented

by Benjamin Peterson in :issue:`8413`.)

* Warnings are now easier to control using the :envvar:`PYTHONWARNINGS`

environment variable as an alternative to using ``-W`` at the command line:

.. code-block:: shell-session

$ export PYTHONWARNINGS='ignore::RuntimeWarning::,once::UnicodeWarning::'

(Suggested by Barry Warsaw and implemented by Philip Jenvey in :issue:`7301`.)

* A new warning category, :exc:`ResourceWarning`, has been added. It is

emitted when potential issues with resource consumption or cleanup

are detected. It is silenced by default in normal release builds but

can be enabled through the means provided by the :mod:`warnings`

module, or on the command line.

A :exc:`ResourceWarning` is issued at interpreter shutdown if the

:data:`gc.garbage` list isn't empty, and if :const:`gc.DEBUG_UNCOLLECTABLE` is

set, all uncollectable objects are printed. This is meant to make the

programmer aware that their code contains object finalization issues.

A :exc:`ResourceWarning` is also issued when a :term:`file object` is destroyed

without having been explicitly closed. While the deallocator for such

object ensures it closes the underlying operating system resource

(usually, a file descriptor), the delay in deallocating the object could

produce various issues, especially under Windows. Here is an example

of enabling the warning from the command line:

.. code-block:: shell-session

$ python -q -Wdefault

>>> f = open("foo", "wb")

>>> del f

__main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'>

(Added by Antoine Pitrou and Georg Brandl in :issue:`10093` and :issue:`477863`.)

* :class:`range` objects now support *index* and *count* methods. This is part

of an effort to make more objects fully implement the

:class:`collections.Sequence <collections.abc.Sequence>` :term:`abstract base class`. As a result, the

language will have a more uniform API. In addition, :class:`range` objects

now support slicing and negative indices, even with values larger than

:data:`sys.maxsize`. This makes *range* more interoperable with lists::

>>> range(0, 100, 2).count(10)

1

>>> range(0, 100, 2).index(10)

5

>>> range(0, 100, 2)[5]

10

>>> range(0, 100, 2)[0:5]

range(0, 10, 2)

(Contributed by Daniel Stutzbach in :issue:`9213`, by Alexander Belopolsky

in :issue:`2690`, and by Nick Coghlan in :issue:`10889`.)

* The :func:`callable` builtin function from Py2.x was resurrected. It provides

a concise, readable alternative to using an :term:`abstract base class` in an

expression like ``isinstance(x, collections.Callable)``:

>>> callable(max)

True

>>> callable(20)

False

(See :issue:`10518`.)

* Python's import mechanism can now load modules installed in directories with

non-ASCII characters in the path name. This solved an aggravating problem

with home directories for users with non-ASCII characters in their usernames.

(Required extensive work by Victor Stinner in :issue:`9425`.)

New, Improved, and Deprecated Modules

=====================================

Python's standard library has undergone significant maintenance efforts and

quality improvements.

The biggest news for Python 3.2 is that the :mod:`email` package, :mod:`mailbox`

module, and :mod:`!nntplib` modules now work correctly with the bytes/text model

in Python 3. For the first time, there is correct handling of messages with

mixed encodings.

Throughout the standard library, there has been more careful attention to

encodings and text versus bytes issues. In particular, interactions with the

operating system are now better able to exchange non-ASCII data using the

Windows MBCS encoding, locale-aware encodings, or UTF-8.

Another significant win is the addition of substantially better support for

*SSL* connections and security certificates.

In addition, more classes now implement a :term:`context manager` to support

convenient and reliable resource clean-up using a :keyword:`with` statement.

email

-----

The usability of the :mod:`email` package in Python 3 has been mostly fixed by

the extensive efforts of R. David Murray. The problem was that emails are

typically read and stored in the form of :class:`bytes` rather than :class:`str`

text, and they may contain multiple encodings within a single email. So, the

email package had to be extended to parse and generate email messages in bytes

format.

* New functions :func:`~email.message_from_bytes` and

:func:`~email.message_from_binary_file`, and new classes

:class:`~email.parser.BytesFeedParser` and :class:`~email.parser.BytesParser`

allow binary message data to be parsed into model objects.

* Given bytes input to the model, :meth:`~email.message.Message.get_payload`

will by default decode a message body that has a

:mailheader:`Content-Transfer-Encoding` of *8bit* using the charset

specified in the MIME headers and return the resulting string.

* Given bytes input to the model, :class:`~email.generator.Generator` will

convert message bodies that have a :mailheader:`Content-Transfer-Encoding` of

*8bit* to instead have a *7bit* :mailheader:`Content-Transfer-Encoding`.

Headers with unencoded non-ASCII bytes are deemed to be :rfc:`2047`\ -encoded

using the *unknown-8bit* character set.

* A new class :class:`~email.generator.BytesGenerator` produces bytes as output,

preserving any unchanged non-ASCII data that was present in the input used to

build the model, including message bodies with a

:mailheader:`Content-Transfer-Encoding` of *8bit*.

* The :mod:`smtplib` :class:`~smtplib.SMTP` class now accepts a byte string

for the *msg* argument to the :meth:`~smtplib.SMTP.sendmail` method,

and a new method, :meth:`~smtplib.SMTP.send_message` accepts a

:class:`~email.message.Message` object and can optionally obtain the

*from_addr* and *to_addrs* addresses directly from the object.

(Proposed and implemented by R. David Murray, :issue:`4661` and :issue:`10321`.)

elementtree

-----------

The :mod:`xml.etree.ElementTree` package and its :mod:`!xml.etree.cElementTree`

counterpart have been updated to version 1.3.

Several new and useful functions and methods have been added:

* :func:`xml.etree.ElementTree.fromstringlist` which builds an XML document

from a sequence of fragments

* :func:`xml.etree.ElementTree.register_namespace` for registering a global

namespace prefix

* :func:`xml.etree.ElementTree.tostringlist` for string representation

including all sublists

* :meth:`xml.etree.ElementTree.Element.extend` for appending a sequence of zero

or more elements

* :meth:`xml.etree.ElementTree.Element.iterfind` searches an element and

subelements

* :meth:`xml.etree.ElementTree.Element.itertext` creates a text iterator over

an element and its subelements

* :meth:`xml.etree.ElementTree.TreeBuilder.end` closes the current element

* :meth:`xml.etree.ElementTree.TreeBuilder.doctype` handles a doctype

declaration

Two methods have been deprecated:

* :meth:`!xml.etree.ElementTree.getchildren` use ``list(elem)`` instead.

* :meth:`!xml.etree.ElementTree.getiterator` use ``Element.iter`` instead.

For details of the update, see `Introducing ElementTree

<https://web.archive.org/web/20200703234532/http://effbot.org/zone/elementtree-13-intro.htm>`_

on Fredrik Lundh's website.

(Contributed by Florent Xicluna and Fredrik Lundh, :issue:`6472`.)

functools

---------

* The :mod:`functools` module includes a new decorator for caching function

calls. :deco:`functools.lru_cache` can save repeated queries to an external

resource whenever the results are expected to be the same.

For example, adding a caching decorator to a database query function can save

database accesses for popular searches:

>>> import functools

>>> @functools.lru_cache(maxsize=300)

... def get_phone_number(name):

... c = conn.cursor()

... c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,))

... return c.fetchone()[0]

>>> for name in user_requests: # doctest: +SKIP

... get_phone_number(name) # cached lookup

To help with choosing an effective cache size, the wrapped function is

instrumented for tracking cache statistics:

>>> get_phone_number.cache_info() # doctest: +SKIP

CacheInfo(hits=4805, misses=980, maxsize=300, currsize=300)

If the phonelist table gets updated, the outdated contents of the cache can be

cleared with:

>>> get_phone_number.cache_clear()

(Contributed by Raymond Hettinger and incorporating design ideas from Jim

Baker, Miki Tebeka, and Nick Coghlan; see `recipe 498245

<https://code.activestate.com/recipes/498245-lru-and-lfu-cache-decorators/>`_\, `recipe 577479

<https://code.activestate.com/recipes/577479-simple-caching-decorator/>`_\, :issue:`10586`, and

:issue:`10593`.)

* The :deco:`functools.wraps` decorator now adds a :attr:`__wrapped__` attribute

pointing to the original callable function. This allows wrapped functions to

be introspected. It also copies :attr:`~function.__annotations__` if

defined. And now it also gracefully skips over missing attributes such as

:attr:`~function.__doc__` which

might not be defined for the wrapped callable.

In the above example, the cache can be removed by recovering the original

function:

>>> get_phone_number = get_phone_number.__wrapped__ # uncached function

(By Nick Coghlan and Terrence Cole; :issue:`9567`, :issue:`3445`, and

:issue:`8814`.)

* To help write classes with rich comparison methods, a new decorator

:deco:`functools.total_ordering` will use existing equality and inequality

methods to fill in the remaining methods.

For example, supplying *__eq__* and *__lt__* will enable

:deco:`~functools.total_ordering` to fill-in *__le__*, *__gt__* and *__ge__*::

@total_ordering

class Student:

def __eq__(self, other):

return ((self.lastname.lower(), self.firstname.lower()) ==

(other.lastname.lower(), other.firstname.lower()))

def __lt__(self, other):

return ((self.lastname.lower(), self.firstname.lower()) <

(other.lastname.lower(), other.firstname.lower()))

With the *total_ordering* decorator, the remaining comparison methods

are filled in automatically.

(Contributed by Raymond Hettinger.)

* To aid in porting programs from Python 2, the :func:`functools.cmp_to_key`

function converts an old-style comparison function to

modern :term:`key function`:

>>> # locale-aware sort order

>>> sorted(iterable, key=cmp_to_key(locale.strcoll)) # doctest: +SKIP

For sorting examples and a brief sorting tutorial, see the `Sorting HowTo

<https://wiki.python.org/moin/HowTo/Sorting/>`_ tutorial.

(Contributed by Raymond Hettinger.)

itertools

---------

* The :mod:`itertools` module has a new :func:`~itertools.accumulate` function

modeled on APL's *scan* operator and Numpy's *accumulate* function:

>>> from itertools import accumulate

>>> list(accumulate([8, 2, 50]))

[8, 10, 60]

>>> prob_dist = [0.1, 0.4, 0.2, 0.3]

>>> list(accumulate(prob_dist)) # cumulative probability distribution

[0.1, 0.5, 0.7, 1.0]

For an example using :func:`~itertools.accumulate`, see the :ref:`examples for

the random module <random-examples>`.

(Contributed by Raymond Hettinger and incorporating design suggestions

from Mark Dickinson.)

collections

-----------

* The :class:`collections.Counter` class now has two forms of in-place

subtraction, the existing *-=* operator for `saturating subtraction

<https://en.wikipedia.org/wiki/Saturation_arithmetic>`_ and the new

:meth:`~collections.Counter.subtract` method for regular subtraction. The

former is suitable for `multisets <https://en.wikipedia.org/wiki/Multiset>`_

which only have positive counts, and the latter is more suitable for use cases

that allow negative counts:

>>> from collections import Counter

>>> tally = Counter(dogs=5, cats=3)

>>> tally -= Counter(dogs=2, cats=8) # saturating subtraction

>>> tally

Counter({'dogs': 3})

>>> tally = Counter(dogs=5, cats=3)

>>> tally.subtract(dogs=2, cats=8) # regular subtraction

>>> tally

Counter({'dogs': 3, 'cats': -5})

(Contributed by Raymond Hettinger.)

* The :class:`collections.OrderedDict` class has a new method

:meth:`~collections.OrderedDict.move_to_end` which takes an existing key and

moves it to either the first or last position in the ordered sequence.

The default is to move an item to the last position. This is equivalent of

renewing an entry with ``od[k] = od.pop(k)``.

A fast move-to-end operation is useful for resequencing entries. For example,

an ordered dictionary can be used to track order of access by aging entries

from the oldest to the most recently accessed.

>>> from collections import OrderedDict

>>> d = OrderedDict.fromkeys(['a', 'b', 'X', 'd', 'e'])

>>> list(d)

['a', 'b', 'X', 'd', 'e']

>>> d.move_to_end('X')

>>> list(d)

['a', 'b', 'd', 'e', 'X']

(Contributed by Raymond Hettinger.)

* The :class:`collections.deque` class grew two new methods

:meth:`~collections.deque.count` and :meth:`~collections.deque.reverse` that

make them more substitutable for :class:`list` objects:

>>> from collections import deque

>>> d = deque('simsalabim')

>>> d.count('s')

2

>>> d.reverse()

>>> d

deque(['m', 'i', 'b', 'a', 'l', 'a', 's', 'm', 'i', 's'])

(Contributed by Raymond Hettinger.)

threading

---------

The :mod:`threading` module has a new :class:`~threading.Barrier`

synchronization class for making multiple threads wait until all of them have

reached a common barrier point. Barriers are useful for making sure that a task

with multiple preconditions does not run until all of the predecessor tasks are

complete.

Barriers can work with an arbitrary number of threads. This is a generalization

of a `Rendezvous <https://en.wikipedia.org/wiki/Synchronous_rendezvous>`_ which

is defined for only two threads.

Implemented as a two-phase cyclic barrier, :class:`~threading.Barrier` objects

are suitable for use in loops. The separate *filling* and *draining* phases

assure that all threads get released (drained) before any one of them can loop

back and re-enter the barrier. The barrier fully resets after each cycle.

Example of using barriers::

from threading import Barrier, Thread

def get_votes(site):

ballots = conduct_election(site)

all_polls_closed.wait() # do not count until all polls are closed

totals = summarize(ballots)

publish(site, totals)

all_polls_closed = Barrier(len(sites))

for site in sites:

Thread(target=get_votes, args=(site,)).start()

In this example, the barrier enforces a rule that votes cannot be counted at any

polling site until all polls are closed. Notice how a solution with a barrier

is similar to one with :meth:`threading.Thread.join`, but the threads stay alive

and continue to do work (summarizing ballots) after the barrier point is

crossed.

If any of the predecessor tasks can hang or be delayed, a barrier can be created

with an optional *timeout* parameter. Then if the timeout period elapses before

all the predecessor tasks reach the barrier point, all waiting threads are

released and a :exc:`~threading.BrokenBarrierError` exception is raised::

def get_votes(site):

ballots = conduct_election(site)

try:

all_polls_closed.wait(timeout=midnight - time.now())

except BrokenBarrierError:

lockbox = seal_ballots(ballots)

queue.put(lockbox)

else:

totals = summarize(ballots)

publish(site, totals)

In this example, the barrier enforces a more robust rule. If some election

sites do not finish before midnight, the barrier times-out and the ballots are

sealed and deposited in a queue for later handling.

See `Barrier Synchronization Patterns

<https://osl.cs.illinois.edu/media/papers/karmani-2009-barrier_synchronization_pattern.pdf>`_

for more examples of how barriers can be used in parallel computing. Also, there is

a simple but thorough explanation of barriers in `The Little Book of Semaphores

<https://greenteapress.com/semaphores/LittleBookOfSemaphores.pdf>`_, *section 3.6*.

(Contributed by Kristján Valur Jónsson with an API review by Jeffrey Yasskin in

:issue:`8777`.)

datetime and time

-----------------

* The :mod:`datetime` module has a new type :class:`~datetime.timezone` that

implements the :class:`~datetime.tzinfo` interface by returning a fixed UTC

offset and timezone name. This makes it easier to create timezone-aware

datetime objects::

>>> import datetime as dt

>>> dt.datetime.now(dt.timezone.utc)

datetime.datetime(2010, 12, 8, 21, 4, 2, 923754, tzinfo=datetime.timezone.utc)

>>> dt.datetime.strptime("01/01/2000 12:00 +0000", "%m/%d/%Y %H:%M %z")