Non-UTF8 encoding line

@loewis

BPO	18961
Nosy	@loewis, @terryjreedy, @vstinner, @benjaminp, @meadori, @ericsnowcurrently, @serhiy-storchaka, @iritkatriel
Files	nonutf8_coding_line.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = None
created_at = <Date 2013-09-07.15:35:49.088>
labels = ['interpreter-core', 'type-bug', 'library', '3.11']
title = 'Non-UTF8 encoding line'
updated_at = <Date 2021-06-18.22:17:37.796>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2021-06-18.22:17:37.796>
actor = 'iritkatriel'
assignee = 'serhiy.storchaka'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Library (Lib)']
creation = <Date 2013-09-07.15:35:49.088>
creator = 'serhiy.storchaka'
dependencies = []
files = ['31648']
hgrepos = []
issue_num = 18961
keywords = []
message_count = 6.0
messages = ['197169', '197621', '197918', '197925', '197953', '396104']
nosy_count = 9.0
nosy_names = ['loewis', 'terry.reedy', 'vstinner', 'benjamin.peterson', 'Arfrever', 'meador.inge', 'eric.snow', 'serhiy.storchaka', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue18961'
versions = ['Python 3.11']

Linked PRs

Here is a file which is accepted by Python interpreter but is rejected by the tokenize module.

$ ./python nonutf8_coding_line.py 
$ ./python -m tokenize nonutf8_coding_line.py 
nonutf8_coding_line.py: error: invalid or missing encoding declaration for 'nonutf8_coding_line.py'

Python itself checks that a line is UTF-8 encoded only if not found a magic comment. The tokenize module checks it before searching (bpo-14629).

Which behavior do you propose to change? Does PEP-263 specify the response to a self-contradictory encoding comment? What do you think it should say?

I raising is the better behavior.

Idle notices that it cannot save the file with iso8859-15 encoding, so it saves it with utf-8 encoding instead. Any Python-aware editor that recognizes and uses the encoding cookie would have to either do the same or refuse to save until the encoding specification were changed.

Trying to run the file brings up a error box with SyntaxError: encoding problem: utf-8. Note that Idle runs a syntax check before creating a new subprocess, connecting to it, and trying to actually run the file. Closing the box takes one back to the editor as the spot where the error was noticed, which in this case is at the end of the line.

Terry: the comment isn't self-contradictory. Instead, the file constitutes a perfectly fine iso-8859-15 byte sequence (albeit a meaningless one: any byte sequence is perfectly fine iso-8559-15, and, in a comment, any characters are allowed by the Python syntax).

AFAICT, the file also follows the wording of PEP-263, as it does not disallow additional text on the encoding line. Therefore, I think tokenize should accept the file.

What about first line? Currently both Python interpreter and the tokenize module decode it from UTF-8 (actually due to bug bpo-18960 Python interprets it twice, in different encodings). PEP-263 says:

1. The complete Python source file should use a single encoding.
   Embedding of differently encoded data is not allowed and will
   result in a decoding error during compilation of the Python
   source code.

I conclude that the first line should be decoded with the encoding specified in the second line. We first should read the first line, check if it isn't a comment or contains encoding cookie, otherwise read the second line, determine the encoding, and decode read lines. Perhaps it will untangle bpo-18960 too.

One issue at a time, please, and bpo-18960 is already its own issue.

In any case, item "1" in "Concepts" of PEP-263 is clear that any deviation from the declared encoding should cause a decoding error.

I've reproduced the same in 3.11:

.\python.bat nonutf8_coding_line.py
Running Release|x64 interpreter...
.\python.bat -m tokenize nonutf8_coding_line.py
Running Release|x64 interpreter...
nonutf8_coding_line.py: error: invalid or missing encoding declaration for 'nonutf8_coding_line.py'

I can no longer reproduce on main with the provided file:

.$ /python -m tokenize nonutf8_coding_line.py 
0,0-0,0:            ENCODING       'iso8859-15'   
1,0-1,21:           COMMENT        '#coding=iso8859-15 Â€'
1,21-1,22:          NL             '\n'           
2,0-2,0:            ENDMARKER      ''

The line in the file cuts and copies as #coding=iso8859-15 ¤ where the final char is '\u00a4'. If I paste this into IDLE Shell, 3.8 or 3.14.0rc2 and hit return, it is seen as a comment and a new >>> prompts is displayed.

When I paste into a file and try to run, I see a popup with "I/O Error\nInvalid encoding 'iso8859-15'\nSaving as UTF-8". When I close that, "SyntaxError: encoding problem: iso8859 with BOM" appears. Subsequently, unless the file is changed so that it tries to save again, only the SyntaxError occurs. Running Check Module does the same. If this indicates a bug in IDLE, there should be a new issue. But I have not ever seen an IDLE user report.

pythongh-63161: Fix tokenize.detect_encoding() (pythonGH-139446)

I decided to not backport these changes to 3.13. Even if this is a bug, fixing it can break a working code.

[3.14] gh-63161: Fix tokenize.detect_encoding() (GH-139446) (GH-140378)

this appears to break code -- specifically PR #139446 -- asottile/pyupgrade#1038

cookie_re is not in the public API. detect_encoding() is in the public API.

while I agree in principle that this is undocumented:

it appears publicly named in dir(tokenize)
it has been consistently present since python 3.0
the public api is insufficient to detect failure-to-match (always produces utf-8 even if there is no coding comment)
the public api is inefficient / cumbersome: (roughly equivalent: tokenize.detect_encoding(io.BytesIO(s.encode()).readline)[0])
obviously I could copy the regex into my source -- however it has changed at least twice (and, for example, doesn't match the PEP so I can't just use that either)

willing to do the last bullet point of course -- but I don't think this breakage should appear in a bugfix release -- even in an undocumented attribute

We can restore it (and use an underscored name for internal needs), but what is the future of it? I don't think that it should be added in the public API, we should work with bytes if the encoding is not yet known.

BTW, similar regular expressions were copied several times in the stdlib and scripts (now they are mostly replaced with tokenize.detect_encoding() or tokenize.open(), and the scripts had been deleted).

yeah at least my usage is in a pretty unusual case -- it's a source code formatter and it's removing the comment if it detects it (since the tool already enforces utf-8-only source code) -- basically just asking "is this an encoding comment" -- not sure the best way to answer that question with a reasonable api without the exact pattern available somehow

cookie_re is not in the public API. detect_encoding() is in the public API.

Maybe it's time to rename the regex to _cookie_re?

I think that the simplest way to you is to copy the regular expression. It was copied in Tools/scripts/findnocoding.py.

Note that there is no need to remove a cookie comment after non-comment and non-empty line , e.g.:

print()
# coding: iso8859-1

Also, this is not an encoding cookie:

\
# coding: iso8859-1

Maybe it's time to rename the regex to _cookie_re?

There are 46 non-underscored names in dir(tokenize) which are not in tokenize.__all__. Is it a time to make a mass renaming? What about other modules?

Is it a time to make a mass renaming?

I don't think so. I'm only proposing to make cookie_re private since it has been modified.

And what then? If we don't restore cookie_re with an old value it will not be different from the current state. And if we add cookie_re with an old value, how long will we keep it and how guarantee that it will nor broke if it no longer used internally? When we finally remove it, we will return to the current situation.

This will also create precedence of preventing changes in non-public API. Well, there were already precedents in the past, but most changes are passed without noticing until too late.

pythongh-63161: Fix tokenize.detect_encoding() (pythonGH-139446)

Navigation Menu

Non-UTF8 encoding line #63161

Linked PRs

30 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Non-UTF8 encoding line #63161

Description

Linked PRs

Activity

serhiy-storchaka commented on Sep 7, 2013

terryjreedy commented on Sep 13, 2013

loewis commented on Sep 16, 2013

serhiy-storchaka commented on Sep 16, 2013

loewis commented on Sep 16, 2013

iritkatriel commented on Jun 18, 2021

StanFromIreland commented on Sep 10, 2025

terryjreedy commented on Sep 11, 2025

30 remaining items

serhiy-storchaka commented on Oct 20, 2025

asottile commented on Oct 29, 2025

serhiy-storchaka commented on Oct 29, 2025

asottile commented on Oct 29, 2025

serhiy-storchaka commented on Oct 29, 2025

asottile commented on Oct 29, 2025

vstinner commented on Oct 29, 2025

serhiy-storchaka commented on Oct 29, 2025

vstinner commented on Oct 29, 2025

serhiy-storchaka commented on Oct 30, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions