Repository navigation
Codec name normalization breaks custom codecs #88886
Description
| BPO | 44723 |
|---|---|
| Nosy | @gpshead, @vstinner, @ezio-melotti, @methane, @bodograumann |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2021-07-23.10:18:59.285>
labels = ['type-bug', '3.9', 'expert-unicode']
title = 'Codec name normalization breaks custom codecs'
updated_at = <Date 2022-01-25.02:02:30.022>
user = 'https://github.com/bodograumann'bugs.python.org fields:
activity = <Date 2022-01-25.02:02:30.022>
actor = 'methane'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2021-07-23.10:18:59.285>
creator = 'bodograumann'
dependencies = []
files = []
hgrepos = []
issue_num = 44723
keywords = []
message_count = 1.0
messages = ['398042']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'vstinner', 'ezio.melotti', 'methane', 'bodograumann']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44723'
versions = ['Python 3.9']
Activity
bodograumann commentedon Jul 23, 2021
This is a follow up on https://bugs.python.org/issue37751 concerning normalization of codec names.
First of all, the changes made therein are not documented correctly.
In the implementation
| Normalization works as follows: all non-alphanumeric
| characters except the dot used for Python package names are
| collapsed and replaced with a single underscore, e.g. ' -;#'
| becomes '_'. Leading and trailing underscores are removed.”
Cf. encodings/init.py
The documentation however only states that:
| Search functions are expected to take one argument, being the encoding name in all lower case letters with hyphens and spaces converted to underscores
Cf. https://docs.python.org/3/library/codecs.html#codecs.register
Secondly, this change breaks lots of iconv codecs with the python-iconv binding. E.g.
ASCII//TRANSLITis now normalized toascii_translit, which iconv does not understand. Codec names which use hyphens also break and iinm not all of them have aliases in iconv without hyphens.Cf. python-iconv #4
How about first looking up the given name and only then, if the given name could not be found, looking for the codec by its normalized name?
serhiy-storchaka commentedon Jul 21, 2025
I am not sure that any normalization is needed in
codecs.lookup()(except maybe normalizing the letter case). It must be done in the search function. For example, standardencodings.search_function()needs to convert encoding name to module name -- therefore filter out/,\,:, and other characters that can be special in filesystem.serhiy-storchaka commentedon Jul 21, 2025
@vstinner, I think that #81932 should be rolled back. Even more, there is no reason to convert space to minus.
pythongh-88886: Remove excessive encoding name normalization
pythongh-88886: Remove excessive encoding name normalization
pythongh-88886: Remove excessive encoding name normalization
gh-88886: Remove excessive encoding name normalization (GH-137167)
pythongh-88886: Remove excessive encoding name normalization (pythonG…