test_local.TestEnUSCollection failures on Solaris 10

@loewis

BPO	16258
Nosy	@loewis, @jcea, @vstinner, @tpn, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2012-10-17.02:19:55.577>
labels = ['extension-modules', 'type-bug', '3.7']
title = 'test_local.TestEnUSCollection failures on Solaris 10'
updated_at = <Date 2017-06-20.14:54:41.733>
user = 'https://github.com/tpn'

bugs.python.org fields:

activity = <Date 2017-06-20.14:54:41.733>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2012-10-17.02:19:55.577>
creator = 'trent'
dependencies = []
files = []
hgrepos = []
issue_num = 16258
keywords = ['3.3regression']
message_count = 17.0
messages = ['173124', '173164', '173166', '173167', '173168', '173171', '173172', '173199', '289382', '296414', '296415', '296416', '296418', '296435', '296440', '296441', '296445']
nosy_count = 6.0
nosy_names = ['loewis', 'jcea', 'vstinner', 'trent', 'serhiy.storchaka', 'petriborg']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue16258'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

Linked PRs

======================================================================
ERROR: test_strxfrm (test.test_locale.TestEnUSCollation)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/cpython/buildslave/3.x.snakebite-solaris10-u10ga2-sparc64-1/build/Lib/test/test_locale.py", line 346, in test_strxfrm
    self.assertLess(locale.strxfrm('a'), locale.strxfrm('b'))
ValueError: character U+101010e is not in range [U+0000; U+10ffff]

======================================================================
ERROR: test_strxfrm_with_diacritic (test.test_locale.TestEnUSCollation)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/cpython/buildslave/3.x.snakebite-solaris10-u10ga2-sparc64-1/build/Lib/test/test_locale.py", line 367, in test_strxfrm_with_diacritic
    self.assertLess(locale.strxfrm('à'), locale.strxfrm('b'))
ValueError: character U+101010e is not in range [U+0000; U+10ffff]

Haven't investigated yet.

With the caveat that I know absolutely nothing about locales, here's what I've been able to reduce the problem down to:

zinc (alias s11, Solaris 11 x64):
    >>> locale.setlocale(locale.LC_ALL, 'C')
    'C'
    >>> locale.strxfrm('a')
    'a'
    >>> locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
    'en_US.UTF-8'
    >>> locale.strxfrm('a')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: character U+10105a3 is not in range [U+0000; U+10ffff]
    >>>

nitrogen (alias s10, Solaris 10 SPARC):

    >>> locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
    'en_US.UTF-8'
    >>> locale.strxfrm('a')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: character U+101010e is not in range [U+0000; U+10ffff]

Not sure how relevant it is, but on both those Solaris boxes, locale.LC_ALL returns 6, whereas on BSD and OS X it always seems to return 0.

I can reproduce this on my x86 Solaris 10 update 10.

With the system Python on s10:

Python 2.6.8 (unknown, Apr 13 2012, 17:08:12) [C] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.strxfrm('a')
'a'
>>> locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
'en_US.UTF-8'
>>> locale.strxfrm('a')
'\x01\x01\x01\x0e\x01\x01\x01\x01\x01\x01\x01\x02\x01\x01\x0fi\x01\x01\x01\x01'
>>> locale.strxfrm('a').decode('utf-8')
u'\x01\x01\x01\x0e\x01\x01\x01\x01\x01\x01\x01\x02\x01\x01\x0fi\x01\x01\x01\x01'

The difference between Python 2 and Python 3 is that Python 3 uses wcsxfrm, not strxfrm. Apparently Solaris' wcsxfrm is some broken thing that returns the same thing as strxfrm, cast to a wchar_t *, hence the character U+101010e (corresponding to the '\x01\x01\x01\x0e' bytestring above).

BTW, this works in python 3.2:

x86, 32 bit python, Solaris 10 update 10:

"""
Python 3.2.3 (default, Apr 12 2012, 13:29:13) 
[GCC 4.7.0] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
'en_US.UTF-8'
>>> locale.strxfrm('a')
'���\U00010f69�'
"""

It only works on Python 3.2 because PyUnicode_FromWideChar is more permissive, it seems. The first character in the wchar_t string returned by Solaris is still 0x101010e.

(by the way, I also tried a memset() before calling wcsxfrm(): no change)

Python 3.2 rejects characters outside the range U+0000-U+10ffff in
some operations, but not everywhere. I fixed Python 3.3 to be more
strict and always reject characters outside this range. I noticed the
Solaris issue with mbstowcs() on locale encodings different than
UTF-8: bpo-13441. I asked if it's more important to be strict on Unicode,
or if we need to handle the wcsxfrm() issue on python-dev:
http://mail.python.org/pipermail/python-dev/2011-December/114759.html

Stefan Krah answered: "Yes, if the cause is a broken mbstowcs() that
sounds good."
http://mail.python.org/pipermail/python-dev/2011-December/114781.html

I asked for help on OpenIndiana IRC channel, but nobody had a locale
encoding different than UTF-8. I didn't have access to a Solaris box,
so I chose to skip failing tests on Solaris.

My commit 2a2d0872d993 (and 7ffe3d304487) skips many locales to
workaround this issue in test__locale.

May be bpo-15954 is related to this issue. Is this issue still reproduced?

I'm getting the same 2 errors in Python 3.4.6 on Solaris 11.

Comes up when you run 'gmake test' or

./python -W default -bb -E -W error::BytesWarning -m test -r -w -j 0 -v test_locale.py

A solution for that would be to return the raw byte string or to return a list of integers, rather than an unicode string.

I don't think that locale.strxfrm() result is supposed to be displayed in a terminal, it should only be used to sort two strings, or to be used as a key function for list.sort() for example.

Currently, the function is documented to return a string:
https://docs.python.org/dev/library/locale.html#locale.strxfrm
"Transforms a string to one that can be used in locale-aware comparisons."

The problem is that we don't have enough developers who care of Solaris/Illimios to fix these issues (propose patches).

test_locale is just *one* example. The curses module is broken for years on Solaris if I recall correctly...

It is possible to use the special "encoding" for transformed strings on platforms with broken wcsxfrm().

All codes < 0x10000 are not changed. Codes >= 0x10000 are encoded as a pair: 0x10000 + (code >> 16), code & 0xffff.

Move to pythongh-60462.

Here results of some microbenchmarks. The first line is unpatched code, the second line is patched code.

$ ./python -m timeit -s "import locale; locale.setlocale(locale.LC_ALL, 'en_US.UTF-8'); s = 'a'*10" "locale.strxfrm(s)"
500000 loops, best of 5: 571 nsec per loop
500000 loops, best of 5: 558 nsec per loop

$ ./python -m timeit -s "import locale; locale.setlocale(locale.LC_ALL, 'en_US.UTF-8'); s = 'à'*10" "locale.strxfrm(s)"
500000 loops, best of 5: 608 nsec per loop
500000 loops, best of 5: 566 nsec per loop

$ ./python -m timeit -s "import locale; locale.setlocale(locale.LC_ALL, 'en_US.UTF-8'); s = '\u9e02'*10" "locale.strxfrm(s)"
500000 loops, best of 5: 476 nsec per loop
500000 loops, best of 5: 475 nsec per loop

$ ./python -m timeit -s "import locale; locale.setlocale(locale.LC_ALL, 'en_US.UTF-8'); s = 'a'*1000" "locale.strxfrm(s)"
5000 loops, best of 5: 40.9 usec per loop
10000 loops, best of 5: 37.3 usec per loop

$ ./python -m timeit -s "import locale; locale.setlocale(locale.LC_ALL, 'en_US.UTF-8'); s = 'à'*1000" "locale.strxfrm(s)"
5000 loops, best of 5: 44.3 usec per loop
5000 loops, best of 5: 39.6 usec per loop

$ ./python -m timeit -s "import locale; locale.setlocale(locale.LC_ALL, 'en_US.UTF-8'); s = '\u9e02'*1000" "locale.strxfrm(s)"
10000 loops, best of 5: 31.2 usec per loop
10000 loops, best of 5: 30.6 usec per loop

You can see that there is no significant overhead, but there is a small speed up (up to 10%).

gh-60462: Fix locale.strxfrm() on Solaris (GH-138242)

pythongh-60462: Fix locale.strxfrm() on Solaris (pythonGH-138242)

remove the test_locale skips - pythongh-60462 fixed the issue

[3.13] gh-60462: Fix locale.strxfrm() on Solaris (GH-138242) (GH-138449)

pythongh-60462: Fix locale.strxfrm() on Solaris (pythonGH-138242)

[3.14] gh-60462: Fix locale.strxfrm() on Solaris (GH-138242) (GH-138448)

Navigation Menu

Repository navigation

test_local.TestEnUSCollection failures on Solaris 10 #60462

Linked PRs

21 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

test_local.TestEnUSCollection failures on Solaris 10 #60462

Description

Linked PRs

Activity

tpn commented on Oct 17, 2012

tpn commented on Oct 17, 2012

jcea commented on Oct 17, 2012

pitrou commented on Oct 17, 2012

jcea commented on Oct 17, 2012

pitrou commented on Oct 17, 2012

pitrou commented on Oct 17, 2012

vstinner commented on Oct 17, 2012

serhiy-storchaka commented on Mar 10, 2017

petriborg commented on Jun 20, 2017

vstinner commented on Jun 20, 2017

vstinner commented on Jun 20, 2017

serhiy-storchaka commented on Jun 20, 2017

21 remaining items

serhiy-storchaka commented on Sep 3, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions