SRE ignores the ASCII flag on character ranges with non-BMP upper bound

Bug report

Bug description:

It seems like SRE ignores the ASCII flag when parsing a character range whose upper bound is beyond the BMP region:

>>> import re

# should match
>>> regex = re.compile("[\ua7aa-\uffff]", re.IGNORECASE)
>>> print(regex.match("\u0266"))
<re.Match object; span=(0, 1), match='ɦ'> 

# should not match
>>> regex = re.compile("[\ua7aa-\U00010000]", re.ASCII | re.IGNORECASE)
>>> print(regex.match("\u0266"))
<re.Match object; span=(0, 1), match='ɦ'>

# must be related to case folding, since \ua7aa folds to \u0266
>>> regex = re.compile("[\ua7ab-\U00010000]", re.ASCII | re.IGNORECASE)
>>> print(regex.match("\u0266"))
None

# correct behavior when upper bound is in BMP
>>> regex = re.compile("[\ua7aa-\uffff]", re.ASCII | re.IGNORECASE)
>>> print(regex.match("\u0266"))
None

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

I think this could be caused by this line:

cpython/Lib/re/_compiler.py

Line 301 in a1c57bc

if fixup:

When the pattern is being compiled in _compiler.py's optimize_charset, the RANGE opcode is translated into the RANGE_UNI_IGNORE opcode. This should probably be done only in Unicode mode. The correct way to check for Unicode mode in that function would be to check if fixes:, because the fixes argument is None in ASCII and Locale mode and a dict in Unicode mode. The code currently uses the condition if fixup:, but fixup is None only in Locale mode and it is a function in both ASCII and Unicode mode. This means that this replacement is used in ASCII mode too.

@serhiy-storchaka

cc @serhiy-storchaka as the RE expert

@jirkamarsik

Thank you for your report @jirkamarsik. Actually, this is a more complex issue.

Currently, the result in the following examples does not depend on the upper bound if it is larger enough.

>>> import re
>>> re.match(r'[N-\uffff]', 'A', re.I|re.A)
<re.Match object; span=(0, 1), match='A'>
>>> re.match(r'[n-\uffff]', 'Z', re.I|re.A)
<re.Match object; span=(0, 1), match='Z'>
>>> re.match(r'[N-\U00010000]', 'A', re.I|re.A)
<re.Match object; span=(0, 1), match='A'>
>>> re.match(r'[n-\U00010000]', 'Z', re.I|re.A)
<re.Match object; span=(0, 1), match='Z'>

But with the proposed fix the last two matches will return None.

I am working on this.

I have found also other bug: re.match(r'[19\U00010400]', '\U00010400', re.I) returns None.

pythongh-126505: Fix bugs in compiling case-insensitive character cla…

gh-126505: Fix bugs in compiling case-insensitive character classes (G…

pythongh-126505: Fix bugs in compiling case-insensitive character cla…

[3.12] gh-126505: Fix bugs in compiling case-insensitive character cl…

[3.13] gh-126505: Fix bugs in compiling case-insensitive character cl…

pythongh-126505: Fix bugs in compiling case-insensitive character cla…

Navigation Menu

Repository navigation

SRE ignores the ASCII flag on character ranges with non-BMP upper bound #126505

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

SRE ignores the ASCII flag on character ranges with non-BMP upper bound #126505

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

jirkamarsik commented on Nov 6, 2024

picnixz commented on Nov 7, 2024

serhiy-storchaka commented on Nov 7, 2024

serhiy-storchaka commented on Nov 7, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions