Repository navigation
3.11 regression: traceback.format_list raises UnicodeDecodeError in certain scenarios #98744
Description
Bug report
Take the following piece of code:
import sys, traceback
try:
ｗｉｄｔｈ
except:
_, _, tb = sys.exc_info()
tblist = traceback.extract_tb(tb)
print(traceback.format_list(tblist))With Python 3.10 and earlier versions, executing a file with this code produces:
[' File "/tmp/test.py", line 3, in <module>\n ｗｉｄｔｈ\n']
With Python 3.11, a UnicodeDecodeError is raised instead:
Traceback (most recent call last):
File "/tmp/test.py", line 9, in <module>
print(traceback.format_list(tblist))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/traceback.py", line 41, in format_list
return StackSummary.from_list(extracted_list).format()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/traceback.py", line 531, in format
formatted_frame = self.format_frame_summary(frame_summary)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/traceback.py", line 478, in format_frame_summary
colno = _byte_offset_to_character_offset(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/traceback.py", line 566, in _byte_offset_to_character_offset
return len(as_utf8[:offset + 1].decode("utf-8"))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xef in position 4: unexpected end of data
Your environment
- CPython versions tested on: 3.10.8 and 3.11.0
- Operating system and architecture: Debian unstable, amd64
Activity
sebastinas commentedon Oct 27, 2022
Current main (d578aae) also has the same issue.
serhiy-storchaka commentedon Oct 28, 2022
cc @ammaraskar, @pablogsal, @isidentical
mdboom commentedon Oct 28, 2022
Also cc @iritkatriel (related to source positions -- in this case a source position in the middle of a UTF-8 codepoint).
isidentical commentedon Oct 28, 2022
It seems like we are not mirroring the behaviour from the C implementation, which uses
errors="replace"instead of the defaultstrict. I'll try to send a PR soon. Thanks for the report @sebastinas!mdboom commentedon Oct 28, 2022
@isidentical: I was just looking at this as well, and there is also an off-by-one difference between this and the C implementation.
pythongh-98744: Prevent column-level decoding crashes when using the …
isidentical commentedon Oct 28, 2022
@mdboom you are right! That is also an issue (in the C Implementation, we do a lengths check to not to overflow; although it is not necessary here. Another problem is the caret positions which actually might differ from node's offsets). Both should now be resolved with #98824. Thanks for catching it.
pythongh-98744: Prevent column-level decoding crashes when using the …
gh-98744: Prevent column-level decoding crashes on traceback module (#…
[3.11] pythongh-98744: Prevent column-level decoding crashes on trace…
[3.11] gh-98744: Prevent column-level decoding crashes on traceback m…
isidentical commentedon Oct 29, 2022
Should be fixed now, thanks everyone!