Repository navigation
Pickletools Default Encoding #126997
Description
Bug report
Bug description:
When unpickling using _pickle.c or pickle.py through load/loads, an encoding can be specified using the encoding argument, with the default being ASCII. However, pickletools does not support custom encodings and instead makes assumptions about what encoding it uses, which can lead to either incorrect data being displayed or erroring/not erroring when the normal unpickling process would error.
The three opcodes that I have found this in are STRING, BINSTRING, and SHORT_BINSTRING:
- On line 359, it assumes ASCII for
STRING - On line 456, it assumes
latin-1forBINSTRING - On line 422, it assumes
latin-1forSHORT_BINSTRING
I think the best solution would be to support encodings as an optional argument in pickletools.py, with the default being set to ASCII (since that's the default encoding for pickle.py and _pickle.c).
CPython versions tested on:
3.11
Operating systems tested on:
Linux
Linked PRs
Metadata
Metadata
Assignees
Labels
Projects
- Status
Activity
kanishka-coder0809 commentedon Nov 19, 2024
hey @Legoclones I think I can work with this issue ...can you please assign me with this
Legoclones commentedon Nov 19, 2024
I don't think I have permission to assign someone
kanishka-coder0809 commentedon Nov 20, 2024
So should I start working upon this and thank make a PR.?
pythongh-126997: Fix support of non-ASCII strings in pickletools
serhiy-storchaka commentedon Nov 20, 2024
It is not a problem if
pickletoolsaccepts an input which is not valid forpickle. But there are real bugs here:pickletoolsdoes not support STRING with non-ASCII argument.pickleallows to specify encoding (ASCII by default).pickletoolsdoes not support GLOBAL with non-ASCII argument.pickleuses UTF-8.pickletools.dis()outputs non-ASCII arguments for BINSTRING and SHORT_BINSTRING as Latin1-encoded.pickleallows to specify encoding (ASCII by default). It would be better to output non-ASCII bytes as\xXX.#127062 fixes them.
serhiy-storchaka commentedon Nov 20, 2024
@kanishka-coder0809, sorry, I already worked on this issue. It is slightly larger than initially was reported. On other hand, I think that I found a better solution than supporting optional argument for encoding. The latter would require more significant rewriting of the pickletools module and likely could be too complex for backporting.
Legoclones commentedon Nov 20, 2024
That looks great!
gh-126997: Fix support of non-ASCII strings in pickletools (GH-127062)
pythongh-126997: Fix support of non-ASCII strings in pickletools (pyt…
pythongh-126997: Fix support of non-ASCII strings in pickletools (pyt…
serhiy-storchaka commentedon Nov 21, 2024
Thank you for your report @Legoclones.
[3.12] gh-126997: Fix support of non-ASCII strings in pickletools (GH…
[3.13] gh-126997: Fix support of non-ASCII strings in pickletools (GH…
pythongh-126997: Fix support of non-ASCII strings in pickletools (pyt…