Python pickler unable to pickle object the native pickler can

Bug report

Bug description:

import io
import pickle


class ZeroCopyByteArray(bytearray):
    def __reduce_ex__(self, protocol):
        return type(self), (pickle.PickleBuffer(self),), None


data = [
    ZeroCopyByteArray(),
    ZeroCopyByteArray(),
]

# this works
f = io.BytesIO()
pickler = pickle.Pickler(f, protocol=5)
pickler.dump(data)

# this hits memo assertion
f = io.BytesIO()
pickler = pickle._Pickler(f, protocol=5)
pickler.dump(data)

Traceback (most recent call last):
  File "bug.py", line 24, in <module>
    pickler.dump(data)
  File "/pickle.py", line 487, in dump
    self.save(obj)
  File "/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/pickle.py", line 932, in save_list
    self._batch_appends(obj)
  File "/pickle.py", line 956, in _batch_appends
    save(x)
  File "/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/pickle.py", line 692, in save_reduce
    save(args)
  File "/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/pickle.py", line 887, in save_tuple
    save(element)
  File "/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/pickle.py", line 842, in save_picklebuffer
    self.save_bytearray(m.tobytes())
  File "/pickle.py", line 821, in save_bytearray
    self.memoize(obj)
  File "/pickle.py", line 508, in memoize
    assert id(obj) not in self.memo
           ^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

CPython versions tested on:

3.11

Operating systems tested on:

Linux

Linked PRs

The ZeroCopyBytearray buffer appears to need the following implementation (it's actually in Lib/test/pickletester) but I confirmed that the issue remains but only if the buffer is empty (otherwise it's fine). The implementation is roughly

class ZeroCopyBytearray(bytearray):
    def __reduce_ex__(self, protocol):
        if protocol >= 5:
            return type(self)._reconstruct, (pickle.PickleBuffer(self),), None
        else:
            return type(self)._reconstruct, (bytes(self),)

    @classmethod
    def _reconstruct(cls, obj):
        with memoryview(obj) as m:
            obj = m.obj
            if type(obj) is cls:
                # Zero-copy
                return obj
            else:
                return cls(obj)

So, the __reduce_ex__ method makes more things than what you provided in your example, but it does not fix the issue at all.

EDIT: I'm investigating the issue in picnixz@a8c1464

Correct, I minimized the example to only the relevant parts for reproduction.

The issue only happens if you use protocol 5 and end up with two objects having different IDs produce a PickleBuffer with the same underlying buffer ID.

Yes, the issue is that this specific test case is not covered (I've added one and I'll try to fix it), which is why we did not see the issue (by specific case, I mean the two different instances that are empty, not that it still works if I use the same non-empty buffer values).

gh-120380: fix Python implementation of `pickle.Pickler` for `bytes` …

pythongh-120380: fix Python implementation of `pickle.Pickler` for `b…

[3.13] gh-120380: fix Python implementation of `pickle.Pickler` for `…

[3.12] gh-120380: fix Python implementation of `pickle.Pickler` for `…

@apmorton

Thank you @apmorton for the report, and @picnixz for the fix!

pythongh-120380: fix Python implementation of `pickle.Pickler` for `b…

Navigation Menu

Repository navigation

Python pickler unable to pickle object the native pickler can #120380

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Python pickler unable to pickle object the native pickler can #120380

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

picnixz commented on Jun 12, 2024

aisk commented on Jun 12, 2024

picnixz commented on Jun 12, 2024

picnixz commented on Jun 12, 2024

apmorton commented on Jun 12, 2024

picnixz commented on Jun 12, 2024

encukou commented on Jun 27, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions