Data race between resize_buffer_lock_held and _Py_DecRefShared on a shared BytesIO buffer

Bug report

Bug description:

In the free-threaded build, a whole-buffer BytesIO.read() returns self->buf by reference (Py_NewRef) instead of a copy,

cpython/Modules/_io/bytesio.c

Lines 432 to 437 in 9e863fa

    
           if (size > 1 && 
        
               self->pos == 0 && size == PyBytes_GET_SIZE(self->buf) && 
        
               FT_ATOMIC_LOAD_SSIZE_RELAXED(self->exports) == 0) { 
        
               self->pos += size; 
        
               return Py_NewRef(self->buf); 
        
           }

so resize_buffer_lock_held() can reallocate that same self->buf in place (_PyBytes_Resize)

cpython/Modules/_io/bytesio.c

Lines 171 to 178 in 9e863fa

    
           if (SHARED_BUF(self)) { 
        
               if (unshare_buffer_lock_held(self, alloc) < 0) 
        
                   return -1; 
        
           } 
        
           else { 
        
               if (_PyBytes_Resize(&self->buf, alloc) < 0) 
        
                   return -1; 
        
           }

while another thread concurrently decrefs the read-returned reference (_Py_DecRefShared), racing on the same buffer object without synchronization.

Reproducer:

import io
from threading import Thread, Barrier

shared = io.BytesIO(b'x' * 64)
N_W, N_R = 2, 8
barrier = Barrier(N_W + N_R)

def writer():
    barrier.wait()
    for i in range(20000):
        try:
            shared.seek(0)
            shared.write(b'a' * (64 + (i & 63)))
        except Exception:
            pass

def reader():
    barrier.wait()
    for _ in range(20000):
        try:
            shared.seek(0)
            shared.read()
        except Exception:
            pass

if __name__ == "__main__":
    threads = [Thread(target=writer) for _ in range(N_W)]
    threads += [Thread(target=reader) for _ in range(N_R)]
    for t in threads: t.start()
    for t in threads: t.join()

TSAN Report:

==================
WARNING: ThreadSanitizer: data race (pid=2292869)
  Read of size 8 at 0x7fffbc080190 by thread T2:
    #0 _PyObject_MiRealloc /cpython/Objects/obmalloc.c:360:13 
    #1 PyObject_Realloc /cpython/Objects/obmalloc.c:1731:12 
    #2 _PyBytes_Resize /cpython/Objects/bytesobject.c:3389:9 
    #3 resize_buffer_lock_held /cpython/./Modules/_io/bytesio.c:176:13 
    #4 write_bytes_lock_held /cpython/./Modules/_io/bytesio.c:216:13 
    #5 _io_BytesIO_write_impl /cpython/./Modules/_io/bytesio.c:782:20 
    #6 bytesio_setstate_lock_held /cpython/./Modules/_io/bytesio.c:927:14 
    #7 bytesio_setstate /cpython/./Modules/_io/bytesio.c:980:11 
    #8 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h:4261:35 
    ...
  Previous atomic write of size 8 at 0x7fffbc080190 by thread T8:
    #0 _Py_atomic_compare_exchange_ssize /cpython/./Include/cpython/pyatomic_gcc.h:130:10 
    #1 _Py_DecRefSharedIsDead /cpython/Objects/object.c:406:15 
    #2 _Py_DecRefSharedDebug /cpython/Objects/object.c:425:9 
    #3 _Py_DecRefShared /cpython/Objects/object.c:433:5 
    #4 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h 
    ...
SUMMARY: ThreadSanitizer: data race /cpython/Objects/obmalloc.c:360:13 in _PyObject_MiRealloc

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

pythongh-151640: add NEWS entry

pythongh-151640: fix NEWS entry filename

pythongh-151640: simplify BytesIO exports checks

pythongh-151640: avoid sharing BytesIO buffer in free-threaded builds

pythongh-151640: add NEWS entry

pythongh-151640: fix NEWS entry filename

pythongh-151640: simplify BytesIO exports checks

pythongh-151640: avoid sharing BytesIO buffer in free-threaded builds

pythongh-151640: add NEWS entry

pythongh-151640: fix NEWS entry filename

pythongh-151640: simplify BytesIO exports checks

pythongh-151640: Track sharing of BytesIO buffer in free-threaded builds

gh-151640: Track sharing of `BytesIO` buffer in free-threaded builds (G…

[3.15] gh-151640: Track sharing of `BytesIO` buffer in free-threaded …

[3.14] gh-151640: Track sharing of `BytesIO` buffer in free-threaded …

I think this could be closed now. I fixed the trivial issues with the backports for 3.14 and 3.15. Thanks for the PR!

Navigation Menu

Repository navigation

Data race between `resize_buffer_lock_held` and `_Py_DecRefShared` on a shared `BytesIO` buffer #151640

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	if (size > 1 &&
	self->pos == 0 && size == PyBytes_GET_SIZE(self->buf) &&
	FT_ATOMIC_LOAD_SSIZE_RELAXED(self->exports) == 0) {
	self->pos += size;
	return Py_NewRef(self->buf);
	}

	if (SHARED_BUF(self)) {
	if (unshare_buffer_lock_held(self, alloc) < 0)
	return -1;
	}
	else {
	if (_PyBytes_Resize(&self->buf, alloc) < 0)
	return -1;
	}

Data race between resize_buffer_lock_held and _Py_DecRefShared on a shared BytesIO buffer #151640

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

nascheme commented on Jul 9, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Data race between `resize_buffer_lock_held` and `_Py_DecRefShared` on a shared `BytesIO` buffer #151640