Skip to content
This repository was archived by the owner on May 20, 2025. It is now read-only.
This repository was archived by the owner on May 20, 2025. It is now read-only.

Correct C API Usage Logic for NO_GIL Multi-threading #133

@DanielLee343

Description

@DanielLee343

Hi Sam, I wonder what's the correct C API calling logic to implement a multi-threading feature in this no_gil CPython. I'm doing some hacking within Modules/gcmodule.c, that I want to mimic gc_get_objects_impl() but for each GC-traced container PyObject, I further call PyObject_GetIter() to obtain all it's inner objects references it holds. I face no problem when executing this logic between tstate = PyGILState_Ensure() and PyGILState_Release(tstate). But apparently it's holding the GIL.

If I don't hold the GIL, the PyObject_GetIter() internally calls _GC_Malloc(), and will seg faults in return mi_heap_calloc(tstate->heaps[mi_heap_tag_gc], nelem, elsize); since the heap structure is messed up.

Then I noticed on PEP 703, about the thread states. In this no_gil CPython 3.9 version, I guess it would be calling _PyThreadState_Swap() to set the thread state ATTACHED, like this:

void *inspect_module_objs(void *arg)
{
    PyThreadState *tstate = _PyThreadState_GET();
    struct _gilstate_runtime_state *gilstate = &tstate->interp->runtime->gilstate;
    if (_PyThreadState_Swap(gilstate, tstate) != NULL)
    {
        Py_FatalError("non-NULL old thread state");
    }
    // my threading logic that I don't want to hold GIL
    // for loop (...){PyObject_GetIter(each_op) ...}
    // ...
    PyEval_ReleaseThread(tstate);
}

This inspect_module_objs() is called by PyThread_start_new_thread(inspect_module_objs, args);
However, it seg faults at _PyThreadState_Swap() since the tstate == NULL if you don't call PyGILState_Ensure(). If I hold the GIL before calling _PyThreadState_Swap() it then leads to Py_FatalError("non-NULL old thread state") somehow.

FYI, originally I asked on python forum here before they told be no_gil in 3.13 main stream is not completed, thus I would like to ask here. Thanks you.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions