[C API] Add an efficient public PyUnicodeWriter API

Feature or enhancement

Creating a Python string object in an efficient way is complicated. Python has private _PyUnicodeWriter API. It's being used by these projects:

Affected projects (5):

Cython (3.0.9)
asyncpg (0.29.0)
catboost (1.2.3)
frozendict (2.4.0)
immutables (0.20)

I propose making the API public to promote it and help C extensions maintainers to write more efficient code to create Python string objects.

API:

typedef struct PyUnicodeWriter PyUnicodeWriter;

PyAPI_FUNC(PyUnicodeWriter*) PyUnicodeWriter_Create(void);
PyAPI_FUNC(void) PyUnicodeWriter_Discard(PyUnicodeWriter *writer);
PyAPI_FUNC(PyObject*) PyUnicodeWriter_Finish(PyUnicodeWriter *writer);

PyAPI_FUNC(void) PyUnicodeWriter_SetOverallocate(
    PyUnicodeWriter *writer,
    int overallocate);

PyAPI_FUNC(int) PyUnicodeWriter_WriteChar(
    PyUnicodeWriter *writer,
    Py_UCS4 ch);
PyAPI_FUNC(int) PyUnicodeWriter_WriteUTF8(
    PyUnicodeWriter *writer,
    const char *str,  // decoded from UTF-8
    Py_ssize_t len);  // use strlen() if len < 0
PyAPI_FUNC(int) PyUnicodeWriter_Format(
    PyUnicodeWriter *writer,
    const char *format,
    ...);

// Write str(obj)
PyAPI_FUNC(int) PyUnicodeWriter_WriteStr(
    PyUnicodeWriter *writer,
    PyObject *obj);

// Write repr(obj)
PyAPI_FUNC(int) PyUnicodeWriter_WriteRepr(
    PyUnicodeWriter *writer,
    PyObject *obj);

// Write str[start:end]
PyAPI_FUNC(int) PyUnicodeWriter_WriteSubstring(
    PyUnicodeWriter *writer,
    PyObject *str,
    Py_ssize_t start,
    Py_ssize_t end);

The internal writer buffer is overallocated by default. PyUnicodeWriter_Finish() truncates the buffer to the exact size if the buffer was overallocated.

Overallocation reduces the cost of exponential complexity when adding short strings in a loop. Use PyUnicodeWriter_SetOverallocate(writer, 0) to disable overallocation just before the last write.

The writer takes care of the internal buffer kind: Py_UCS1 (latin1), Py_UCS2 (BMP) or Py_UCS4 (full Unicode Character Set). It also implements an optimization if a single write is made using PyUnicodeWriter_WriteStr(): it returns the string unchanged without any copy.

Example of usage (simplified code from Python/unionobject.c):

static PyObject *
union_repr(PyObject *self)
{
    unionobject *alias = (unionobject *)self;
    Py_ssize_t len = PyTuple_GET_SIZE(alias->args);

    PyUnicodeWriter *writer = PyUnicodeWriter_Create();
    if (writer == NULL) {
        return NULL;
    }

    for (Py_ssize_t i = 0; i < len; i++) {
        if (i > 0 && PyUnicodeWriter_WriteUTF8(writer, " | ", 3) < 0) {
            goto error;
        }
        PyObject *p = PyTuple_GET_ITEM(alias->args, i);
        if (PyUnicodeWriter_WriteRepr(writer, p) < 0) {
            goto error;
        }
    }
    return PyUnicodeWriter_Finish(writer);

error:
    PyUnicodeWriter_Discard(writer);
    return NULL;
}

Linked PRs

pythongh-119182: Optimize PyUnicode_FromFormat() UTF-8 decoder

The main problem with the current private PyUnicodeWriter C API is that it requires allocating the PyUnicodeWriter value on the stack, but its layout is an implementation detail, and exposing such API would prevent future changes. The proposed new C API allocates the data in dynamic memory, which makes it more portable and future proof. But this can add additional overhead. Also, if we use dynamic memory, why not make PyUnicodeWriter a subclass of PyObject? Then Py_DECREF could be used to destroy it, we could store multiple writers in a collection, and we can even provide Python interface for it.

The proposed new C API allocates the data in dynamic memory, which makes it more portable and future proof. But this can add additional overhead.

I ran benchmarks and using the proposed public API remains interesting in terms of performance: see benchmarks below.

Also, if we use dynamic memory, why not make PyUnicodeWriter a subclass of PyObject? Then Py_DECREF could be used to destroy it, we could store multiple writers in a collection, and we can even provide Python interface for it.

Adding a Python API is appealing, but I prefer to restrict this discussion to a C API and only discuss later the idea of exposing it at the Python level.

For the C API, I don't think that Py_DECREF() semantics and inheriting from PyObject are really worth it.

pythongh-119182: Optimize PyUnicode_FromFormat() UTF-8 decoder

I renamed functions:

PyUnicodeWriter_WriteString() => PyUnicodeWriter_WriteUTF8(): API with const char *str.
PyUnicodeWriter_WriteStr() => PyUnicodeWriter_WriteString(): API with PyObject *str.
PyUnicodeWriter_FromFormat() => PyUnicodeWriter_Format().

@encukou

@encukou:

I see the PR hides underscored API that some existing projects use. I thought we weren't doing that any more.

Right, I would like to hide/remove the internal API from the public C API in Python 3.14 while adding the new public C API. The private _PyUnicodeWriter API exposes the _PyUnicodeWriter structure (members). Its API is more complicated and more error-prone.

I prepared a PR for pythoncapi-compat to check that it's possible to implement the new API on Python 3.6-3.13: python/pythoncapi-compat#95

pythongh-119182: Add PyUnicodeWriter C API

There is some confusion with names. The String suffix usually means the C string (const char *) argument. Str is only used in PyObject_Str() which is the C analogue of the str() function.

So, for consistency we should use PyUnicodeWriter_WriteString() for writing the C string. This left us with the question what to do with Python strings. PyUnicodeWriter_WriteStr() implies that str() is called for argument. Even if we add such API, it is worth to have also a more restricted function which fails if non-string is passed by accident.

This left us with the question what to do with Python strings.

We can refer to them as "Unicode", such as: PyUnicodeWriter_WriteUnicode(). Even if the Python type is called "str", in C, it's the PyUnicodeObject: https://docs.python.org/dev/c-api/unicode.html

Unfortunately Unicode as a suffix was used in the legacy C API related to Py_UNICODE *. Currently it is only used in the C API to "unicode-escape" and "raw-unicode-escape", so we could restore it with a new meaning, but it will be the first case of using it in this role.

Perhaps we can just omit any suffix and use PyUnicodeWriter_Write()?

About bikeshedding. PyPy provides __pypy__.builders.StringBuilder with append() and build() methods. Do you think that "String Builder" with append and build methods API (names) makes more sense than "Unicode Writer" with write and finish methods?

I'd prefer a PyUnicodeWriter_WriteStr that calls PyObject_Str() (which should be cheap for actual PyUnicode objects). It would pair well with the proposed future additions, PyUnicodeWriter_WriteRepr & PyUnicodeWriter_WriteAscii :)

PyUnicodeWriter_WriteSubstring can still take PyUnicode only.

PyUnicodeWriter_WriteUTF8 is a good name. Do you want to support zero-terminated strings (e.g. by passing -1 as the length)?

I'd prefer a PyUnicodeWriter_WriteStr that calls PyObject_Str() (which should be cheap for actual PyUnicode objects). It would pair well with the proposed future additions, PyUnicodeWriter_WriteRepr & PyUnicodeWriter_WriteAscii :)

As written previously, you can already use:

PyUnicodeWriter_Format("%S", obj) to write str(obj).
PyUnicodeWriter_Format("%R", obj) to write repr(obj).
PyUnicodeWriter_Format("%A", obj) to write ascii(obj).

Currently, there is no optimization for these code paths. It's the same as creating a temporary string, write the string, delete the string. It's just a convenient API for that. Later we can imagine further optimizations.

Proposed PyUnicodeWriter_WriteStr() / PyUnicodeWriter_WriteString() (not sure about the name) fails with TypeError if the argument is not a Python str object. I'm only looking for a good name for the name function. I don't want to call PyObject_Str(). The API is really designed for performance. It should do at least work as possible and have a straightforward API.

If later we consider that a new function would be added, I would prefer PyUnicodeWriter_WriteObjectStr() name for str(obj).

PyUnicodeWriter_WriteUTF8 is a good name. Do you want to support zero-terminated strings (e.g. by passing -1 as the length)?

I didn't write the API documentation yet. It's already supported, passing -1 already calls strlen().

PyUnicodeWriter_WriteSubstring can still take PyUnicode only.

Right, it raises TypeError if the argument is not a Python str objet. Same than PyUnicodeWriter_WriteString().

IMO, the best name is PyUnicodeWriter_WriteStr, except it's a bit ambiguous -- people might expect it to call str(). We can solve the ambiguity by simply making it do that, as a convenience to the user. It won't affect performance in any meaningful way.

IMO, the best name is PyUnicodeWriter_WriteStr, except it's a bit ambiguous -- people might expect it to call str().

I don't see why users would expect that. I don't know any existing API with a similar name which call str(), only PyObject_Str() calls it. If it's ambiguous, we can make it explicit in the documentation.

It won't affect performance in any meaningful way.

It's not about performance, but the API. I want a function to only write a string, and nothing else.

It's not about performance, but the API. I want a function to only write a string, and nothing else.

But why? This API feels more like print() (which implicitly calls str() if needed) or f-string interpolation (which does something similar) and less like TextIO.write() (which insists on a str instance). I like this convenience.

My issue is that my proposed API is based on an existing implementation which is around for 12 years. It's uneasy for me to think "ouf of the box" to design a new better API, but that's why I opened this discussion :-) To get other opinions to help me to design a better usable API.

If the majority prefers calling str(), ok, let's switch to that for PyUnicodeWriter_WriteStr().

I checked the Python code base, there are a few code places using repr() with a writer: dict, list, tuple, union, context, token, etc. So I propose to add also PyUnicodeWriter_WriteRepr().

Update:

Rename PyUnicodeWriter_Free() to PyUnicodeWriter_Discard().
Rename PyUnicodeWriter_WriteString() to PyUnicodeWriter_WriteStr(): the function now calls str(obj).
Add PyUnicodeWriter_WriteRepr(): call repr(obj).

I opened an issue for the C API Working Group: capi-workgroup/decisions#27

pythongh-119182: Add PyUnicodeWriter C API

pythongh-119182: Decode PyUnicode_FromFormat() format from UTF-8

pythongh-119182: Use strict error handler in PyUnicode_FromFormat()

pythongh-119182: Add PyUnicodeWriter_WriteWideChar()

gh-119182: Add PyUnicodeWriter C API (#119184)

I opened an issue for the C API Working Group: capi-workgroup/decisions#27

API approved. I merged my PR.

pythongh-119182: Add PyUnicodeWriter_DecodeUTF8Stateful()

I ran again #119182 (comment) benchmark: PyUnicodeWriter is 1.8x faster than string concatenation.

$ env/bin/python bench.py 
.....................
bench_concat: Mean +- std dev: 1.13 us +- 0.02 us
.....................
bench_writer: Mean +- std dev: 632 ns +- 22 ns

pythongh-119182: Optimize PyUnicode_FromFormat()

pythongh-119182: Use public PyUnicodeWriter API in union_repr()

pythongh-119182: Use public PyUnicodeWriter API in ga_repr()

gh-119182: Use public PyUnicodeWriter API in ga_repr() (#120799)

gh-119182: Use public PyUnicodeWriter API in union_repr() (#120797)

gh-119182: Optimize PyUnicode_FromFormat() (#120796)

pythongh-119182: Use public PyUnicodeWriter in contextvar_tp_repr()

gh-119182: Use public PyUnicodeWriter in contextvar_tp_repr() (#120809)

gh-119182: Add PyUnicodeWriter_DecodeUTF8Stateful() (#120639)

pythongh-119182: Rewrite PyUnicodeWriter tests in Python

gh-119182: Rewrite PyUnicodeWriter tests in Python (#120845)

pythongh-119182: Add PyUnicodeWriter_WriteUCS4() function

pythongh-119182: Use PyUnicodeWriter_WriteWideChar()

gh-119182: Use PyUnicodeWriter_WriteWideChar() (#120851)

pythongh-119182: Add checks to PyUnicodeWriter APIs

gh-119182: Add checks to PyUnicodeWriter APIs (#120870)

gh-119182: Add PyUnicodeWriter_WriteUCS4() function (#120849)

The API was added, it comes with its test suite (in Python!), and many string types are now supported (UTF-8, wide string, UCS4, etc.). I close the issue.

pythongh-119182: Add PyUnicodeWriter C API (python#119184)

pythongh-119182: Use public PyUnicodeWriter API in ga_repr() (python#…

pythongh-119182: Use public PyUnicodeWriter API in union_repr() (pyth…

pythongh-119182: Optimize PyUnicode_FromFormat() (python#120796)

pythongh-119182: Use public PyUnicodeWriter in contextvar_tp_repr() (p…

pythongh-119182: Add PyUnicodeWriter_DecodeUTF8Stateful() (python#120639

pythongh-119182: Rewrite PyUnicodeWriter tests in Python (python#120845)

pythongh-119182: Use PyUnicodeWriter_WriteWideChar() (python#120851)

pythongh-119182: Add checks to PyUnicodeWriter APIs (python#120870)

pythongh-119182: Add PyUnicodeWriter_WriteUCS4() function (python#120849

pythongh-119182: Add PyUnicodeWriter C API (python#119184)

pythongh-119182: Use public PyUnicodeWriter API in ga_repr() (python#…

pythongh-119182: Use public PyUnicodeWriter API in union_repr() (pyth…

pythongh-119182: Optimize PyUnicode_FromFormat() (python#120796)

pythongh-119182: Use public PyUnicodeWriter in contextvar_tp_repr() (p…

pythongh-119182: Add PyUnicodeWriter_DecodeUTF8Stateful() (python#120639

pythongh-119182: Rewrite PyUnicodeWriter tests in Python (python#120845)

pythongh-119182: Use PyUnicodeWriter_WriteWideChar() (python#120851)

pythongh-119182: Add checks to PyUnicodeWriter APIs (python#120870)

pythongh-119182: Add PyUnicodeWriter_WriteUCS4() function (python#120849

See also #121710 : [C API] Add PyBytesWriter API.

pythongh-119182: Add PyUnicodeWriter C API (python#119184)

pythongh-119182: Use public PyUnicodeWriter API in ga_repr() (python#…

pythongh-119182: Use public PyUnicodeWriter API in union_repr() (pyth…

pythongh-119182: Optimize PyUnicode_FromFormat() (python#120796)

pythongh-119182: Use public PyUnicodeWriter in contextvar_tp_repr() (p…

pythongh-119182: Add PyUnicodeWriter_DecodeUTF8Stateful() (python#120639

pythongh-119182: Rewrite PyUnicodeWriter tests in Python (python#120845)

pythongh-119182: Use PyUnicodeWriter_WriteWideChar() (python#120851)

pythongh-119182: Add checks to PyUnicodeWriter APIs (python#120870)

pythongh-119182: Add PyUnicodeWriter_WriteUCS4() function (python#120849

pythongh-119182: Complete PyUnicodeWriter documentation

gh-119182: Complete PyUnicodeWriter documentation (#127607)

pythongh-119182: Complete PyUnicodeWriter documentation (python#127607)

pythongh-119182: Use public PyUnicodeWriter in wrap_strftime()

pythongh-119182: Use public PyUnicodeWriter in time_strftime()

pythongh-119182: Use public PyUnicodeWriter in ast_unparse.c

gh-119182: Use public PyUnicodeWriter in time_strftime() (#129207)

gh-119182: Use public PyUnicodeWriter in wrap_strftime() (#129206)

pythongh-119182: Use public PyUnicodeWriter in Python-ast.c

gh-119182: Use public PyUnicodeWriter in ast_unparse.c (#129208)

gh-119182: Use public PyUnicodeWriter in Python-ast.c (#129209)

pythongh-119182: Use public PyUnicodeWriter in stringio.c

gh-119182: Use public PyUnicodeWriter in stringio.c (#129243)

pythongh-119182: Use public PyUnicodeWriter in _json.c

gh-119182: Use public PyUnicodeWriter in _json.c (#129249)

Navigation Menu

Repository navigation

[C API] Add an efficient public PyUnicodeWriter API #119182

Feature or enhancement

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[C API] Add an efficient public PyUnicodeWriter API #119182

Description

Feature or enhancement

Linked PRs

Activity

serhiy-storchaka commented on May 22, 2024

vstinner commented on May 22, 2024

vstinner commented on May 22, 2024

vstinner commented on May 22, 2024

serhiy-storchaka commented on May 23, 2024

vstinner commented on May 23, 2024

serhiy-storchaka commented on May 23, 2024

vstinner commented on May 23, 2024

encukou commented on May 23, 2024

vstinner commented on May 23, 2024

vstinner commented on May 23, 2024

encukou commented on May 23, 2024

vstinner commented on May 23, 2024

gvanrossum commented on May 23, 2024

vstinner commented on May 24, 2024

vstinner commented on May 24, 2024

vstinner commented on May 24, 2024

vstinner commented on Jun 17, 2024

vstinner commented on Jun 18, 2024

vstinner commented on Jun 27, 2024

vstinner commented on Jul 13, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions