Skip to content

PyArg_ParseTupleAndKeywords() and non-ASCII keyword names #110815

Closed
Closed
PyArg_ParseTupleAndKeywords() and non-ASCII keyword names#110815
Labels
3.11only security fixes3.12only security fixes3.13only security fixestopic-C-APItopic-unicodetype-bugAn unexpected behavior, bug, or errortype-crashA hard crash of the interpreter, possibly with a core dump

Description

@serhiy-storchaka
serhiy-storchaka
opened on Oct 13, 2023

Most of C strings in the C API are implied to be UTF-8 encoded. PyArg_ParseTupleAndKeywords() mostly works with non-ASCII keyword names as they are UTF-8 encoded. Except one case, when you pass argument by keyword with invalid non-ASCII name to a function that has optional parameter with non-ASCII UTF-8 encoded name. In this case you get a crash in the debug build.

It was caused by combination of f4934ea and a83a6a3 (bpo-28701, #72887). Before these changes you simply got inaccurate or even wrong error message.

Examples:

  1. Parameters: "ä"
    Keyword arguments: "ë"
    Old behavior: TypeError "'ë' is an invalid keyword argument for this function"
    Current behavior: crash
    Expected behavior: TypeError "'ë' is an invalid keyword argument for this function"

  2. Parameters: "ä"
    Keyword arguments: "ä"
    Old behavior: TypeError "invalid keyword argument for this function"
    Current behavior: crash
    Expected behavior: TypeError "'ä' is an invalid keyword argument for this function"

  3. Parameters: "ä", "ë"
    Keyword arguments: "ä", "ë"
    Old behavior: TypeError "'ë' is an invalid keyword argument for this function"
    Current behavior: crash
    Expected behavior: TypeError "'ä' is an invalid keyword argument for this function"

In case 1 the pre-bpo-28701 behavior was correct, in case 2 it was correct but not precise (it failed to find the name of invalid keyword argument), in case 3 it was wrong (it found wrong name). In all cases there is a crash currently.

Linked PRs

Activity

  1. serhiy-storchaka
    added
    type-bugAn unexpected behavior, bug, or error
    3.11only security fixes
    type-crashA hard crash of the interpreter, possibly with a core dump
    topic-C-API
    3.12only security fixes
    3.13only security fixes
    on Oct 13, 2023
  2. serhiy-storchaka
    added a commit that references this issue on Oct 13, 2023

    pythongh-110815: Support non-ASCII keyword names in PyArg_ParseTupleA…

    6c154d3
  3. serhiy-storchaka
    added
    topic-unicode
    on Oct 13, 2023

  4. serhiy-storchaka commented on Oct 13, 2023

    @serhiy-storchaka
    serhiy-storchaka
    on Oct 13, 2023
    MemberAuthor

    I am not sure how to classify this issue: as a bug or a new feature. On one hand, it was a regression in 3.7, crash is worse than wrong error message. On other hand, it was not explicitly documented that non-ASCII names are supported, there were no tests, and if it worked, it was only by accident. And it was a long time ago. So, official support of non-ASCII names can be a new feature.

  5. serhiy-storchaka
    added 3 commits that reference this issue on Oct 13, 2023

    pythongh-110815: Support non-ASCII keyword names in PyArg_ParseTupleA…

    5153eb8

    pythongh-110815: Improve tests for PyArg_ParseTupleAndKeywords()

    c41cacb

    gh-110815: Improve tests for PyArg_ParseTupleAndKeywords() (GH-110817)

    548ce09
  6. miss-islington
    added a commit that references this issue on Oct 13, 2023

    pythongh-110815: Improve tests for PyArg_ParseTupleAndKeywords() (pyt…

    dd99f04
  7. serhiy-storchaka
    added a commit that references this issue on Oct 13, 2023

    [3.12] gh-110815: Improve tests for PyArg_ParseTupleAndKeywords() (GH…

    1c44f88

  8. terryjreedy commented on Oct 13, 2023

    @terryjreedy
    terryjreedy
    on Oct 13, 2023
    Member

    Is the issue visible from Python code:

    def f(ä=''): pass

f(ä="ë")
f(ä="ä")

    works. If something like this were to crash, I would say definitely a bugfix to be backported.

  9. serhiy-storchaka commented on Oct 14, 2023

    @serhiy-storchaka
    serhiy-storchaka
    on Oct 14, 2023
    MemberAuthor

    It is an issue with PyArg_ParseTupleAndKeywords(), a C API used for parsing keyword arguments in extension functions. Python defined functions do not use it. Also, it is an issue with argument names, not values. Non-ASCII argument names are forbidden in the core interpreter and the stdlib, but third parties may have reasons to use them.

  10. serhiy-storchaka
    added a commit that references this issue on Oct 14, 2023

    gh-110815: Support non-ASCII keyword names in PyArg_ParseTupleAndKeyw…

    7284e0e
  11. serhiy-storchaka
    closed this as completedon Oct 14, 2023

  12. terryjreedy commented on Oct 15, 2023

    @terryjreedy
    terryjreedy
    on Oct 15, 2023
    Member

    Thanks. Makes sense now.

  13. aisk
    added 2 commits that reference this issue on Feb 11, 2024

    pythongh-110815: Improve tests for PyArg_ParseTupleAndKeywords() (pyt…

    3dd9087

    pythongh-110815: Support non-ASCII keyword names in PyArg_ParseTupleA…

    421748a
  14. Glyphack
    added 2 commits that reference this issue on Sep 2, 2024

    pythongh-110815: Improve tests for PyArg_ParseTupleAndKeywords() (pyt…

    75f7150

    pythongh-110815: Support non-ASCII keyword names in PyArg_ParseTupleA…

    717a68a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.11only security fixes3.12only security fixes3.13only security fixestopic-C-APItopic-unicodetype-bugAn unexpected behavior, bug, or errortype-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions