UnicodeDecodeError or SystemError when using f-string with lambda and non-ASCII characters

Bug report

Bug description:

When using an f-string in combination with lambda functions that return non-ASCII characters, either a UnicodeDecodeError or a SystemError is raised, depending on the specific modification made to the code. These errors do not occur consistently, but rather vary based on how the lambdas and string content are altered.

Steps to Reproduce

def test1(foo, bar):
    return ""

def test2():
    return f"{test1(
        foo=lambda: '、、、、、、、、、、、、、、、、、',
        bar=lambda: 'abcdefghijklmnopqrstuvwxyz 123456789 123456789',
    )}"

Run the code with python <file.py>, which triggers the following error:

UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 28-29: unexpected end of data

Expected Behavior

The code should execute without raising any errors.

Actual Behavior

The following behaviors are observed when making the specified modifications. Each of these cases is independent:

Removing one 、 character from the first string (foo): No error.
Removing all 、 characters from the first string: SystemError: Negative size passed to PyUnicode_New.
Removing foo= (i.e., not passing foo by keyword): No error.
Removing lambda: (i.e., making either argument or both into a str type instead of a Callable[[], str]): No error.
Removing any character from the second string (bar): No error.

Other Relevant Information

The bug does not reproduce in Python 3.13.1, but it does reproduce in Python 3.13.2 and 3.12.9.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

@lysnikolaou

It seems to be a parser bug.

cc @lysnikolaou @pablogsal

On it

The problem was introduced by:

❯ git bisect bad
a3797492179c249417a06d2499a7d535d453ac2c is the first bad commit
commit a3797492179c249417a06d2499a7d535d453ac2c (HEAD)
Author: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
Date:   Wed Jan 22 01:50:22 2025 +0100

    [3.13] gh-129093: Fix f-string debug text sometimes getting cut off when expression contains `!` (GH-129159) (#129163)

    gh-129093: Fix f-string debug text sometimes getting cut off when expression contains `!` (GH-129159)
    (cherry picked from commit 767cf708449fbf13826d379ecef64af97d779510)

    Co-authored-by: Tomas R <tomas.roun8@gmail.com>

 Lib/test/test_fstring.py                                                          | 18 ++++++++++++++++++
 Misc/NEWS.d/next/Core_and_Builtins/2025-01-21-23-35-41.gh-issue-129093.0rvETC.rst |  2 ++
 Parser/lexer/lexer.c                                                              |  4 +---
 3 files changed, 21 insertions(+), 3 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Core_and_Builtins/2025-01-21-23-35-41.gh-issue-129093.0rvETC.rst

pythongh-130618: Fix parser error when using lambdas inside f-strings

gh-130618: Fix parser error when using lambdas inside f-strings (#130638

pythongh-130618: Fix parser error when using lambdas inside f-strings (…

[3.13] pythongh-130618: Fix parser error when using lambdas inside f-…

[3.12] pythongh-130618: Fix parser error when using lambdas inside f-…

[3.13] gh-130618: Fix parser error when using lambdas inside f-strings (

[3.12] gh-130618: Fix parser error when using lambdas inside f-strings (

@gaesa

Thanks for the report @gaesa

Navigation Menu

Repository navigation

`UnicodeDecodeError` or `SystemError` when using `f-string` with `lambda` and non-ASCII characters #130618

Bug report

Bug description:

Steps to Reproduce

Expected Behavior

Actual Behavior

Other Relevant Information

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

UnicodeDecodeError or SystemError when using f-string with lambda and non-ASCII characters #130618

Description

Bug report

Bug description:

Steps to Reproduce

Expected Behavior

Actual Behavior

Other Relevant Information

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

vstinner commented on Feb 27, 2025

pablogsal commented on Feb 27, 2025

pablogsal commented on Feb 27, 2025

pablogsal commented on Mar 1, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`UnicodeDecodeError` or `SystemError` when using `f-string` with `lambda` and non-ASCII characters #130618