PEP 701 – Syntactic formalization of f-strings

Changes in the C tokenizer
Categorize failing tests
Fix failing tests or modify/remove them as needed
Changes in Python tokenizer

Linked PRs

I found that no one has claimed test_eof yet, so I made some work. :)
Failed test case: test_eof.test_eof_with_line_continuation

I looked at its commit history. This test case is a regression test for crash, so it seems like a good choice to keep the case and update the error message directly.

cpython/Lib/test/test_eof.py

Lines 39 to 40 in 72186aa

    
           def test_eof_with_line_continuation(self): 
        
               expect = "unexpected EOF while parsing (<string>, line 1)"

Update unexpected EOF while parsing (<string>, line 1) to (unicode error) 'unicodeescape' codec can't decode bytes in position 0-1: truncated \xXX escape (<string>, line 1)

@pablogsal

@pablogsal in test_type_comments we have not used any f-strings.

@pablogsal

@pablogsal in test_type_comments we have not used any f-strings.

The one failing there is this problem:

======================================================================
FAIL: test_fstring (test.test_type_comments.TypeCommentTests.test_fstring)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_type_comments.py", line 275, in test_fstring
    for tree in self.parse_all(fstring, minver=6):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_type_comments.py", line 239, in parse_all
    with self.assertRaisesRegex(SyntaxError, expected_regex,
AssertionError: SyntaxError not raised : feature_version=(3, 4)

----------------------------------------------------------------------

which I think is a feature version problem.

@pablogsal

@pablogsal in test_type_comments we have not used any f-strings.

The one failing there is this problem:

======================================================================
FAIL: test_fstring (test.test_type_comments.TypeCommentTests.test_fstring)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_type_comments.py", line 275, in test_fstring
    for tree in self.parse_all(fstring, minver=6):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_type_comments.py", line 239, in parse_all
    with self.assertRaisesRegex(SyntaxError, expected_regex,
AssertionError: SyntaxError not raised : feature_version=(3, 4)

----------------------------------------------------------------------

which I think is a feature version problem.

cpython/Lib/test/test_type_comments.py

Line 223 in 4695709

lowest = 4 # Lowest minor version supported

We can just change this to 6, and this test will be pass.
I don't research this problem, but this solution looks like the simplest.
However... supporting syntax of python3.4 && 3.5 looks cinda strange.

We can just change this to 6, and this test will be pass.
I don't research this problem, but this solution looks like the simplest.
However... supporting syntax of python3.4 && 3.5 looks cinda strange.

I don't think that will work because we are not doing version checking anymore. See previous comments. The fix is probably to no pass feature_version.

Looks like no one is analyzing test_exceptions. I will look into it these two days.

4 platforms seem to have met the same problem here.

@isidentical

@isidentical @lysnikolaou i have pushed some rules for error messages, please take a look and complete them with more if you have some spare cycles. With these the failures in test_fstring have decreased notably

I can confirm that the total number of failures has been decrased from 88 to 63. I'll try to see what are the most high impact ones and submit a PR to clear them.

If anyone intends to work on any of the remaining tasks in test_fstring, please double check with this PR (pablogsal#52) since it brings down the total failures to 30 with some explanations/required decisions for the rest.

After looking into the failure in test_exceptions.

check(b'Python = "\xcf\xb3\xf2\xee\xed" +', 1, 18)

Old parser and the new parser raises the same exception (UnicodeDecodeError), but with different col_offset. This is because it was raised by the wrong token.

I would consider it a bug in the old parser. Just as this comment mentions,

cpython/Parser/string_parser.c

Lines 38 to 48 in 64cb1a4

    
                       /* This is needed, in order for the SyntaxError to point to the token t, 
        
                          since _PyPegen_raise_error uses p->tokens[p->fill - 1] for the 
        
                          error location, if p->known_err_token is not set. */ 
        
                       p->known_err_token = t; 
        
                       if (octal) { 
        
                           RAISE_SYNTAX_ERROR("invalid octal escape sequence '\\%.3s'", 
        
                                              first_invalid_escape); 
        
                       } 
        
                       else { 
        
                           RAISE_SYNTAX_ERROR("invalid escape sequence '\\%c'", c); 
        
                       }

the error token was not correctly set in the old parser.

Maybe we should open an issue for the old parse? But the possible fixing might be error-prone, since we might need to keep track of every possible code path.

As for the new parser, I think a change in the test case would be fine.

I am working on an PR to fix test_unparse this weekend.

_PyPegen_concatenate_strings did not implement concatenating empty Constant with FormattedValue, resulting unparse failure.

Hi, I got some bad news.

I have been testing against memory leaks with ./python -m test -j $(nproc) -R :
~30% of the tests failed on current head 270b661

For example,

0:01:01 load avg: 13.77 [157/433/42] test_unittest failed (reference leak)
beginning 9 repetitions
test_unittest leaked [89, 89, 89, 89] references, sum=356
test_unittest leaked [90, 89, 89, 89] memory blocks, sum=357
.......
0:01:16 load avg: 15.00 [185/433/49] test_inspect failed (reference leak)
beginning 9 repetitions
test_inspect leaked [429, 429, 429, 429] references, sum=1716
test_inspect leaked [318, 318, 318, 317] memory blocks, sum=1271

These references are most likely to leak during the compilation (such as using import, using compile/exec/eval, or using ast.parse)

We might need to look into that.

Update: Memory Leakage fixed by commit pablogsal@d8b12e2

The root cause is that someone forgot to use _PyArena_AddPyObject in 3 places.

This is very tricky because _PyArena_AddPyObject was scattered in many subroutines. Sometimes you should add _PyArena_AddPyObject, but sometimes you should not (add will cause a negative ref count).

Just like an old saying, managing memory by hand is so much pain, and also error-prone. This check can be done, by analyzing the PyObject*s registered to the arena by the time the AST was created, but that is a totally different story.

Only 12 test left in test_fstring and we are ready to go!

I'll have a look at them today.

I think we can make it before beta 1. Formalizing fstring error messages (see pablogsal#52 (comment)) can be left to the end, since it is non-functional.

I pushed a commit and we are now down to 8

I think we can make it before beta 1. Formalizing fstring error messages (see pablogsal#52 (comment)) can be left to the end, since it is non-functional.

Yes and no: there may be some design consequences and I am not a fan of deactivating tests to reactivate them later.

There's a small discrepancy around the conversion character between our version and the string parser. The string parser does not allow spaces before or after the conversion character. For example:

>>> f"{'3'!r : >10s}"
  File "<stdin>", line 1
    f"{'3'!r : >10s}"
                     ^
SyntaxError: f-string: expecting '}'

Our version, while not allowing for spaces between ! and the character, allows spaces after.

>>> f"{'3'!r : >10s}"
"       '3'"

What do you think about this? I feel okay about it, although it isn't really consistent with our decision on spaces after the ! character.

What do you think about this? I feel okay about it, although it isn't really consistent with our decision on spaces after the ! character.

I feel that this is fine, and would be very difficult to disallow to be honest because these are different tokens anyway. It won't affect existing code and I don't see any reason to forbid it going foward.

@lysnikolaou

7 tests to go thanks to @lysnikolaou ❤️

With pablogsal#65, we only have two errors to go.

======================================================================
FAIL: test_format_specifier_expressions (test.test_fstring.TestCase.test_format_specifier_expressions) (str='f\'{"s"!r{":10"}}\'')
----------------------------------------------------------------------
  File "<string>", line 1
    f'{"s"!r{":10"}}'
            ^
SyntaxError: f-string: expecting '}'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/lysnikolaou/repos/python/cpython/Lib/test/test_fstring.py", line 32, in assertAllRaise
    with self.assertRaisesRegex(exception_type, regex):
AssertionError: "f-string: invalid conversion character 'r{"': expected 's', 'r', or 'a'" does not match "f-string: expecting '}' (<string>, line 1)"

======================================================================
FAIL: test_format_specifier_expressions (test.test_fstring.TestCase.test_format_specifier_expressions) (str="f'{4:{/5}}'")
----------------------------------------------------------------------
  File "<string>", line 1
    f'{4:{/5}}'
          ^
SyntaxError: f-string: expecting '}'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/lysnikolaou/repos/python/cpython/Lib/test/test_fstring.py", line 32, in assertAllRaise
    with self.assertRaisesRegex(exception_type, regex):
AssertionError: "f-string: invalid syntax" does not match "f-string: expecting '}' (<string>, line 1)"

These are hard to handle correctly, if we have the invalid_replacement_field rule stay the same to catch those expected '}' cases. There are the following 3 options, I guess.

Take more time to figure out whether we can find a smart way to differentiate between these things (can't see something since yesterday)
Be okay with sometimes emitting a expecting '}' error, while there are other syntax errors within the f-string (keeping invalid_replacement_field as-is).
Be okay with sometimes emitting a invalid syntax error, when we could have done a better error (probably making invalid_replacement_field require both a conversion character and a valid format spec).

Thoughts?

Catching all errors seems unrealistic. But I think we do have an opportunity to improve.

By changing the invalid_replacement_field, we will have a better complain message about what to expect, and we can capture the syntax error from the field (yield_expr | star_expressions).

invalid_replacement_field:
    | '{' a='=' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "f-string: expression required before '='") }
    | '{' a=':' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "f-string: expression required before ':'") }
    | '{' a='!' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "f-string: expression required before '!'") }
    | '{' a='}' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "f-string: empty expression not allowed") }
    | '{' (yield_expr | star_expressions) !('=' | '!' | ':' | '}') {
        PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR("f-string: expecting '=', or '!', or ':', or '}'")
    }
    | '{' (yield_expr | star_expressions) "=" !('!' | ':' | '}') {
        PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR("f-string: expecting '!', or ':', or '}'")
    }
    | '{' (yield_expr | star_expressions) "="? invalid_conversion_character
    | '{' (yield_expr | star_expressions) "="? ['!' NAME] !(':' | '}') {
        PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR("f-string: expecting ':' or '}'")
    }
    | '{' (yield_expr | star_expressions) "="? ['!' NAME] [':' fstring_format_spec*] !'}' {
        PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR("f-string: expecting '}'")
    }

With this new grammer,

  File "a.py", line 1
    f'{"s"!r{":10"}}'
            ^
SyntaxError: f-string: expecting ':' or '}'

much better I think.

Totally agree. Good idea! Will you open a PR?

Sure

Hi, there. Will anyone take a look at pablogsal#67?

@isidentical

We are almost there, seems that we still have a small failure:

FAIL: test_random_files (test.test_tokenize.TestRoundtrip.test_random_files)

@isidentical this may be that we lack some changes in the untokenize function

@pablogsal

@pablogsal We lack changes in both the tokenize and the untokenize function. It cannot recognize the new f-string grammar. See pablogsal#67 (comment)

@pablogsal

@pablogsal We lack changes in both the tokenize and the untokenize function. It cannot recognize the new f-string grammar. See pablogsal#67 (comment)

We should be ok excluding the files that use the new funky syntax until we have the new tokenize implementation

Seems that we have a failing tests in the buildbots:

https://buildbot.python.org/all/#/builders/802/builds/760

Looking into the failure.

gh-102856: Initial implementation of PEP 701 (#102855)

pythongh-102856: Skip test_mismatched_parens in WASI builds

gh-102856: Skip test_mismatched_parens in WASI builds (#103633)

pythongh-102856: Skip failing test in test_fstring to fix WASI buildbot

pythongh-102856: Clean some of the PEP 701 tokenizer implementation

fixup! pythongh-102856: Clean some of the PEP 701 tokenizer implement…

gh-102856: Clean some of the PEP 701 tokenizer implementation (#103634)

This code is kind of wrong:

cpython/Parser/tokenizer.c

Lines 2513 to 2517 in 6be7aee

    
           tok->cur = (char *)current_tok->f_string_start; 
        
           tok->cur++; 
        
           tok->line_start = current_tok->f_string_multi_line_start; 
        
           int start = tok->lineno; 
        
           tok->lineno = tok->first_lineno;

This fails when the f-string closes in a different line due to the expression taking multiple lines:

>>> f"blech{
... 1
... +
... 1
... }
... "

Or for example, if you do

>>> f"123{1}
...

In both cases the values of tok->cur and tok->line_start are garbage. I suspect the problem is that we are not updating current_tok->f_string_start and current_tok->f_string_multi_line_start if the f-string expression takes multiple lines and is not triple quotes.

To check this, please compile with --with-address-sanitizer

Merge branch 'main' into superopt

@cmaureir

@cmaureir is going to work on allowing comments as per the specification.

pythongh-102856: Allow comments inside multi-line f-string expresions

@mgmacias95

@mgmacias95 @pablogsal How's the work on the tokenize module going? Are we going to be able to add that until the beta? Is it going to be considered okay to add it later as well?

@lysnikolaou

hi @lysnikolaou, I just created a draft PR (#104323) please take a look.

pythongh-102856: Allow comments inside multi-line f-string expresions

gh-102856: Python tokenizer implementation for PEP 701 (#104323)

gh-102856: Tokenize performance improvement (#104731)

gh-102856: Allow comments inside multi-line f-string expresions (#104006

gh-102856: Add changes related to PEP 701 in 3.12 What's New docs (#1…

pythongh-102856: Add changes related to PEP 701 in 3.12 What's New do…

[3.12] gh-102856: Add changes related to PEP 701 in 3.12 What's New d…

@lysnikolaou

Closing this as we already have a What's New entry, C tokenizer, and Python tokenizer. Let's tackle any small remaining items in separate issues from now on.

Probably we need to alter https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals. @lysnikolaou can you take a go at that?

pythongh-102856: Add missing quote to fix doctest (pythonGH-104852)

gh-102856: Add missing quote to fix doctest (#104852)

@lysnikolaou

Probably we need to alter https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals. @lysnikolaou can you take a go at that?

Sure!

[3.12] gh-102856: Add missing quote to fix doctest (GH-104852) (#104854)

pythongh-102856: Update Formatted string literals docs section after …

pythongh-102856: Update "Formatted string literals" docs section afte…

gh-102856: Update "Formatted string literals" docs section after PEP7…

pythongh-102856: Update "Formatted string literals" docs section afte…

[3.12] gh-102856: Update "Formatted string literals" docs section aft…

Ah great! I didn't know someone would continue PEP 536. Happy that it's happening!

Shouldn't PEP 536 be mentioned in this one?

Ah great! I didn't know someone would continue PEP 536. Happy that it's happening!

Shouldn't PEP 536 be mentioned in this one?

It's mentioned in the PEP:

https://peps.python.org/pep-0701/

Btw as heads up: We don't monitor normally closed issues so is very likely that people won't answer to comments when the issue is closed :)

Thanks! Seems like I missed the mention then. Perfect!

Navigation Menu

Repository navigation

PEP 701 – Syntactic formalization of f-strings #102856

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	def test_eof_with_line_continuation(self):
	expect = "unexpected EOF while parsing (<string>, line 1)"

	/* This is needed, in order for the SyntaxError to point to the token t,
	since _PyPegen_raise_error uses p->tokens[p->fill - 1] for the
	error location, if p->known_err_token is not set. */
	p->known_err_token = t;
	if (octal) {
	RAISE_SYNTAX_ERROR("invalid octal escape sequence '\\%.3s'",
	first_invalid_escape);
	}
	else {
	RAISE_SYNTAX_ERROR("invalid escape sequence '\\%c'", c);
	}

	tok->cur = (char *)current_tok->f_string_start;
	tok->cur++;
	tok->line_start = current_tok->f_string_multi_line_start;
	int start = tok->lineno;
	tok->lineno = tok->first_lineno;

PEP 701 – Syntactic formalization of f-strings #102856

Description

Linked PRs

Activity

CharlieZhao95 commented on Mar 22, 2023

ramvikrams commented on Mar 23, 2023

pablogsal commented on Mar 23, 2023

Eclips4 commented on Mar 23, 2023

pablogsal commented on Mar 23, 2023

sunmy2019 commented on Mar 24, 2023

pablogsal commented on Mar 24, 2023

isidentical commented on Mar 25, 2023

isidentical commented on Mar 25, 2023

sunmy2019 commented on Mar 25, 2023

sunmy2019 commented on Mar 31, 2023

sunmy2019 commented on Mar 31, 2023

pablogsal commented on Apr 6, 2023

lysnikolaou commented on Apr 6, 2023

sunmy2019 commented on Apr 6, 2023

pablogsal commented on Apr 6, 2023

pablogsal commented on Apr 6, 2023

lysnikolaou commented on Apr 6, 2023

pablogsal commented on Apr 6, 2023

pablogsal commented on Apr 6, 2023

lysnikolaou commented on Apr 7, 2023

sunmy2019 commented on Apr 7, 2023

sunmy2019 commented on Apr 7, 2023

lysnikolaou commented on Apr 7, 2023

sunmy2019 commented on Apr 7, 2023

sunmy2019 commented on Apr 11, 2023

pablogsal commented on Apr 13, 2023

sunmy2019 commented on Apr 13, 2023

pablogsal commented on Apr 13, 2023

pablogsal commented on Apr 13, 2023

lysnikolaou commented on Apr 13, 2023

pablogsal commented on Apr 19, 2023

pablogsal commented on Apr 22, 2023

lysnikolaou commented on May 9, 2023

mgmacias95 commented on May 9, 2023

pablogsal commented on May 24, 2023

lysnikolaou commented on May 24, 2023

flying-sheep commented on Oct 2, 2023

pablogsal commented on Oct 3, 2023

flying-sheep commented on Oct 3, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions