--semantic never breaks a sentence that ends in a code span or emphasis (SENTENCE_END_RE delimiter class) #68

Claude wrote this but I reviewed it and I'm the one posting it

Summary

--semantic does not break at a sentence boundary when the terminating ./?/! directly follows
an inline code span, bold, italic, or strikethrough. A sentence ending in a plain word, a quoted
string, or a parenthetical breaks correctly, so the failing cases are specifically Markdown markup
delimiters sitting between the last letter and the terminator.

This bites hard in technical prose, where sentences routinely end on a backticked identifier or
filename. In one repo of mine it accounts for 180 unbroken sentence boundaries across 56 of 96
Markdown files — most of the value of --semantic.

Root cause

flowmark/linewrapping/sentence_split_regex.py:

SENTENCE_END_RE = regex.compile(r"(\b\p{L}+[\p{Ll}])([.?!]['\"’”)]?|['\"’”)][.?!]) *$")

The pattern requires the terminator to sit immediately after a letter, optionally separated by one
character from the class ['\"’”)]. That class covers quotes and a closing paren but not the
Markdown emphasis delimiters `, *, _, or ~, so `token`. never matches and the
splitter treats the sentence as unfinished.

Confirmed directly against the heuristic:

from flowmark.linewrapping.sentence_split_regex import heuristic_end_of_sentence as h

h("ends with word.")        # True
h('ends with "quote".')     # True
h("ends with (paren).")     # True
h("ends with `code`.")      # False
h("ends with **bold**.")    # False
h("ends with *ital*.")      # False
h("ends with _ital_.")      # False
h("ends with ~~strike~~.")  # False
h("ends with `code`?")      # False
h("ends with `code`!")      # False

Adding `, *, _, and ~ to that character class looks like it would cover this, though
multi-character delimiters (**, ~~) need more than a single optional character, so the
[...]? may need to become a quantified group.

Reproduction

At the API level:

from flowmark import split_sentences_regex

S = " Then a second sentence of adequate length here. And a third sentence to be sure."

split_sentences_regex("The system uses a token." + S)
# ['The system uses a token.',
#  'Then a second sentence of adequate length here.',
#  'And a third sentence to be sure.']

split_sentences_regex("The system uses a `token`." + S)
# ['The system uses a `token`. Then a second sentence of adequate length here.',
#  'And a third sentence to be sure.']

End to end, with reformat_text(..., semantic=True) or flowmark --semantic --width 100000:

Actual for the code-span case:

The system uses a `token`. Then a second sentence of adequate length here.
And a third sentence to be sure.

Expected:

The system uses a `token`.
Then a second sentence of adequate length here.
And a third sentence to be sure.

Note the second boundary does break, so this is not a whole-paragraph failure — only the boundary
preceded by the code span is skipped.

One repro gotcha

SENTENCE_MIN_LENGTH = 15 means a short two-sentence line such as One two. Three four. stays on
one line regardless of --width. My first attempt at a minimal repro was misleading for this
reason. Every example above is padded well past that threshold, and the plain-word control breaks at
the same length.

Versions

• flowmark v0.7.3 (Python), via uvx --from flowmark flowmark, Linux x86_64.

The Rust port reproduces the same matrix identically (flowmark 0.3.2, base v0.3.2), which fits the
root cause: the two use different Markdown parsers (marko vs comrak) but share this sentence-split
heuristic. Worth a case in the parity corpus from flowmark-rs#60.