Skip to content

Preserve offsets of repeated sentences after whitespace - #601

Open
FanWu-ai wants to merge 1 commit into
sloria:devfrom
FanWu-ai:fix-repeated-sentence-offsets-20261011
Open

FanWu-ai wants to merge 1 commit into
sloria:devfrom
FanWu-ai:fix-repeated-sentence-offsets-20261011

Conversation

@FanWu-ai

Copy link
Copy Markdown

Sentence offsets drift behind the input because the search cursor advances by sentence length without accounting for intervening whitespace. For seven repetitions of Yes., the sixth sentence incorrectly reuses the fifth sentence's start index.

Advance the cursor to the actual end of the current sentence. Add eight regression cases covering repeated text, long gaps, tabs, blank lines, CRLF, Unicode and unaffected controls, including serialized offsets. The AUTHORS entry follows the contribution guide.

Validation:

  • Pristine full suite: 224 passed, 1 existing unconditional skip. New tests on original source: 5 failed, 3 passed.
  • Final full tox matrix (Python 3.10–3.14 and configured py310-lowest): 232 passed, 1 existing skip in each of six environments.
  • All six pinned hooks, clean docs build, distributions, strict Twine and diff whitespace checks pass.
  • 525 differential cases match NLTK sentence spans; an additional independent 1,000 mixed-sentence cases pass.

The configured py310-lowest environment actually installs locked NLTK 3.10.3, so this does not claim NLTK 3.9 coverage. Docs retain two existing duplicate-description warnings. Supplemental Pyright reports the same 200 errors and 4 warnings before and after; it is not a configured local/CI gate. Official NLTK corpora were installed manually with index hashes verified, without overriding downloader security checks.

AI assistance: OpenAI Codex was used to investigate, implement and test this fix. No human review is claimed. Hosted CI has not yet run for this change.

AI assistance: OpenAI Codex was used for investigation, implementation, regression tests and automated validation.
@FanWu-ai
FanWu-ai marked this pull request as ready for review October 11, 2026 17:47

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant