Self Checks

• I have searched for existing issues (including closed ones).
• I confirm that I am using English to submit this report.
• Non-English title submissions will be closed directly.
• I have not modified this template.

Describe the problem

The single string field parser_config.delimiter is consumed by seven different parsing implementations depending only on the file extension. They disagree on the parsing rule, at least one silently makes the shipped default value a no-op, and the whitespace handling has five distinct bugs (see "Whitespace-specific gotchas" below).

Affected functions, all of which read parser_config.delimiter:

Concrete user-visible bugs

1. The shipped default is a no-op for markdown.
   The default for .md/.mdx/.markdown is parser_config.get("delimiter", "\n!?;。；！？") (see rag/app/naive.py:1132). MarkdownElementExtractor.get_delimiters only matches backtick-wrapped tokens, so the default delimiter string is silently dropped. The markdown path splits purely by markdown structure (headers/code blocks/lists); the user's delimiter setting is ignored.

2. Mixing bare chars with backtick-wrapped tokens loses the bare chars in five paths.
   For input \n`##`; (the example in the tooltip), naive_merge, naive_merge_with_images, and _build_cks enter their has_custom branch and drop \n and ;, keeping only ##. PDF/DOCX/email/book files chunk only on ## until chunk_token_num is reached. The same string is interpreted correctly by parser_txt.

3. TxtParser doesn't sort longest-first.
   For input `##`# the regex alternation ##|# matches # first (Python alternation is leftmost-first with no backtracking), so the .txt/code paths mis-split overlapping delimiters. The .md path on the same string sorts longest-first and gets it right.

4. Inconsistent semantics for markdown vs everything else.
   In .md, the delimiter is an additional splitter on top of markdown structure (element extraction is the primary split). In .txt, it is the primary splitter. In .pdf/.docx, it is a sentence-boundary-aware splitter inside naive_merge. Same field name, three different roles.

5. Duplicates aren't deduped in get_delimiters.
   Input `a`a`a` produces regex a|a|a instead of a. Cosmetic, but symptomatic of the duplication.

6. Case-insensitive delimiter matching (re.I). Tracked separately as [Bug] Delimiter parsing uses re.I (case-insensitive), causing unintended splits on both cases of the same character #17384 — applies only to get_delimiters and parser_txt.

Whitespace-specific gotchas

The parser does handle whitespace characters as delimiters (re.escape(" ") → "\\ ", re.escape("\n") → "\\n", etc.) but with several sharp edges. Empirical results from running the three parser variants against text = "alpha beta\tgamma\n\ndelta epsilon zeta\n":

7. Bare whitespace is silently dropped by markdown and has_custom.
   This is the same bug as impl tags api #2 above, but for whitespace. A user who pastes a paragraph break \n\n for a .pdf gets the same splits as a single newline because the bare-char dedupe-via-alternation produces \\\n|\\\n. The correct input for paragraph splitting is `\n\n` (backtick-wrapped), but nothing in the UI tells users this.

8. Double-whitespace bare form is wrong.
   \n\n (bare two newlines) and \t\t (bare two tabs) and (bare two spaces) all collapse to the single-char equivalent in the bare-char fallback path. Multi-character whitespace delimiters MUST be backtick-wrapped to work correctly. The frontend gives no hint about this.

9. CRLF normalization only happens in naive_merge.
   naive_merge line 1163 normalizes line endings before splitting:

   sections = [(s.replace("\r\n", "\n").replace("\r", "\n"), pos) for s, pos in sections]

   But _build_cks, naive_merge_with_images, parser_txt, and markdown_parser have no equivalent normalization. On Windows-line-ending files, parser_txt and _build_cks will produce double-splits (one at \r, one at \n) for any user with a \r or \r\n in their delimiter.

10. The frontend only round-trips \n / \t / \r.
    web/src/components/delimiter-form-field.tsx translates the literal 2-char escape sequences to real characters on input and back again on display, but only for those three. Other whitespace escapes survive as literal characters:

    User types	Stored as	Works as delimiter?
    \n	real newline char	✓ (split on every newline)
    \t	real tab char	✓ (split on every tab)
    \r	real CR char	✓ (but see CRLF double-split above)
    \f	literal \f (2 chars)	matches literal backslash-f in text, not form-feed
    \v	literal \v (2 chars)	matches literal backslash-v, not vertical-tab
    \u00a0	literal \u00a0 (6 chars)	matches literal backslash-u-0-0-a-0, not NBSP
    \\n (4 source chars)	literal \\n (3 chars)	matches literal \n in text
    An actual NBSP pasted in	real \u00a0 char	✓

    So the supported whitespace escapes are only \n, \t, \r. Other whitespace requires either pasting the actual character or using the API.

11. Whitespace-only delimiters are invisible in the input.
    A delimiter value of " " (single space) makes the input box look empty — the value is stored as a single space character, but the user can't tell whether they typed anything. There's no character count, no separator visualization. This is why users default to multi-character delimiters and then trip on bug init python part #7.

Reproduction

# In a Python shell with the same modules:
from rag.nlp import naive_merge, get_delimiters
from deepdoc.parser.txt_parser import RAGFlowTxtParser
from deepdoc.parser.markdown_parser import MarkdownElementExtractor

s = "\n`##`;"   # what the tooltip tells users to type

print(get_delimiters(s))                 # backslash-n + ## + ;
print(naive_merge(["x##y"], 128, s))      # only '##' is used
print(RAGFlowTxtParser.parser_txt("x##y\n", 128, s))  # all three used
print(MarkdownElementExtractor("x##y").extract_elements(s))  # only '##'

The four functions return four different sets of effective delimiters for the same input string. The same divergence applies to whitespace: bare \n works in parser_txt but is dropped by markdown and naive_merge's has_custom branch.

Proposed solution

Collapse all six Python implementations into one shared helper (e.g. in rag/nlp/__init__.py or a new rag/nlp/delim.py) and call it from every site. The helper should:

1. Accept a string.
2. Treat anything between matching ` characters as one multi-character delimiter (escape its content with re.escape).
3. Treat every character outside backticks as its own single-character delimiter.
4. Deduplicate.
5. Sort longest-first so ## matches before #.
6. Return the compiled regex pattern, or "" for empty / whitespace-only input.
7. Not use re.I — see [Bug] Delimiter parsing uses re.I (case-insensitive), causing unintended splits on both cases of the same character #17384.
8. Apply CRLF/CR normalization to the input string once at the top of the helper, before parsing. \r\n → \n, standalone \r → \n. This single change fixes the Windows-line-ending double-split bug across all five paths.

The .md path additionally needs a fallback when the helper returns "": it should fall back to the existing markdown structure-based extraction (_extract_delimited_elements with no pattern), which is what already happens for the default delimiter. That is the correct behavior for the markdown case — the user has not opted into extra splitting — and matches what the markdown parser does today for the bare-chars-only default.

The proposed behavior (extended to include whitespace cases):

Behavioral note on bare multi-whitespace: collapsing "\n\n" to "\n" (via dedupe) is a change from today's behavior where the regex alternation \\\n|\\\n happens to match the same text. The output for any given text is identical, but if a downstream caller introspects the produced delimiter list (none do today), the result changes. The acceptance test should pin this down explicitly.

Scope

• Six implementations of the same parser → one helper.
• CRLF normalization centralized in the helper.
• No API or schema change.
• No behavior change for inputs that previously worked; only the divergent cases become consistent.

Files to change

• rag/nlp/__init__.py — remove the three inline re.finditer snippets from naive_merge, naive_merge_with_images, _build_cks; remove the get_delimiters function (or re-export it from the new helper for backwards compat); have all three call the helper. Also drop the per-call-site CRLF normalization in naive_merge (now done in the helper).
• deepdoc/parser/txt_parser.py — replace the inline regex with a call to the helper. Drop the encode("utf-8").decode("unicode_escape").encode("latin1").decode("utf-8") dance: that round-trip was defending against users passing literal \n (2 chars) over the API; the frontend already converts \n → real newline, and the helper does no further escaping, so the dance is redundant.
• deepdoc/parser/markdown_parser.py — replace MarkdownElementExtractor.get_delimiters with a call to the helper. The fallback to structure-based extraction when the helper returns "" should be preserved.
• web/src/components/delimiter-form-field.tsx — extend the escape round-trip table to either (a) include \f, \v, \u00a0, \\ for consistency, or (b) explicitly reject unknown escapes with a validation error. Pick one and document it. Today they are silently mishandled.
• New: rag/nlp/delim.py (or similar) — single source of truth.

Acceptance criteria

• All six sites produce the same regex pattern for the same input string.
• The shipped default "\n!?;。；！？" keeps working for .txt, .pdf, .docx, etc.
• The shipped default for .md/.mdx continues to result in structure-based splitting (no extra delimiter splits), because the helper returns "" for it.
• The tooltip's example \n`##`; produces three effective delimiters regardless of file type.
• A bare-whitespace input (" ", "\t", "\n") splits on every occurrence of that whitespace character.
• A backtick-wrapped whitespace input (`\n\n`, `\t\t`) splits only on the exact N-character whitespace sequence.
• A \r\n-line-ending document is split identically to a \n-line-ending document for every delimiter, regardless of file type.
• Whitespace-only delimiter values render visibly in the input (e.g. a placeholder glyph or character counter). Optional UX criterion; can ship in a follow-up if it bloats this PR.
• New unit tests in test/unit_test/rag/test_naive_merge.py (and a new test/unit_test/rag/test_delim.py) cover the table above plus empty input, dedupe, longest-first, unicode characters, embedded backticks, CRLF normalization, and every escape in the frontend's round-trip table.
• Existing tests in test_naive_merge.py still pass.

Related

• [Docs] "Delimiter for text" tooltip does not explain the meta-delimiter (backticks) or the parsing rule #17382 — tooltip clarity (docs fix, separate concern)
• [Bug] Delimiter parsing uses re.I (case-insensitive), causing unintended splits on both cases of the same character #17384 — case-insensitive matching (re.I), independent small fix
• [Question]: Multiple delimiters for"Delimiter for text" #7436 — user confusion about multiple delimiters
• [Bug]: Delimiter setting not working as intended #4704 — "Delimiter setting not working" bug
• [Question]: For the Delimiter for text with multiple characters, which of the following usages of backticks is correct? #9680 — multiple-character delimiter question