Skip to content

Fix multi-byte delimiters split across read buffers - #1854

Open
methakon wants to merge 1 commit into
tqdm:masterfrom
methakon:fix/1847-multibyte-delim-split
Open

methakon wants to merge 1 commit into
tqdm:masterfrom
methakon:fix/1847-multibyte-delim-split

Conversation

@methakon

@methakon methakon commented Oct 1, 2026

Copy link
Copy Markdown

Fix multi-byte delimiters split across read buffers

Fixes #1847.

posix_pipe searched only the fresh chunk from each read:

i = tmp.find(delim)

When the delimiter is multi-byte and --buf_size is smaller than it, the delimiter's bytes land in two consecutive reads, so neither chunk contains it. The saved prefix is appended without ever being searched, and the delimiter is missed.

Searching buf + tmp finds it. Because buf is non-empty in exactly the case that matters, the indices have to refer to the concatenation:

combined = buf + tmp
i = combined.find(delim)
if i < 0:
    buf = combined
    break
fp_write(combined[:i + len_delim])
callback(1 if callback_len else combined[:i])
buf = b''
tmp = combined[i + len_delim:]

The record passed to the callback must also come from combined, and the leftover chunk is combined[i + len_delim:]. Using tmp[i + len_delim:] would silently drop the bytes still sitting in buf — which is what a partial fix that only changes the find() does. That version fixed the count but still mis-sliced the record, and it regressed one buffer size; the full form below is correct at every size.

Reproduced

With a 2-byte delimiter and --buf_size 2:

$ printf '1é2é3' | python -m tqdm --delim 'é' --buf-size 2 --bar-format '{n}it'
2it          # expected 3it

$ printf '1é2é3' | python -m tqdm --delim 'é' --buf-size 2 --update --bar-format '{n}it'
SyntaxError: invalid syntax   # while parsing '1é2'

The crash is the same root cause: the first delimiter byte leaked into the record, so literal_eval was handed b'1\xc3\xa92'.

After

--buf_size counting mode --update
1, 2, 3, 4, 5, 7, 64 3it 6it

The stdout payload is byte-for-byte unchanged, and single-byte delimiters are unaffected since they cannot straddle a boundary.

Tests

tests_main.py::test_main_delim_multibyte_split covers the counting and --update paths at three buffer sizes and asserts the payload is preserved. Reverting cli.py makes it fail with 2it.

Full suite: 173 passed, 7 skipped (the skips are numpy, rich, dask and keras, none of which are installed in this environment).

Fixes tqdm#1847.

posix_pipe searched only the fresh chunk from each read:

    i = tmp.find(delim)

When the delimiter is multi-byte and --buf_size is smaller than it, the
delimiter's bytes land in two consecutive reads, so neither chunk contains it.
The saved prefix is then appended without ever being searched, and the
delimiter is missed.

Searching buf + tmp finds it, and because buf is non-empty in exactly the case
that matters, the indices have to refer to the concatenation:

    combined = buf + tmp
    i = combined.find(delim)
    if i < 0:
        buf = combined
        break
    fp_write(combined[:i + len_delim])
    callback(1 if callback_len else combined[:i])
    buf = b''
    tmp = combined[i + len_delim:]

Note that the record passed to the callback must also come from combined, and
the leftover chunk is combined[i + len_delim:]. Using tmp[i + len_delim:] would
silently drop the bytes still sitting in buf, which is what a partial fix that
only changes the find() does.

Verified against a 2-byte delimiter at --buf_size 1, 2, 3, 4, 5, 7 and 64:

  before: 2it instead of 3it, and --update died with
          SyntaxError: invalid syntax, because the first delimiter byte leaked
          into the record and left literal_eval parsing '1\xc3\xa92'
  after:  3it in counting mode, 6it with --update, at every buffer size

The stdout payload is byte-for-byte unchanged, and single-byte delimiters are
unaffected since they cannot straddle a boundary.

New test tests_main_delim_multibyte_split covers the counting and --update
paths at three buffer sizes and asserts the payload is preserved. Reverting
cli.py makes it fail with 2it.

The full suite passes: 174 passed, 5 skipped (numpy, dask and keras are not
installed here).
@methakon
methakon requested a review from casperdcl as a code owner October 1, 2026 09:17
@APX403

APX403 commented Oct 12, 2026

Copy link
Copy Markdown

I ran into the same undercount while reading posix_pipe and reproduced it on 9cf5a12 with a two-byte delimiter and a two-byte read buffer: 1é2é3 is counted as two records instead of three, with the bytes written to stdout unchanged. With this change applied locally the probe counts three and tests/tests_main.py still passes.

One thing worth flagging for whoever reviews this: #1848 fixes the same issue (and the --update crash on the same input) and was opened a few days earlier, so the two will need to be reconciled.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multibyte delimiters crossing read buffers undercount records and break --update

2 participants