Summary
BPETokenizer::init_byte_mappings (cactus-engine/src/bpe.cpp) maps bytes 161..255 to identity code points. GPT-2 byte-level BPE (bytes_to_unicode()) excludes byte 173 (U+00AD, soft hyphen) from that range and assigns it the next free code point after the other non-printables (U+0143). Any UTF-8 sequence containing byte 0xAD therefore tokenizes differently from the reference tokenizer and decodes to U+FFFD.
Repro (LiquidAI/LFM2.5-230M bundle, cactus_tokenize / Cactus.tokenize in the RN bridge)
| input |
cactus 2.1.0 |
HF reference |
歯 (E6 AD AF) |
[672, 0, 617] (byte, unk, byte) |
[33687] |
歯医者 |
[672, 0, 617, 8773, 4046] |
[33687, 8773, 4046] |
明日の朝9時に歯医者に電話することを思い出させて |
17 tokens, 1 unknown |
15 tokens |
Only characters containing byte 0xAD are affected — everything else in the sentence tokenizes identically to HF. Decode side of the same table: cactus run on the stock lfm2.5-230m-cq4 bundle echoes 歯医者 as �丈医者. Observed end-to-end in a React Native app: a tool-call argument 歯医者に電話する came back as �Reception医者に電話する.
Root cause
for (int i = 161; i <= 255; ++i) bytes.push_back(i); // includes 173
...
} else if (byte >= 161 && byte <= 255) { // identity branch also catches 173GPT-2: identity ranges are 33..126, 161..172, 174..255; all other bytes (0..32, 127..160, 173) get 256+n in increasing byte order → 173 → U+0143.
Fix
Exclude 173 from both the identity list and the identity branch condition (the branch is selected by byte value, so moving the byte between lists alone is a no-op), and append 173 to remaining_bytes after 160. PR: #805. Verified: the resulting 256-entry mapping equals HF GPT2Tokenizer.bytes_to_unicode() byte-for-byte; cactus_tokenize("歯") → [33687] after the change.
Related: #803 (Python-binding ensure_ascii escaping — a separate, input-path bug; both were found while running a multilingual tool-calling model on cactus).
Environment: cactus v2.1.0 / main 09cb35a, macOS ARM64 + iOS simulator.
Summary
BPETokenizer::init_byte_mappings(cactus-engine/src/bpe.cpp) maps bytes 161..255 to identity code points. GPT-2 byte-level BPE (bytes_to_unicode()) excludes byte 173 (U+00AD, soft hyphen) from that range and assigns it the next free code point after the other non-printables (U+0143). Any UTF-8 sequence containing byte 0xAD therefore tokenizes differently from the reference tokenizer and decodes to U+FFFD.Repro (LiquidAI/LFM2.5-230M bundle,
cactus_tokenize/Cactus.tokenizein the RN bridge)歯(E6 AD AF)[672, 0, 617](byte, unk, byte)[33687]歯医者[672, 0, 617, 8773, 4046][33687, 8773, 4046]明日の朝9時に歯医者に電話することを思い出させてOnly characters containing byte 0xAD are affected — everything else in the sentence tokenizes identically to HF. Decode side of the same table:
cactus runon the stocklfm2.5-230m-cq4bundle echoes歯医者as�丈医者. Observed end-to-end in a React Native app: a tool-call argument歯医者に電話するcame back as�Reception医者に電話する.Root cause
GPT-2: identity ranges are 33..126, 161..172, 174..255; all other bytes (0..32, 127..160, 173) get 256+n in increasing byte order → 173 → U+0143.
Fix
Exclude 173 from both the identity list and the identity branch condition (the branch is selected by byte value, so moving the byte between lists alone is a no-op), and append 173 to
remaining_bytesafter 160. PR: #805. Verified: the resulting 256-entry mapping equals HFGPT2Tokenizer.bytes_to_unicode()byte-for-byte;cactus_tokenize("歯")→[33687]after the change.Related: #803 (Python-binding
ensure_asciiescaping — a separate, input-path bug; both were found while running a multilingual tool-calling model on cactus).Environment: cactus v2.1.0 / main 09cb35a, macOS ARM64 + iOS simulator.