Support parallel encoding to handle super-long text. When input exceeds 10KB, the text is split into chunks and encoded concurrently using std::async, then the results are stitched back together.

Changes

src/sentencepiece_processor.cc

• EncodeParallel(input, pieces, num_threads) — new overload returning vector<string>
• EncodeParallel(input, ids, num_threads) — new overload returning vector<int>
• EncodeParallel(input, spt, num_threads) — new overload returning SentencePieceText (core implementation)
  • Falls back to serial Encode() for inputs <10KB or when num_threads=1
  • Normalizes the entire input first, then splits into chunks via SplitInputIntoChunks
  • Encodes each chunk with std::async and combines results
  • Calls PopulateSentencePieceText to build the final output with correct byte offsets
• SplitInputIntoChunks(input, num_chunks) — new helper that splits text at UTF-8 character boundaries (avoids splitting multi-byte sequences)
• Added #include <future> and #include <thread>

src/sentencepiece_processor.h

• Added EncodeParallel virtual methods (3 overloads) to the public API
• Added convenience wrappers:
  • EncodeAsPiecesParallel(input, num_threads)
  • EncodeAsIdsParallel(input, num_threads)
  • EncodeAsImmutableProtoParallel(input, num_threads)
• Added SplitInputIntoChunks as a private helper

src/sentencepiece_processor_test.cc

Added 5 test cases:

Implementation Details

• Chunk boundaries are adjusted backward to avoid splitting UTF-8 continuation bytes (string_util::IsTrailByte)
• Uses std::async(std::launch::async, ...) for thread-safe parallel execution
• Normalization is done once upfront, then chunks work on the normalized text
• Final PopulateSentencePieceText maps byte offsets back to original (unnormalized) input
  Fixes Support parallel encoding #1229