Unsupervised Word Segmentation for Neural Machine Translation and Text Generation

Byte Pair Encoding for Python!

nfelib - bindings Python para e ler e gerir XML de NF-e, NFS-e nacional, CT-e, MDF-e, BP-e

Fast bare-bones BPE for modern tokenizer training

Syllable-aware BPE tokenizer for the Amharic language (አማርኛ) – fast, accurate, trainable.

Build LLM from scratch

Subword Encoding in Lattice LSTM for Chinese Word Segmentation

Simple-to-use scoring function for arbitrarily tokenized texts.

BBPE 底层实现

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

Learning BPE embeddings by first learning a segmentation model and then training word2vec

Byte-Pair Encoding (BPE) (subword-based tokenization) algorithm implementaions from scratch with python

Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization [arXiv 2025]

Code for the paper "BPE stays on SCRIPT"

Subword-augmented Embedding for Cloze Reading Comprehension (COLING 2018)

This repository provides a clear, educational implementation of Byte Pair Encoding (BPE) tokenization in plain Python. The focus is on algorithmic understanding, not raw performance.

Free to use tokenizers trained on the Darija language.

Byte-Pair Encoding tokenizer for training large language models on huge datasets

Docker container to expose the OpenAI tokenizer as a REST service