Unsupervised Word Segmentation for Neural Machine Translation and Text Generation
-
Updated
Aug 7, 2024 - Python
#
bpe
Here are 82 public repositories matching this topic...
Fast bare-bones BPE for modern tokenizer training
-
Updated
Jun 23, 2025 - Python
Full-stack open-source AI engine for building language models — tokenizer training, transformer architecture, cognitive reasoning and chat pipeline.
nlp open-source machine-learning deep-learning tokenizer transformer language-model ai-engine conversational-ai rag bpe turkish-nlp ai- llm llm-training llm-engine- cognitive-ai-
-
Updated
Mar 28, 2026 - Python
Build LLM from scratch
-
Updated
Apr 12, 2026 - Python
Syllable-aware BPE tokenizer for the Amharic language (አማርኛ) – fast, accurate, trainable.
nlp machine-learning deep-learning tokenizer python3 text-processing amharic african-languages language-processing ethiopic low-resource-languages bpe llm bpe-tokenizer amharic-tokenizer amharictokenizer amhtokenizer
-
Updated
Nov 17, 2025 - Python
Subword Encoding in Lattice LSTM for Chinese Word Segmentation
-
Updated
Apr 25, 2019 - Python
Simple-to-use scoring function for arbitrarily tokenized texts.
-
Updated
Feb 19, 2025 - Python
A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust
python rust openai pypi-package bpe byte-pair-encoding huggingface tokenizers llm tiktoken bpe-tokenizer byte-pair-tokenizer
-
Updated
Mar 19, 2025 - Python
Geometric Byte Pair Encoding of Protein Structure (ICLR 2026)
-
Updated
Mar 29, 2026 - Python
Learning BPE embeddings by first learning a segmentation model and then training word2vec
-
Updated
Dec 18, 2022 - Python
Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization [arXiv 2025]
-
Updated
Dec 10, 2025 - Python
Byte-Pair Encoding (BPE) (subword-based tokenization) algorithm implementaions from scratch with python
python nlp natural-language-processing tokenizer data-preprocessing data-cleaning bpe byte-pair-encoding subword-tokenization
-
Updated
Jan 30, 2023 - Python
Code for the paper "BPE stays on SCRIPT"
-
Updated
Mar 4, 2026 - Python
Subword-augmented Embedding for Cloze Reading Comprehension (COLING 2018)
-
Updated
Nov 6, 2018 - Python
This repository provides a clear, educational implementation of Byte Pair Encoding (BPE) tokenization in plain Python. The focus is on algorithmic understanding, not raw performance.
-
Updated
Aug 28, 2024 - Python
Byte-Pair Encoding tokenizer for training large language models on huge datasets
python machine-learning deep-learning tokenizer chunking from-scratch bpe byte-pair-encoding large-language-models llm bpe-tokenizer
-
Updated
Jun 4, 2024 - Python
Improve this page
Add a description, image, and links to the bpe topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the bpe topic, visit your repo's landing page and select "manage topics."