A multilingual morphological analysis library.

🎤 vibrato: Viterbi-based accelerated tokenizer

Ready-made tokenizer library for working with GPT and tiktoken

Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

Rust re-implementation of OpenFST - library for constructing, combining, optimizing, and searching weighted finite-state transducers (FSTs). A Python binding is also available.

A WHATWG-compliant HTML5 tokenizer and tag soup parser

Japanese Morphological Analyzer written in Rust

A tool that allow you to run SQL-like query on local files instead of database files using the GitQL SDK.

Chinese tokenizer for tantivy, based on jieba-rs

Lindera tokenizer for Tantivy.

llm_utils: Basic LLM tools, best practices, and minimal abstraction.

Viterbi-based accelerated tokenizer (Python wrapper)

Thai natural language processing library in Rust, with Python and Node bindings.

Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

C language lexer & parser & virtual interpreter from scratch in Rust

🛥 Vaporetto is a fast and lightweight pointwise prediction based tokenizer. This is a Python wrapper for Vaporetto.

Rust wrapper for the BlingFire tokenization library

Rust bindings to libpostal

A Japanese tokenizer for Tantivy, based on TinySegmenter.