A multilingual morphological analysis library.

🎤 vibrato: Viterbi-based accelerated tokenizer

Ready-made tokenizer library for working with GPT and tiktoken

Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

Rust re-implementation of OpenFST - library for constructing, combining, optimizing, and searching weighted finite-state transducers (FSTs). A Python binding is also available.

A WHATWG-compliant HTML5 tokenizer and tag soup parser

Japanese Morphological Analyzer written in Rust

A tool that allow you to run SQL-like query on local files instead of database files using the GitQL SDK.

CLI tool – estimates LLM tokens/costs and runs provider-aware load tests for OpenAI, Anthropic, OpenRouter, or custom endpoints.

Chinese tokenizer for tantivy, based on jieba-rs

Lindera tokenizer for Tantivy.

llm_utils: Basic LLM tools, best practices, and minimal abstraction.

Viterbi-based accelerated tokenizer (Python wrapper)

Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

Thai natural language processing library in Rust, with Python and Node bindings.

C language lexer & parser & virtual interpreter from scratch

🛥 Vaporetto is a fast and lightweight pointwise prediction based tokenizer. This is a Python wrapper for Vaporetto.

A real-time LLM stream interceptor for token-level interaction research

Rust wrapper for the BlingFire tokenization library