A simple brainf**k interpreter made in rust.

A simple and flexible lexer library written in pure rust.

🀄 The Jieba Chinese Analyzer for INFINI Pizza.

Next-Generation Byte Transformer made in Rust

Parser, tokenizer and AST generator for HF

Minimal implementation of BPE (Byte Pair Encoding) algorithm in Rust

Split chat messages by maximum chat completion token count

Lexer and parser generator without any dependencies.

Text tokenization service (Rust, Axum) (Q1:2025)

Rust wrapper for the Alpino tokenizer

Implementation of BPE Tokenizer in Rust

[WIP] A rust based interpreter for a language name "lox"

A compiled mecab-ipadic-neologd dictionary for vibrato

Blazingly fast word embeddings with Tokenlearn

Lox tree-walk interpreter in Rust

Fast BPE algorithm to generate byte pair encodings from text corpus, it's written in rust and approximately 20x faster than it's python implementation

A tiny rust markdown to html compiler for my blog

Huggingface tokenizers for Haskell

A Rust based gRPC server using Tonic which exposes a Tokenizer that splits a string into words and returns the tokens.

tokenizer, lexer, parser, or whatever in rust