A tiny javascript bundler implementation.

JavaScript Reverse Engineering Toolkit (JSRETK) - Experimental tools for analyzing (minified/obfuscated) JavaScript

Tiny JavaScript tokenizer.

Python expressions parser and evaluator written in Javascript (https://github.com/odoo/odoo/tree/14.0/addons/web/static/lib/py.js)

Tokenizing strings of text. Regex extracting arrays of words and optionally numbers, emojis, tags, usernames and email addresses from strings. For Node.js and the browser. When you need more than just [a-z] regular expressions.

A secure, tokenized mobile wallet system optimized for vault transactions and revenue growth. Open Wallet is an open-source wallet infrastructure that transforms how businesses handle payments by leveraging tokenized vaults to maximize revenue while maintaining PCI DSS compliance.

Ultra-fast, client-side token counter for large text blobs

A JavaScript implementation of a Thai syllable segmenter using Conditional Random Fields (CRF). This tool segments Thai text into syllables, which is crucial for many NLP tasks in Thai language processing including text-to-speech, phonetic analysis, and language learning applications.

a lightweight no-dependency fork from transformers.js (only tokenizers)

Source Code Preparation Tool for LLMs

A simple browser-based tool to encode text into token IDs and decode them back, with a built-in token mapping viewer.

View the tokenisation of your words using the tokeniser for a Stable Diffusion model.

JS tokenizer for LLaMA 3 and LLaMA 3.1

Visualiza cómo los modelos de IA procesan y tokenizan tu texto

Tokenize a string.

🌿 NodeJS PHP Parser - extract AST or tokens

A react native port of Kagome: a Japanese Morphological Analyzer

Token Dispension System

My implementation of JackAnalyzer from Nand2Tetris

A simple tokenizer visualizer for processing text data into tokens to illustrate tokenization process in AI models for education purpose.