📄 | Recursive descent parser | Abstract Syntax Trees | Tokenizer

Lightweight tokenizer and syntax highlighter

A platform-agnostic Mash creation algorithm

Node wrapper around jumanpp, Japanese text tokenizer

Tokenizes Chinese texts into words.

fast, native syntax highlighting [wip]

Visualiza cómo los modelos de IA procesan y tokenizan tu texto

Ultra-fast, client-side token counter for large text blobs

javascript wordcloud generator supporting persian language using server + client

Token Dispension System

A simple tokenizer visualizer for processing text data into tokens to illustrate tokenization process in AI models for education purpose.

A JavaScript implementation of a Thai syllable segmenter using Conditional Random Fields (CRF). This tool segments Thai text into syllables, which is crucial for many NLP tasks in Thai language processing including text-to-speech, phonetic analysis, and language learning applications.

Yet another powerful tokenizer in js.

Rich Expressions

Chinese tokenizer base on nodejieba and pullword

A simple XML parser in NodeJS

A jQuery tool to tokenizer value on input

Set of three widgets built using React.

The client connects to a frontend server using a middleware load balancer that manages Docker instances with a "least connected" algorithm. New instances register dynamically, and a monitoring endpoint shows logs and request counts. The service counts tokens in text.