Node based tokenizers for open source LLMs hosted on HuggingFace.

An advanced tokenizer made with typescript

this repository contient a minimal parser

AI Model Tokenization with Realtime Token Counting Built with Next.js

A parser generator.

Markup tokenizer transform stream

Interactive tool to understand how text is tokenized by language models. Visualize tokens, analyze efficiency, and optimize your LLM prompts.

🪁 Parser grammar builder. Write parsers with pattern matching.

A JSON parser made in typescript with tokenizer, ast, parser ...

Lexical analyzer for JavaScript

University Credits tokenazitation: Public good on-chain for global learning certification (ETH Global Istanbul 2023 Submission: prized). For university credits issued tokens and for less wealthy schools claiming an NFT for students evaluated succesfully.

Design System construído com React, Storybook e Stitches, organizado em um monorepo para facilitar a colaboração e o gerenciamento.

Interface interativa para simulação de pré-vendas de tokens e criação de lançamentos fictícios em tempo real.

An extendable ChordPro parser.

A wrapper library around https://github.com/takuyaa/kuromoji.js that intelligently groups Japanese morphemes into words

Lightweight JSON tokenizer with symmetric dictionary encoding for compression and optimization

A lightweight, fast, and extensible line-by-line content tokenizer and parser

🇯🇵🍜 learner‑friendly Japanese tokenizer / segmenter

Fast & Non-regex based lexical analysis for Deno.