An advanced tokenizer made with typescript

A parser generator.

A Byte Pair Encoder for ServiceNow

Interactive tool to understand how text is tokenized by language models. Visualize tokens, analyze efficiency, and optimize your LLM prompts.

A JSON parser made in typescript with tokenizer, ast, parser ...

University Credits tokenazitation: Public good on-chain for global learning certification (ETH Global Istanbul 2023 Submission: prized). For university credits issued tokens and for less wealthy schools claiming an NFT for students evaluated succesfully.

Lightweight JSON tokenizer with symmetric dictionary encoding for compression and optimization

A language named Piglet to describe the flow chart

Unfinished

SQL Analyzer is a tool designed for analyzing SQL queries. The package enables parsing SQL queries, building an Abstract Syntax Tree (AST), and analyzing dependencies and relations used in queries.

Custom Tokenizer (Next.js + React + TypeScript)

React component to convert a text string into React Nodes

Tool for inspecting typescript token list

High-performance CSS parser optimized for static analysis and formatting

ZephyrAssets is a blockchain-based platform specializing in the tokenization of real-world assets. Leveraging smart contracts and blockchain technology, we provide a secure and transparent way to digitize, manage, and trade physical assets such as real estate, art, and collectibles

🪵 The API for building streaming tokenizers and lexers.

linguistic-helper.web.app/ Linguistic Helper is a platform to help students, faculties and Researchers to perform their work with help of Computational Linguistic and the modern tech making it an easy and efficient way of learning Linguistics

Navigate within a string

Interactive tool for exploring Byte Pair Encoding tokenization step-by-step.

Convert XPath 3.1 expression into token objects