Trent + Chippi = TRIPPI Programming Language (Project for CS451)

The implementation of MaxMatch algorithm in golang

Tokenize a prompt and/or get the number of tokens a prompt will take

A simple tokenizer library for parsing text or any text files

Token counting utility

Simple text base math solver for basic operations

An experimental database implementation written in pure Go.

Parse JSON with imperfect syntax and capture byte offsets of tokens.

Simple rule-based tokenizer writted in go

parse and evaluate formula.

Utilities for referencing byte and line+column offsets in UTF-8 streams.

Simple & fast Encoder/Decoder for tiktoken vocabulary.

A simple text tokenizer

StringUtils, Lexer and Tokenizer for Go! :)

High-performance tokenizer implementations in Go with unified CLI. Features Llama 3 tokenizer with exact compatibility, streaming support, and comprehensive tooling.

Octok is an OCONF line format tokenizer written in Go. It is a part of goconf "config" package.

Zero-allocation JSON tokenizer

Generate source code for a tokenizer from a given configuration, designed with simplicity, performance and readability in mind

Package of useful golang libraries

Longbridge Plain Text format.