Trent + Chippi = TRIPPI Programming Language (Project for CS451)

Text to tokes

A LaTeX parser, in Go

Built a full-featured interpreter with zero external dependencies, using Vaughan Pratt’s top-down operator precedence parsing technique

This is a simple LL(1) grammar tokenizer & parser written in Go.

This is a library to scan, parse and tokenize text.

A simple tokenizer library for parsing text or any text files

Token counting utility

The implementation of MaxMatch algorithm in golang

go-bert-tokenizer

Tokenize a prompt and/or get the number of tokens a prompt will take

Go bindings for Github's bpe-openai tokenizer implementation.

A BPE tokenizer built with golang

The Cidoka Programming Language

A lexer generator for golang

create click tracks from harmonixset annotations on the fly with libmetro, libsoundio, and stk

Simple text base math solver for basic operations

An experimental database implementation written in pure Go.

Parse JSON with imperfect syntax and capture byte offsets of tokens.

tokenize string as command line args