Compilatore yield, set e for con LFM testing.

A silly weekend project that parses XML

CS 280 Assignment 4 Language Interpreter

These are files and projects from my CS 008 - Data Structures class

A C++ Parser Project

Fast & efficient BPE tokenizer written in C & python for LLM tranining

Chinese sentence token

- RTFM -

Simple C++ way of dealing with JSON

C++ function to tokenize the string.

Tokenizer utilities for tsn-lang

CS 280 Assignment 2 Lexical Analyzer

A basic interpreter which interprets and executes a program line by line

Json Tokenizer.

A C++ console application that allows evaluating expressions on a JSON file.

BPE & Unigram Vocab Training library

CS280 homework 2 at NJIT that reads a file or standard input and splits it into tokens

Interpreter for the Datalog programming language. Parse, tokenize, AST, and execute database queries using relational algebra operators

From-scratch BPE tokenizer implementation in C++ with UTF-8 support including emojis.