Lexical Analyser for Watt

Converts BBC BASIC 2 source code into UEF files.

Compilatore yield, set e for con LFM testing.

small project

Modern UTF-8 aware C++ tokenizer with vocabulary support, ideal for NLP and transformer models. Header-only and zero-dependency.

A silly weekend project that parses XML

HRTDS, pronounced "Hearts", is an acronym for "Human-Readable Typed Data Serialization".

Implementation of C++ lexical analyzer to demonstrate how it actually works as a part of the compiler.

Custom tokenizer loosely based on Byte-Pair Encoding

A C++ Parser Project

Fast wordpiece, sentencepiece tokenizer by Trie, OpenMP, SIMD, MemoryPool

Fast & efficient BPE tokenizer written in C & python for LLM tranining

Compact C++17 obfuscation pipeline demo (tokenizer → renamer → mutator → string protector) with CLI showcase and tests.

- RTFM -

Chinese sentence token

No C anymore only CSL ( C Style Lanugage)

Modern C++ brainfuck interpreter/compiler with LLVM 18

A very fast and low memory usage C++ automaton tokenizer that breaks an input string into a list of tokens looking at tabs, spaces, new lines, and detects special tokens like numbers, prces, personal noms, emails, lexemes, etc. It allows to specify delimeters and detect special cases.

Gradient Boosting Dicision Tree(LightGBM)を用い、教師ありで自然言語の分かちと形態素の推定を学習＆予想します。名称は珊瑚(sango)にしたい

These are files and projects from my CS 008 - Data Structures class