A character-level GPT built from scratch in C++ — autograd, multi-head attention, stacked transformer blocks, and AdamW, all hand-written with zero dependencies beyond the standard library.
This Project was built for learning purposes only, therefore it only uses CPU power.
Do not Expect it to perform anywhere close to modern language models.
Put a plain-text training corpus at data/data.txt. The model is
character-level, so any UTF-8 text works.
make
./train # trains, prints loss, saves weights into data/ every 500 steps
./gen # loads weights from data/, prints generated text
Weights are written into data/ as plain-text files (one per matrix). Training
from nothing only needs data/data.txt to exist — the rest is generated.
The architecture constants at the top of train.cpp and generate.cpp —
D, FF_HIDDEN, HEADS, BLOCKS, WINDOW — must be identical in both files.
The weights are shaped by them; if they disagree, gen loads garbage. Change
one, change both, retrain.
Bigger D / BLOCKS = better output, slower training.