This project is my implementation from scratch of a Generative Pretrained Transformer (GPT).

Book Attention is all you need Dropout

This project is named Sturnus after the genus of starlings. Starlings are capable of imitating many sounds such as the songs of other birds, human vistling or even mobile ring tunes. Generative pretrained transformers are similarily able to imitate human language even if they do not hold any conscious understanding of the language. Still, the capabilities of GPTs can seem almost magical - a bit like the feathers of metalic purple, green and blue that can be seen on starlings in the sun.