Skip to content
#

mha

Here are 35 public repositories matching this topic...

A GPT-style autoregressive language model built from scratch in PyTorch - no AutoModelForCausalLM, no GPT2LMHeadModel, no pretrained Transformer implementation. Implements token embeddings, positional embeddings, causal self-attention, grouped-query attention (GQA), Transformer decoder blocks, AdamW optimization, checkpointed GPU training.

  • Updated Aug 18, 2026
  • Jupyter Notebook

Add this topic to your repo

To associate your repository with the mha topic, visit your repo's landing page and select "manage topics."

Learn more