🧠 miniGPT — A Decoder-Only Transformer Built From Scratch

Built to understand GPT-style Transformers by implementing one from scratch — not a production model or a state-of-the-art system. Trained on modest data with modest compute; its purpose is to demonstrate a correct, working understanding of the architecture, training loop, and generation process end to end.

miniGPT is a compact, decoder-only Transformer implemented from scratch in PyTorch — no transformers library, no pretrained weights. Every component (attention, embeddings, normalization, the training loop, and inference-time sampling) is hand-written, so the mechanics of a GPT-style model are fully transparent — the same core ideas that power GPT-2 and GPT-3, scaled down.

flowchart TD
    A["Token IDs (B, T)"] --> B["Token Embedding"]
    A --> C["Positional Embedding"]
    B --> D["Sum"]
    C --> D
    D --> E["Transformer Block × N"]

    subgraph Block["Pre-Norm Transformer Block"]
        direction TB
        E1["LayerNorm → Self-Attention → Residual Add"] --> E4["LayerNorm → FeedForward (GELU) → Residual Add"]
    end

    E --> F["Final LayerNorm"]
    F --> G["Linear Head → Logits (vocab_size)"]

Self-attention, briefly: input is projected into Q/K/V, split across n_head heads, scored as (Q·Kᵀ)/√head_size, masked so no token can attend to future positions (causal masking — what makes the model autoregressive), softmaxed, and used to weight V. Heads are concatenated and projected back to n_embd.

Self-supervised next-token prediction: the model predicts token t+1 from tokens 0..t at every position simultaneously.

flowchart LR
    A[Raw Text] --> B[BPE Tokenize] --> C["Input/Target Pairs (shift by 1)"]
    C --> D[Forward Pass] --> E[Cross-Entropy Loss] --> F[Backprop] --> G[AdamW Step]
    G --> H{Eval Interval?}
    H -- Yes --> I[Eval Loss + Sample + Checkpoint]
    H -- No --> C
    I --> C

• Gradient accumulation simulates a larger batch than fits in memory.
• Mixed precision (torch.amp) speeds up CUDA training.
• LR schedule: linear warmup, then cosine decay to min_lr.
• Every eval_interval steps: estimate train/val loss, generate a sample, save latest.pt and (if improved) best.pt.

Autoregressive: one token generated at a time, fed back in as input for the next step.

flowchart LR
    A[Prompt] --> B[Tokenize] --> C[Forward Pass] --> D["Logits ÷ Temperature"]
    D --> E["Top-k / Top-p Filter"] --> F[Softmax] --> G[Sample Token] --> H[Append] --> C

git clone https://github.com/MohamedAbdelaiem/miniGPT.git
cd miniGPT
pip install -r requirements.txt

python train.py      # train (auto-resumes from checkpoints/latest.pt)
python generate.py   # generate text / compare decoding strategies

from config import GPTConfig
from model import GPTLanguageModel
from tokenizer import setup_tokenizer
from generate import generate_text
from utils import load_checkpoint
import torch

config = GPTConfig()
encode, decode, config.vocab_size = setup_tokenizer()
model = GPTLanguageModel(config).to(config.device)
optimizer = torch.optim.AdamW(model.parameters(), lr=config.learning_rate)
scaler = torch.amp.GradScaler('cuda', enabled=(config.device == 'cuda'))
load_checkpoint('checkpoints/best.pt', model, optimizer, scaler, config.device)

print(generate_text(model, encode, decode, config,
                     prompt="Once upon a time", max_new_tokens=200,
                     temperature=0.8, top_p=0.9))

miniGPT/
├── config.py         # Hyperparameters
├── tokenizer.py       # GPT-2 BPE + dataset loading
├── model.py            # Model definition
├── utils.py              # Training helpers
├── train.py                # Training loop
├── generate.py               # Inference & sampling
├── requirements.txt
├── checkpoints/               # latest.pt, best.pt
└── outputs/run_*/               # checkpoints, samples, losses.png

MIT License