GPT-mini

A GPT implementation in PyTorch with self-training and configurable architecture.

python train_designer.py --no-interact --force   # generate gpt_mini3.json
python train_noipc_ddp.py -d 0,1                 # train on 2 RTX 3090s (no-P2P)
python train_ipc_ddp.py -d 0,1                   # train on 2 V100 SXM2 (P2P/NVLink)

• gpt_mini3_draft.json — draft architecture config (model, vocab, training defaults, paths)
• gpt_mini3.json — working config generated by train_designer.py (model, training, tokenizer, paths)

python train_designer.py                      # interactive: select GPUs, choose config
python train_designer.py --no-interact        # auto-select best config
python train_designer.py --no-interact --force  # overwrite existing config
python train_designer.py --scan-vocab         # scan corpus for vocab size

GPU Memory Analysis: train_designer.py calculates per-GPU memory usage and proposes valid (seq_length, batch_size) combinations that fit your hardware. Memory formula:

GPU_mem = fixed_gb + var_gb

fixed_gb = params × 12 / world_size / 1024³  (weights + grad + Adam states)
var_gb   = bs_per_gpu × seq_length × K × n_layer × 1.45 / 1024³
K        = 13 × n_embd + 4 × n_heads × seq_length

Where:

• ×12: 2B (FP16 weights) + 4B (grad) + 4B (Adam m) + 2B (Adam v) per parameter
• K breakdown: 13×n_embd (QKV+MLP+residuals) + 4×n_heads×seq_length (attention matrix)
• ×1.45: backward pass overhead (activations + gradients + optimizer scratch)

The attention term 4 × n_heads × seq_length² is the dominant variable cost — reducing seq_length has the biggest impact on memory. The MLP feedforward 13 × n_embd is second.

Breaking point example (32L/12H/hd=128, n_embd=1536, 2xRTX 3090):

seq_length=512, bs_per_gpu=32:  44.6 GB  [OOM]
seq_length=512, bs_per_gpu=8:   15.1 GB  [OK]
seq_length=256, bs_per_gpu=16:  12.9 GB  [OK]
seq_length=64,  bs_per_gpu=128: 17.7 GB  [OK]

python calc_params.py                           # model param count only
python calc_params.py gpt_mini3.json            # custom config path

python train_designer.py --no-interact         # auto-select config
python train_designer.py --no-interact --force # overwrite existing config

Interactive mode prompts for GPU selection, shows memory analysis, and proposes valid (seq_length, batch_size) combinations that fit your hardware.

Three triggers, all writing to the same base tier (checkpoints/<hash>/):

Resume reads checkpoints/<hash>/train.log once, extracts global_batch, and continues from that exact batch.

Two trainers, choose based on your GPU topology:

How they differ: DDP's all_reduce on CUDA tensors uses CUDA IPC, which requires P2P (peer-to-peer) access between GPUs. Our RTX 3090s have PXB (PCIe bridge) topology — no P2P — so DDP crashes with segfault (exit code 0xC0000005). The no-IPC trainer syncs gradients via CPU where gloo works fine on localhost.

# P2P GPUs (NVLink):
python train_ipc_ddp.py -d 0,1               # train on 2 GPUs
python train_ipc_ddp.py -d 0,1,2,3           # train on 4 GPUs

# No-P2P GPUs (PCIe bridge):
python train_noipc_ddp.py -d 0,1             # train on 2 GPUs, CPU grad sync

How it works: Uses torch.multiprocessing.spawn with one process per GPU. Each rank runs the same training loop but processes different data via LazyDistributedSampler (avoiding torch.randperm MemoryError on 453M-token dataset).

Master address: 127.0.0.1:29500 — set BEFORE mp.spawn to avoid Windows hostname resolution issues.

Ctrl+C handling: Signal handlers on each rank call dist.destroy_process_group() before exit. Main process waits 2s for graceful shutdown, then force-kills children.

Memory: Each GPU loads full model weights but only processes batch_size / world_size samples. On no-P2P systems, gradients are copied to CPU for all_reduce and copied back — adds ~50-100ms overhead per step, negligible compared to forward/backward compute.

{
  "model": {
    "n_layer": 32,
    "n_head": 12,
    "head_dim": 128,
    "seq_length": 512,
    "vocab": { "max_vocab_size": 32768, "max_word_len": 20 }
  },
  "training-defaults": {
    "batch_size": 64,
    "lr": 0.0003,
    "checkpoint_interval": 10000,
    "checkpoint_every_min": 30
  },
  "paths": {
    "data_dir": "E:\\training\\data",
    "extra_data_dirs": ["E:\\training\\data2\\bulgarian-corpus-33b"],
    "checkpoint_dir": "E:\\training\\checkpoints",
    "cache_dir": "E:\\training\\cache"
  }
}

training-defaults provides default training parameters that train_designer.py merges with calculated epochs and checkpoint_every into the final working config.

Download and process TinyStories-compatible corpus from chitanka.info — Bulgarian literature with fairy tales, fables, children's stories (1–100 KB each).

Downloads all texts once. Blocked queries are skipped and saved to registry for later retry.

python dataset_dl.py --api chitanka-xml

Runs download cycles repeatedly. Blocked queries are retried in each cycle after a configurable delay. Continues until all queries succeed or max-cycles reached.

python dataset_dl.py --api chitanka-xml --max-cycles 100 --cycle-interval 1800

• --max-cycles — maximum number of retry cycles (default: 100)
• --cycle-interval — seconds between cycles (default: 1800 = 30 min, matches chitanka.info rate limit reset)
• First N cycles use shorter interval (interval/3), then switch to full interval

Retries only previously blocked queries from the registry.

python dataset_dl.py --retry --api chitanka-xml

View download progress, blocked queries, and registry state.

python dataset_dl.py --status
python dataset_dl.py --status --api chitanka-xml

python dataset_dl.py --list

1. Metadata Collection — Uses chitanka.info XML search API (/texts/search.xml) to find texts matching queries (приказка, басня, разказ, смехурко, сръчко, грам, гатанк, стихотворение, баща ми, детски)
2. Text Download — Uses Playwright (headless Chrome) to bypass Cloudflare protection. Each text is fetched individually with 15s wait for Cloudflare challenge completion.
3. Text Extraction — Parses HTML, extracts story content between "Към текста" and "Към началото", strips UI elements, filters by length (15–2048 chars per line)
4. Deduplication — MD5 hash of text body, keeps only unique content
5. Checkpoint — Saves progress every 10 texts. Safe to Ctrl+C and resume.

chitanka.info blocks IPs after ~10–20 requests. Playwright bypasses Cloudflare challenge but not IP ban.

Recommended: Default 30-min intervals. Each cycle downloads ~50–100 texts. Total ~200 texts after 2–3 cycles.

• Checkpoints saved to E:\training\data2\extracted\.download_checkpoint.txt
• Registry saved to E:\training\data2\dataset_dl_registry.json
• Registry tracks: downloaded IDs, failed IDs, blocked queries, cycle count, timestamps
• Interrupt with Ctrl+C — checkpoint and registry are saved before exit
• Resume by running the same command again

After download, process and build the training corpus:

python dataset_builder.py --input E:\training\data2 --combine E:\training\data2\chitanka_corpus.txt

• Language detection: filters for Bulgarian (cyrillic ratio > 30%)
• Deduplication: MD5 hash of text lines
• Output: single clean text file (1 line per text, 15–2048 chars)

Set corpora in config to train on multiple datasets:

"corpora": [
  {"path": "E:\\training\\data\\tinystories.txt", "weight": 1.0},
  {"path": "E:\\training\\data\\wiki_train.txt", "weight": 0.5},
  {"path": "E:\\training\\data2\\chitanka_corpus.txt", "weight": 0.3}
]

Or use extra_data_dirs for automatic discovery:

"paths": {
  "data_dir": "E:\\training\\data",
  "extra_data_dirs": ["E:\\training\\data2"]
}

{
  "model": { "n_layer": 16, "n_head": 6, "head_dim": 128, "seq_length": 64 },
  "training": { "epochs": 5, "batch_size": 256, "lr": 0.0003, "checkpoint_every": 1, "checkpoint_interval": 10000, "checkpoint_every_min": 30 },
  "tokenizer": { "max_vocab_size": 32768, "max_word_len": 20 },
  "paths": { "data_dir": "E:\\training\\data", "extra_data_dirs": ["E:\\training\\data2"], "checkpoint_dir": "E:\\training\\checkpoints" }
}

• Integrated Training — model trains directly on the dataset within the script
• Auto Vocab Sizing — corpus-sampled, rounded to power of 2
• Auto Epoch Sizing — Chinchilla scaling law calculator (N=50 default)
• Text Generation — next-token prediction with temperature sampling
• Transformer Architecture — multi-head attention, layer norm, feed-forward
• Weight Tying — output projection tied to input embeddings
• CUDA Support — automatically uses GPU if available
• Checkpointing — batch/time/epoch triggers with global batch resume
• Resumable Downloads — handles SSL interruptions, resumes from partial file

• Word-level tokenization. For production, consider BPE or WordPiece.
• Adjust n_layer, n_head, head_dim based on hardware and dataset size.
• Chinchilla: optimal tokens = ~200x params; 50x is practical minimum.