BeviaLLM - From-Scratch Mini LLM (NumPy, CPU)

This project is an educational, from-scratch implementation of a tiny GPT-like language model using pure Python + NumPy (no PyTorch, no JAX, no autograd libraries).

The goal is to deeply understand:

• Token embeddings & positional embeddings
• Causal self-attention
• Transformer blocks (LayerNorm, MLP, residuals)
• Softmax + cross-entropy
• Backpropagation implemented manually
• Optimizers (AdamW)
• Text sampling

This is not meant to be fast or production-grade. It is meant to be readable and hackable.

TL;DR - Get a tiny LLM training on your MacBook CPU in 2 minutes.

Create a data.txt file (must not be empty):

This is a small training corpus for my from-scratch LLM.
It learns characters and predicts the next one.
I want to understand transformers deeply.

(Optional) Enlarge the dataset so the model can actually learn patterns:

python - <<'PY'
t = open("data.txt","r",encoding="utf-8").read()
open("data.txt","w",encoding="utf-8").write((t + "\n") * 2000)
print("New size:", len(open("data.txt").read()))
PY

pip install numpy

(Recommended: use a virtual environment)

python main.py \
  --data data.txt \
  --ctx 64 \
  --dim 64 \
  --layers 1 \
  --mlp_hidden 128 \
  --batch 8 \
  --steps 2000 \
  --log_every 50 \
  --sample_every 500 \
  --sample_len 200

You should see:

• Training loss decreasing
• Periodic sampled text printed to the console

This setup uses the existing .venv and ensures debug runs include the required --data arg.

• VS Code -> File -> Open...
• Select: /Users/vincentbevia/PycharmProjects/BeviaLLM

• Python (Microsoft)
• Pylance (Microsoft)

• Cmd+Shift+P -> Python: Select Interpreter
• Choose: .../BeviaLLM/.venv/bin/python

If it is not listed, click Enter interpreter path... and paste:

/Users/vincentbevia/PycharmProjects/BeviaLLM/.venv/bin/python

Create a folder at the project root: .vscode/

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "BeviaLLM: Train (debug)",
      "type": "python",
      "request": "launch",
      "program": "${workspaceFolder}/main.py",
      "console": "integratedTerminal",
      "justMyCode": true,
      "args": [
        "--data", "data.txt",
        "--ctx", "64",
        "--dim", "64",
        "--layers", "1",
        "--mlp_hidden", "128",
        "--batch", "8",
        "--steps", "2000",
        "--log_every", "50",
        "--sample_every", "500",
        "--sample_len", "200"
      ]
    }
  ]
}

{
  "python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python",
  "python.analysis.typeCheckingMode": "basic",
  "python.terminal.activateEnvironment": true,
  "editor.formatOnSave": true
}

• Run -> Start Debugging
• Select the "BeviaLLM: Train (debug)" configuration if prompted

In the VS Code terminal (View -> Terminal):

ls -l data.txt
wc -c data.txt

If it is empty, add a little text or reuse the Quick Start snippet.

python main.py --data data.txt --ctx 64 --dim 64 --layers 1 --mlp_hidden 128 --batch 8 --steps 500 --log_every 50

Create .vscode/tasks.json:

{
  "version": "2.0.0",
  "tasks": [
    {
      "label": "BeviaLLM: train",
      "type": "shell",
      "command": "${workspaceFolder}/.venv/bin/python",
      "args": [
        "${workspaceFolder}/main.py",
        "--data", "data.txt",
        "--ctx", "64",
        "--dim", "64",
        "--layers", "1",
        "--mlp_hidden", "128",
        "--batch", "8",
        "--steps", "2000",
        "--log_every", "50",
        "--sample_every", "500",
        "--sample_len", "200"
      ],
      "problemMatcher": [],
      "group": { "kind": "build", "isDefault": true }
    }
  ]
}

Then run via Cmd+Shift+P -> Tasks: Run Task -> BeviaLLM: train.

• VS Code runs system Python, not .venv. Fix: Step 3 + python.defaultInterpreterPath in settings.json.
• Working directory is wrong, so it cannot find data.txt. Fix: Keep data.txt in repo root and use the launch.json above.

• Character-level language model (no tokenizer required)
• Causal self-attention (GPT-style)
• Manual forward + backward pass for all layers
• AdamW optimizer implemented from scratch
• Text sampling with temperature
• CPU-only (runs on MacBook / laptop)

The code is modularized into a clean package for readability:

BeviaLLM/
├── main.py              # CLI + training loop
├── data.txt             # Training text
├── bevialm/
│   ├── __init__.py      # Package exports
│   ├── utils.py         # set_seed, softmax, cross_entropy_loss, init_weight
│   ├── optimizer.py     # AdamW optimizer
│   ├── layers.py        # Embedding, Linear, LayerNorm, CausalSelfAttention, TransformerBlock
│   ├── model.py         # CharTransformerLM
│   └── data.py          # load_text, build_vocab, encode, get_batch, sample
└── .vscode/             # VS Code debug/task configs

• Python 3.10+
• NumPy

This project is designed so you can open the bevialm/ modules and trace gradients all the way through:

This is CPU-only and written in NumPy:

• It is slow by design
• Small models train in minutes
• Larger configs will be very slow

Use small configs while learning.

Once you are comfortable:

• Implement multi-head attention (currently single-head in bevialm/layers.py)
• Replace ReLU with GELU
• Add dropout
• Add learning rate warmup + cosine decay
• Replace char-level model with BPE tokenizer
• Add checkpoint saving/loading
• Visualize attention maps
• Add unit tests for each module

This project is for learning and experimentation only. It is not optimized, not secure, and not intended for production use.

• Vaswani et al., Attention Is All You Need
• Andrej Karpathy - Let us build GPT from scratch
• CS231n - Backpropagation fundamentals

Understanding LLMs at this level makes you:

• Better at debugging real models
• More critical of black-box AI
• Stronger at system design for AI-powered software

Have fun breaking it and rebuilding it.