A custom high-performance C++ inference engine for transformer-based language models. This engine is designed to run a 30 million parameter model trained on the TinyStories dataset.

This project is specifically developed for macOS. To build and run:

# Step 1: Build the project
make

# Step 2: Run inference
./a.out

The Makefile uses -march=native which enables CPU-specific SIMD instructions. On macOS Apple Silicon, this enables NEON. On Intel Macs, this enables SSE/AVX. For Linux or Windows, you may need to modify the Makefile to use appropriate SIMD flags for your CPU architecture.

This project implements a complete inference pipeline for decoder-only transformer models. The engine features optimized matrix multiplication with SIMD instructions, a GPT-2 compatible Byte Pair Encoding tokenizer, and a modular architecture that separates tensor operations, model components, and tokenization.

Inference/
├── engine/
│   ├── tensor.hpp          # Tensor class with matmul, softmax, LayerNorm
│   ├── tokenizer.hpp       # GPT-2 BPE tokenizer implementation
│   ├── runner.hpp          # Model, Transformer, Runner classes
│   ├── parser.hpp          # Weight loading from .npy files
│   ├── npy.hpp             # NumPy file parser (external library)
│   └── json.hpp            # JSON parser (external library)
├── weights/
│   ├── embeddings.weight.npy
│   ├── out.weight.npy
│   ├── out.bias.npy
│   └── transforms.*.npy    # Transformer block weights
├── gpt2_vocab.json         # Tokenizer vocabulary
├── merges.txt              # BPE merge rules
├── tokenizer.py            # Python script to export tokenizer
├── export_tokenizer.py     # Alternative tokenizer export script
├── convert.py              # Model conversion utilities
├── run.cpp                 # Main entry point
├── Makefile                # Build configuration
└── README.md               # This file

The engine supports a decoder-only transformer with the following configuration:

• C++17 compatible compiler (g++ or clang++)
• Python 3.x with transformers library (for tokenizer export)

make

The Makefile uses the following optimization flags:

Before running inference, export the GPT-2 tokenizer vocabulary:

python tokenizer.py

This creates gpt2_vocab.json and merges.txt in the project root.

./a.out

The Runner class initializes the model with weights from the weights/ directory and performs autoregressive generation.

Edit the run() function in runner.hpp to modify:

Model config: vocab=50257, d_model=256, heads=8, blocks=6
Loaded vocab: 50257 tokens
Loaded merges: 50000 BPE merge rules
Model initialized!
Tokenized: 6 tokens
Generating...
after embeddings: 6x256
after block 0: 6x256
after block 1: 6x256
after block 2: 6x256
after block 3: 6x256
after block 4: 6x256
after block 5: 6x256
logits: 6x50257

=== Generated Text ===
Hello, how are you? Once upon a time...
======================

The Tensor class implements the following operations:

Weights are stored in NumPy .npy format and loaded at runtime. The naming convention follows:

transforms.{block}.{component}.{head}.{weight|bias}.npy

The engine includes comprehensive error handling:

• Shape mismatch detection for all tensor operations
• Try-catch blocks around attention and forward passes
• Validation of weight dimensions during initialization
• Graceful handling of missing tokenizer files
• Detailed error messages with tensor shapes

This project is for educational and research purposes.

• TinyStories dataset for model training
• HuggingFace transformers for tokenizer reference
• GPT-2 architecture as the model foundation

Note: AI assistance was used during development to locate bugs, fix bugs, add error handling, and write boilerplate code. The npy.hpp and json.hpp files are not written by the author; they are external libraries obtained from GitHub. The tokenizer.hpp implementation was written by Claude Sonnet 4.6.