Token IDs
↓
Embedding
↓
┌─────────────────────────────┐
│ Transformer Block × N │
│ │
│ RMSNorm │
│ ↓ │
│ Multi-Head Attention + RoPE │
│ ↓ │
│ Residual │
│ ↓ │
│ RMSNorm │
│ ↓ │
│ SwiGLU │
│ ↓ │
│ Residual │
└─────────────────────────────┘
↓
RMSNorm
↓
Linear
↓
Logits
% make testdata examplecurl -fs -o testdata/merge_rules.gob https://raw.githubusercontent.com/itsubaki/gpt/refs/heads/embed-dim-256/testdata/merge_rules.gob
curl -fs -o testdata/tiny_codes.bin https://raw.githubusercontent.com/itsubaki/gpt/refs/heads/embed-dim-256/testdata/tiny_codes.bin
curl -fs -o testdata/model_gpt.gob https://raw.githubusercontent.com/itsubaki/gpt/refs/heads/embed-dim-256/testdata/model_gpt.gob
curl -fs -o testdata/model_gpt_sft.gob https://raw.githubusercontent.com/itsubaki/gpt/refs/heads/embed-dim-256/testdata/model_gpt_sft.gob
curl -fs -o testdata/model_gpt_grpo.gob https://raw.githubusercontent.com/itsubaki/gpt/refs/heads/embed-dim-256/testdata/model_gpt_grpo.gob### Instruction:
Write a is_prime function
### Response:
def is_prime(n):
if n < 2:
return False
for i in range(2, int(n**0.5) + 1):
if n % i == 0:
return False
return True% make dl
curl -fs -o testdata/tiny_codes.txt https://raw.githubusercontent.com/oreilly-japan/deep-learning-from-scratch-6/refs/heads/main/codebot/tiny_codes.txt
curl -fs -o testdata/tiny_codes_sft.json https://raw.githubusercontent.com/oreilly-japan/deep-learning-from-scratch-6/refs/heads/main/codebot/tiny_codes_sft.json% make tokenize
go run ./cmd/tokenize -vocab-size 1000
Training BPE 100%|██████████████████████████████| 743/743
saved merge rules to testdata/merge_rules.gob
...
995 -> "are"
996 -> ")."
997 -> " my"
998 -> "emain"
999 -> "<|endoftext|>"
byte count: 6487033
token count: 2640742
compression ratio: 2.456519038967078
encoding time: 1.459157917s
saved tokens to testdata/tiny_codes.bin% make pretrain
go run ./cmd/pretrain/main.go
Pre-Training 100%|██████████████████████████████| 20000/20000make generate
go run ./cmd/generate/main.go --prompt 'def add(a, b):'def add(a, b):
if b == 0:
return (a)
return a + b
print(a, b)
% make sft
go run ./cmd/sft/main.go
SFT 100%|██████████████████████████████| 500/500% make chat
go run ./cmd/chat/main.go --prompt 'Write a loop'### Instruction:
Write a loop
### Response:
for i in range(5):
print(i)
else:
print('done')
### Instruction:
Who are you?
### Response:
I'm CodeBot. How can I assist you today?
### Instruction:
3+9=
### Response:
12
% make grpo
go run ./cmd/grpo/main.go
GRPO 100%|██████████████████████████████| 100/100% make eval
go run ./cmd/eval/main.go --batch-size 100
6+8=14 true
5+5=10 true
8+8=15 false
...
7+2=9 true
accuracy: 99 %