See related: #1
Test 1a - NUMA/SDRAM Tweaks (SDRAM_BANKLOW=1) - FIRMWARE 6 JAN
Tested with Pi OS Bookworm updated Friday Dec 20, with Tue 12 Nov 16:10:44 UTC 2024 (1731427844) firmware
Device
CPU/GPU
Model
Speed
Power (Peak)
Pi 5 - 16GB
CPU
llama3.2:3b
4.59 Tokens/s
11.2 W
Pi 5 - 16GB
CPU
llama3.1:8b
1.87 Tokens/s
10.9 W
Pi 5 - 16GB
CPU
llama2:13b
1.20 Tokens/s
10.5 W
Test 1b - NUMA/SDRAM Tweaks (SDRAM_BANKLOW=1) - FIRMWARE 8 JAN
Tested with the latest firmware as of Jan 8 2025
Device
CPU/GPU
Model
Speed
Power (Peak)
Pi 5 - 16GB
CPU
llama3.2:3b
4.88 Tokens/s
11.9 W
Pi 5 - 16GB
CPU
llama3.1:8b
2.17 Tokens/s
11.6 W
Pi 5 - 16GB
CPU
llama2:13b
1.36 Tokens/s
10.9 W
Test 2 - No Tweaks (NUMA interleave:0)
Device
CPU/GPU
Model
Speed
Power (Peak)
Pi 5 - 16GB
CPU
llama3.2:3b
4.51 Tokens/s
11.0 W
Pi 5 - 16GB
CPU
llama3.1:8b
2.05 Tokens/s
10.8 W
Pi 5 - 16GB
CPU
llama2:13b
1.35 Tokens/s
10.8 W
Compared to the Pi 5 8GB
Device
CPU/GPU
Model
Speed
Power (Peak)
Pi 5 - 8GB
CPU
llama3.2:3b
4.61 Tokens/s
13.9 W
Pi 5 - 8GB
CPU
llama3.1:8b
1.99 Tokens/s
13.2 W
Pi 5 - 8GB
CPU
llama2:13b
DNF
DNF
👍 React with 👍 3berkerogluu, Akkiesoft and JackDaniells121