Edition 2026.10 · October 14, 2026 · Ollaya 0.13.0
Every open decision model, measured on real hardware.
Accuracy, calibration, speed and memory of 67 models on the GPUs and CPUs people own. Every model runs through the same released runtime, checked against its authors’ code, and every number links to its raw data.
- Models
- 67
- Tasks
- 14
- 13 public datasets and typed-decisions
- Questions per model
- 5,880
- human and gold labels
- Devices
- 2
- RTX 5090, Threadripper 3960X
Leaderboard on the RTX 5090
NVIDIA GPU, CUDA, 32 GB. Pick the hardware closest to yours.
| 1 | pplx-decider:flashPareto Perplexity AI | 73.7 | 82.4 | 72.6 | 0.016 | 127 ms | 10.4 GB | 4B | Apache-2.0 |
| 2 | jeb:27btrained on split frontier-infra | 69.1 | 79.2 | 80.2 * | 0.033 | 264 ms | 16.2 GB | 27B | Apache-2.0 |
| 2 | d3:mini vLLM Semantic Router | 68.0 | 78.8 | 73.7 | 0.030 | 202 ms | 10.8 GB | 4B | Apache-2.0 |
| 2 | d3:flash vLLM Semantic Router | 67.8 | 79.0 | 71.2 | 0.030 | 317 ms | 18.0 GB | 9B | Apache-2.0 |
| 2 | jeb:9bParetotrained on split frontier-infra | 66.0 | 77.2 | 79.5 * | 0.020 | 111 ms | 9.1 GB | 9B | Apache-2.0 |
| 2 | winnow:12b EldanRing | 65.7 | 77.3 | 70.2 | 0.141 | 138 ms | 15.5 GB | 12B | Apache-2.0 |
| 2 | clef:flash Cloudflare | 65.1 | 77.2 | 70.3 | 0.025 | 268 ms | 19.6 GB | 9B | Apache-2.0 |
| 3 | strands-decider:e4b Strands Agents | 64.6 | 76.5 | 69.9 | 0.039 | 213 ms | 15.4 GB | 8.9B | Apache-2.0 |
| 3 | cygnet:12b blockbrain-ai | 64.6 | 76.8 | 68.4 | 0.103 | 152 ms | 18.1 GB | 12B | Apache-2.0 |
| 3 | decider:4b Mapika | 64.5 | 76.9 | 67.8 | 0.094 | 197 ms | 11.0 GB | 4.2B | Apache-2.0 |
| 3 | gliclass-v4:9b Knowledgator | 64.3 | 76.3 | 65.0 | 0.034 | 263 ms | 21.0 GB | 9.5B | Apache-2.0 |
| 5 | jeb:4bParetotrained on split frontier-infra | 63.9 | 75.9 | 80.0 * | 0.048 | 90.7 ms | 5.4 GB | 4B | Apache-2.0 |
| 5 | decision2:lux vLLM Semantic Router | 63.2 | 75.6 | 70.7 | 0.056 | 364 ms | 19.9 GB | 7.94B | Apache-2.0 |
| 5 | d1:3bPareto LiquidAI | 63.1 | 75.1 | 65.0 | 0.063 | 50.8 ms | 4.0 GB | 3B | LFM Open 1.0 |
| 5 | kev:9b Jared Palmer | 62.6 | 75.3 | 72.2 | 0.058 | 248 ms | 17.9 GB | 7.9B | Apache-2.0 |
| 5 | nimble:9b-v3 Bespoke Labs | 62.4 | 75.4 | 64.8 | 0.092 | 942 ms | 20.7 GB | 9B | CC-BY-NC-4.0 + Apache-2.0 |
| 5 | d3:nano vLLM Semantic Router | 62.2 | 75.6 | 62.5 | 0.062 | 88.9 ms | 8.2 GB | 2B | Apache-2.0 |
| 5 | nimble:9b Bespoke Labs | 61.9 | 74.8 | 66.5 | 0.022 | 1.01 s | 20.1 GB | 9B | Apache-2.0 |
| 8 | strands-decider:2b Strands Agents | 61.3 | 74.5 | 61.6 | 0.039 | 97.6 ms | 8.5 GB | 2B | Apache-2.0 |
| 8 | kev:4b Jared Palmer | 61.2 | 74.5 | 66.9 | 0.062 | 171 ms | 10.9 GB | 4.2B | Apache-2.0 |
| 8 | credence:e4b Txoka | 61.0 | 74.1 | 72.5 | 0.054 | 121 ms | 6.2 GB | 7.5B | Apache-2.0 |
| 8 | decision2:nox vLLM Semantic Router | 60.9 | 74.6 | 68.0 | 0.050 | 251 ms | 12.1 GB | 4.21B | Apache-2.0 |
| 12 | credence:e4b-calibrated Txoka | 60.5 | 73.7 | 72.4 | 0.038 | 114 ms | 6.2 GB | 7.5B | Apache-2.0 |
| 12 | winnow:e4b EldanRing | 59.9 | 73.4 | 72.4 | 0.071 | 113 ms | 6.2 GB | 7.5B | Apache-2.0 |
| 15 | gliclass-v4:2b Knowledgator | 59.1 | 73.1 | 58.9 | 0.031 | 82.2 ms | 8.9 GB | 2.4B | Apache-2.0 |
| 18 | strands-decider:e2b Strands Agents | 58.0 | 71.6 | 64.6 | 0.065 | 116 ms | 10.1 GB | 4.6B | Apache-2.0 |
| 19 | decider:2b Mapika | 57.5 | 72.4 | 60.8 | 0.089 | 84.1 ms | 8.3 GB | 1.9B | Apache-2.0 |
| 25 | gliclass:v4-edgePareto Knowledgator | 55.8 | 71.4 | 50.6 | 0.070 | 16.3 ms | 2.6 GB | 440M | Apache-2.0 |
| 25 | snap:2b logitlab | 55.8 | 70.2 | 64.8 | 0.049 | 82.3 ms | 3.4 GB | 2B | Apache-2.0 |
| 26 | snap:2b-q4_k_m logitlab | 54.4 | 69.1 | 65.5 | 0.043 | 78.2 ms | 2.5 GB | 2B | Apache-2.0 |
| 26 | jeeves:9btrained on split PostHog | 54.0 | 75.6 | 79.0 * | 0.018 | 388 ms | 18.0 GB | 9B | Apache-2.0 |
| 31 | decision2:sol vLLM Semantic Router | 50.0 | 67.8 | 55.5 | 0.053 | 89.4 ms | 10.8 GB | 1.88B | Apache-2.0 |
| 31 | d3:lite vLLM Semantic Router | 49.9 | 67.0 | 52.6 | 0.077 | 65.2 ms | 4.4 GB | 0.8B | Apache-2.0 |
| 31 | winnow:e2b EldanRing | 49.4 | 65.9 | 61.6 | 0.191 | 104 ms | 3.5 GB | 4.6B | Apache-2.0 |
| 31 | jevk5:4brejects 12 % alibiserikbay | 48.6 | 62.1 | 62.9 | 0.040 | 95.3 ms | 5.8 GB | 4B | Apache-2.0 |
| 32 | decider:0.8b Mapika | 47.8 | 66.9 | 50.5 | 0.061 | 60.1 ms | 5.0 GB | 0.75B | Apache-2.0 |
| 32 | decision2:eos vLLM Semantic Router | 47.0 | 66.1 | 46.0 | 0.093 | 68.5 ms | 5.8 GB | 0.75B | Apache-2.0 |
| 34 | d1-omni:600m Liquid AI | 45.5 | 63.9 | 57.5 | 0.075 | 16.6 ms | 2.6 GB | 587M | LFM Open 1.0 |
| 37 | d3:edge vLLM Semantic Router | 42.9 | 63.2 | 42.3 | 0.092 | 27.7 ms | 2.6 GB | 0.6B | Apache-2.0 |
| 39 | sura:2b Ollaya | 40.9 | 61.0 | 57.0 | 0.060 | 62.6 ms | 7.9 GB | 2B | Apache-2.0 |
| 39 | kev:0.8b Jared Palmer | 40.8 | 61.1 | 46.0 | 0.035 | 62.3 ms | 4.5 GB | 0.76B | Apache-2.0 |
| 39 | laya:typed-decisionstrained on split Convai Innovations | 40.3 | 59.4 | 76.7 * | 0.022 | 20.4 ms | 1.6 GB | 421M | Apache-2.0 |
| 40 | decision:eos vLLM Semantic Router | 38.1 | 59.1 | 51.2 | 0.087 | 68.7 ms | 5.8 GB | 0.75B | Apache-2.0 |
| 40 | decision2:kai vLLM Semantic Router | 38.0 | 60.8 | 47.6 | 0.046 | 30.8 ms | 3.6 GB | 0.60B | Apache-2.0 |
| 40 | sura:0.8b Ollaya | 37.2 | 58.6 | 56.0 | 0.057 | 69.3 ms | 4.2 GB | 0.8B | Apache-2.0 |
| 40 | laya:multilingualPareto Convai Innovations | 36.7 | 57.9 | 34.9 | 0.155 | 14.3 ms | 1.4 GB | 322M | Apache-2.0 |
| 41 | coba:1brejects 8 % Ollaya | 36.3 | 56.1 | 60.8 | 0.090 | 20.1 ms | 3.6 GB | 1.1B | Apache-2.0 |
| 42 | coba:400mrejects 8 % Ollaya | 35.8 | 56.1 | 59.1 | 0.089 | 20.5 ms | 1.6 GB | 421M | Apache-2.0 |
| 49 | decima:baserejects 6 % A. M. Madani | 31.0 | 55.0 | 48.4 | 0.052 | 26.7 ms | 2.7 GB | 321M | Apache-2.0 |
| 49 | layarejects 8 %router Convai Innovations | 30.4 | 53.5 | 35.8 | 0.103 | – | – | Apache-2.0 | |
| 49 | laya:enrejects 8 % Convai Innovations | 30.1 | 53.2 | 35.8 | 0.106 | 18.3 ms | 1.6 GB | 421M | Apache-2.0 |
| 49 | decima:agent A. M. Madani | 27.9 | 53.4 | 48.1 | 0.228 | 25.3 ms | 2.7 GB | 321M | Apache-2.0 |
| 52 | decima:guard A. M. Madani | 25.8 | 51.8 | 40.4 | 0.126 | 26.3 ms | 2.7 GB | 321M | Apache-2.0 |
| 52 | von:1.1 Victor Hugo Panisa | 23.7 | 48.5 | 44.6 | 0.098 | 19.3 ms | 2.6 GB | 395M | Apache-2.0 |
| 53 | decima:base-cleanrejects 6 % A. M. Madani | 23.5 | 49.2 | 47.4 | 0.067 | 26.4 ms | 2.9 GB | 321M | Apache-2.0 |
| 54 | nli:deberta-v3-largerejects 6 % Moritz Laurer | 21.0 | 45.8 | 53.6 | 0.196 | 17.7 ms | 2.6 GB | 435M | MIT |
| 56 | decima:smallParetorejects 9 % A. M. Madani | 17.2 | 46.2 | 41.4 | 0.065 | 14.2 ms | 1.2 GB | 122M | Apache-2.0 |
| 56 | nli:modernbert-largerejects 5 % Moritz Laurer | 17.1 | 42.3 | 51.5 | 0.111 | 16.8 ms | 2.6 GB | 396M | Apache-2.0 |
| 57 | gliclass:largePareto Knowledgator | 15.3 | 43.4 | 47.7 | 0.284 | 14.1 ms | 2.6 GB | 439M | Apache-2.0 |
| 57 | arbiter:4brejects 57 % Codekins · Zyot Lab | 15.2 | 42.0 | 0.0 | 0.137 | 135 ms | 8.3 GB | 4.3B | Apache-2.0 + Gemma |
| 57 | decima:guard-small A. M. Madani | 14.8 | 42.4 | 40.0 | 0.140 | 26.6 ms | 1.2 GB | 140M | Apache-2.0 |
| 62 | clm:8b Contrastive-LM | 9.3 | 37.7 | 35.7 | 0.372 | 94.5 ms | 15.3 GB | 8.2B | Apache-2.0 |
| 62 | largha:150mtrained on splitrejects 9 % Phocinae | 9.1 | 39.8 | 87.9 * | 0.026 | 17.8 ms | 1011 MB | 144M | Apache-2.0 |
| 62 | coba:150m-ternarytrained on splitrejects 8 % Ollaya | 4.6 | 62.2 | 53.5 | 0.053 | 15.4 ms | 1.1 GB | 164M | Apache-2.0 |
| 62 | coba:400m-ternarytrained on splitrejects 8 % Ollaya | 4.4 | 65.4 | 55.9 | 0.047 | 19.6 ms | 1.6 GB | 421M | Apache-2.0 |
| 64 | coba:68m-ternaryParetotrained on splitrejects 8 % Ollaya | 1.6 | 60.3 | 48.3 | 0.076 | 13.6 ms | 899 MB | 75M | Apache-2.0 |
| – | qwen3guard:0.6bbuilt-in questions Qwen | – | – | – | – | 30.2 ms | 3.6 GB | 0.6B | Apache-2.0 |
Score: chance-corrected accuracy over 14 tasks (0 is guessing, 100 is every answer right), with its 95 % interval; models whose intervals overlap share a rank. Public: Bespoke Labs’ public benchmark, 3,880 human-labeled questions. Typed: typed-decisions, 2,000 questions. ECE: calibration error, lower is better. Latency: median of 20 warm five-question requests over HTTP. * trained on that dataset’s train split, so the score leaves it out. How we measure.