Skip to content

Edition 2026.10 · October 14, 2026 · Ollaya 0.13.0

Every open decision model, measured on real hardware.

Accuracy, calibration, speed and memory of 67 models on the GPUs and CPUs people own. Every model runs through the same released runtime, checked against its authors’ code, and every number links to its raw data.

Models
67
Tasks
14
13 public datasets and typed-decisions
Questions per model
5,880
human and gold labels
Devices
2
RTX 5090, Threadripper 3960X

Leaderboard on the RTX 5090

NVIDIA GPU, CUDA, 32 GB. Pick the hardware closest to yours.

Highest score
pplx-decider:flash
score 73.7 · 127 ms · 10.4 GB
Best under 100 ms
jeb:4b
score 63.9 · 90.7 ms · 5.4 GB
Best in 8 GB of VRAM
jeb:4b
score 63.9 · 90.7 ms · 5.4 GB
Fastest
coba:68m-ternary
score 1.6 · 13.6 ms · 899 MB
Score against latencyPareto frontier (9): nothing is both faster and higher scoringother models
Score against median latency on the RTX 5090: 65 models, 9 on the Pareto frontier010203040506070801020501002005001k2kMedian latency, five questions (ms, log scale)Scorejeb:27b: score 69.1, 264 msd3:mini: score 68.0, 202 msd3:flash: score 67.8, 317 mswinnow:12b: score 65.7, 138 msclef:flash: score 65.1, 268 msstrands-decider:e4b: score 64.6, 213 mscygnet:12b: score 64.6, 152 msdecider:4b: score 64.5, 197 msgliclass-v4:9b: score 64.3, 263 msdecision2:lux: score 63.2, 364 mskev:9b: score 62.6, 248 msnimble:9b-v3: score 62.4, 942 msd3:nano: score 62.2, 88.9 msnimble:9b: score 61.9, 1.01 sstrands-decider:2b: score 61.3, 97.6 mskev:4b: score 61.2, 171 mscredence:e4b: score 61.0, 121 msdecision2:nox: score 60.9, 251 mscredence:e4b-calibrated: score 60.5, 114 mswinnow:e4b: score 59.9, 113 msgliclass-v4:2b: score 59.1, 82.2 msstrands-decider:e2b: score 58.0, 116 msdecider:2b: score 57.5, 84.1 mssnap:2b: score 55.8, 82.3 mssnap:2b-q4_k_m: score 54.4, 78.2 msjeeves:9b: score 54.0, 388 msdecision2:sol: score 50.0, 89.4 msd3:lite: score 49.9, 65.2 mswinnow:e2b: score 49.4, 104 msjevk5:4b: score 48.6, 95.3 msdecider:0.8b: score 47.8, 60.1 msdecision2:eos: score 47.0, 68.5 msd1-omni:600m: score 45.5, 16.6 msd3:edge: score 42.9, 27.7 mssura:2b: score 40.9, 62.6 mskev:0.8b: score 40.8, 62.3 mslaya:typed-decisions: score 40.3, 20.4 msdecision:eos: score 38.1, 68.7 msdecision2:kai: score 38.0, 30.8 mssura:0.8b: score 37.2, 69.3 mscoba:1b: score 36.3, 20.1 mscoba:400m: score 35.8, 20.5 msdecima:base: score 31.0, 26.7 mslaya:en: score 30.1, 18.3 msdecima:agent: score 27.9, 25.3 msdecima:guard: score 25.8, 26.3 msvon:1.1: score 23.7, 19.3 msdecima:base-clean: score 23.5, 26.4 msnli:deberta-v3-large: score 21.0, 17.7 msnli:modernbert-large: score 17.1, 16.8 msarbiter:4b: score 15.2, 135 msdecima:guard-small: score 14.8, 26.6 msclm:8b: score 9.3, 94.5 mslargha:150m: score 9.1, 17.8 mscoba:150m-ternary: score 4.6, 15.4 mscoba:400m-ternary: score 4.4, 19.6 mspplx-decider:flash: score 73.7, 127 msjeb:9b: score 66.0, 111 msjeb:4b: score 63.9, 90.7 msd1:3b: score 63.1, 50.8 msgliclass:v4-edge: score 55.8, 16.3 mslaya:multilingual: score 36.7, 14.3 msdecima:small: score 17.2, 14.2 msgliclass:large: score 15.3, 14.1 mscoba:68m-ternary: score 1.6, 13.6 msd1:3bjeb:4bjeb:9bpplx-decider:flash
1pplx-decider:flashPareto
Perplexity AI
73.782.472.60.016127 ms10.4 GB4BApache-2.0
2jeb:27btrained on split
frontier-infra
69.179.280.2 *0.033264 ms16.2 GB27BApache-2.0
2d3:mini
vLLM Semantic Router
68.078.873.70.030202 ms10.8 GB4BApache-2.0
2d3:flash
vLLM Semantic Router
67.879.071.20.030317 ms18.0 GB9BApache-2.0
2jeb:9bParetotrained on split
frontier-infra
66.077.279.5 *0.020111 ms9.1 GB9BApache-2.0
2winnow:12b
EldanRing
65.777.370.20.141138 ms15.5 GB12BApache-2.0
2clef:flash
Cloudflare
65.177.270.30.025268 ms19.6 GB9BApache-2.0
3strands-decider:e4b
Strands Agents
64.676.569.90.039213 ms15.4 GB8.9BApache-2.0
3cygnet:12b
blockbrain-ai
64.676.868.40.103152 ms18.1 GB12BApache-2.0
3decider:4b
Mapika
64.576.967.80.094197 ms11.0 GB4.2BApache-2.0
3gliclass-v4:9b
Knowledgator
64.376.365.00.034263 ms21.0 GB9.5BApache-2.0
5jeb:4bParetotrained on split
frontier-infra
63.975.980.0 *0.04890.7 ms5.4 GB4BApache-2.0
5decision2:lux
vLLM Semantic Router
63.275.670.70.056364 ms19.9 GB7.94BApache-2.0
5d1:3bPareto
LiquidAI
63.175.165.00.06350.8 ms4.0 GB3BLFM Open 1.0
5kev:9b
Jared Palmer
62.675.372.20.058248 ms17.9 GB7.9BApache-2.0
5nimble:9b-v3
Bespoke Labs
62.475.464.80.092942 ms20.7 GB9BCC-BY-NC-4.0 + Apache-2.0
5d3:nano
vLLM Semantic Router
62.275.662.50.06288.9 ms8.2 GB2BApache-2.0
5nimble:9b
Bespoke Labs
61.974.866.50.0221.01 s20.1 GB9BApache-2.0
8strands-decider:2b
Strands Agents
61.374.561.60.03997.6 ms8.5 GB2BApache-2.0
8kev:4b
Jared Palmer
61.274.566.90.062171 ms10.9 GB4.2BApache-2.0
8credence:e4b
Txoka
61.074.172.50.054121 ms6.2 GB7.5BApache-2.0
8decision2:nox
vLLM Semantic Router
60.974.668.00.050251 ms12.1 GB4.21BApache-2.0
12credence:e4b-calibrated
Txoka
60.573.772.40.038114 ms6.2 GB7.5BApache-2.0
12winnow:e4b
EldanRing
59.973.472.40.071113 ms6.2 GB7.5BApache-2.0
15gliclass-v4:2b
Knowledgator
59.173.158.90.03182.2 ms8.9 GB2.4BApache-2.0
18strands-decider:e2b
Strands Agents
58.071.664.60.065116 ms10.1 GB4.6BApache-2.0
19decider:2b
Mapika
57.572.460.80.08984.1 ms8.3 GB1.9BApache-2.0
25gliclass:v4-edgePareto
Knowledgator
55.871.450.60.07016.3 ms2.6 GB440MApache-2.0
25snap:2b
logitlab
55.870.264.80.04982.3 ms3.4 GB2BApache-2.0
26snap:2b-q4_k_m
logitlab
54.469.165.50.04378.2 ms2.5 GB2BApache-2.0
26jeeves:9btrained on split
PostHog
54.075.679.0 *0.018388 ms18.0 GB9BApache-2.0
31decision2:sol
vLLM Semantic Router
50.067.855.50.05389.4 ms10.8 GB1.88BApache-2.0
31d3:lite
vLLM Semantic Router
49.967.052.60.07765.2 ms4.4 GB0.8BApache-2.0
31winnow:e2b
EldanRing
49.465.961.60.191104 ms3.5 GB4.6BApache-2.0
31jevk5:4brejects 12 %
alibiserikbay
48.662.162.90.04095.3 ms5.8 GB4BApache-2.0
32decider:0.8b
Mapika
47.866.950.50.06160.1 ms5.0 GB0.75BApache-2.0
32decision2:eos
vLLM Semantic Router
47.066.146.00.09368.5 ms5.8 GB0.75BApache-2.0
34d1-omni:600m
Liquid AI
45.563.957.50.07516.6 ms2.6 GB587MLFM Open 1.0
37d3:edge
vLLM Semantic Router
42.963.242.30.09227.7 ms2.6 GB0.6BApache-2.0
39sura:2b
Ollaya
40.961.057.00.06062.6 ms7.9 GB2BApache-2.0
39kev:0.8b
Jared Palmer
40.861.146.00.03562.3 ms4.5 GB0.76BApache-2.0
39laya:typed-decisionstrained on split
Convai Innovations
40.359.476.7 *0.02220.4 ms1.6 GB421MApache-2.0
40decision:eos
vLLM Semantic Router
38.159.151.20.08768.7 ms5.8 GB0.75BApache-2.0
40decision2:kai
vLLM Semantic Router
38.060.847.60.04630.8 ms3.6 GB0.60BApache-2.0
40sura:0.8b
Ollaya
37.258.656.00.05769.3 ms4.2 GB0.8BApache-2.0
40laya:multilingualPareto
Convai Innovations
36.757.934.90.15514.3 ms1.4 GB322MApache-2.0
41coba:1brejects 8 %
Ollaya
36.356.160.80.09020.1 ms3.6 GB1.1BApache-2.0
42coba:400mrejects 8 %
Ollaya
35.856.159.10.08920.5 ms1.6 GB421MApache-2.0
49decima:baserejects 6 %
A. M. Madani
31.055.048.40.05226.7 ms2.7 GB321MApache-2.0
49layarejects 8 %router
Convai Innovations
30.453.535.80.103––Apache-2.0
49laya:enrejects 8 %
Convai Innovations
30.153.235.80.10618.3 ms1.6 GB421MApache-2.0
49decima:agent
A. M. Madani
27.953.448.10.22825.3 ms2.7 GB321MApache-2.0
52decima:guard
A. M. Madani
25.851.840.40.12626.3 ms2.7 GB321MApache-2.0
52von:1.1
Victor Hugo Panisa
23.748.544.60.09819.3 ms2.6 GB395MApache-2.0
53decima:base-cleanrejects 6 %
A. M. Madani
23.549.247.40.06726.4 ms2.9 GB321MApache-2.0
54nli:deberta-v3-largerejects 6 %
Moritz Laurer
21.045.853.60.19617.7 ms2.6 GB435MMIT
56decima:smallParetorejects 9 %
A. M. Madani
17.246.241.40.06514.2 ms1.2 GB122MApache-2.0
56nli:modernbert-largerejects 5 %
Moritz Laurer
17.142.351.50.11116.8 ms2.6 GB396MApache-2.0
57gliclass:largePareto
Knowledgator
15.343.447.70.28414.1 ms2.6 GB439MApache-2.0
57arbiter:4brejects 57 %
Codekins · Zyot Lab
15.242.00.00.137135 ms8.3 GB4.3BApache-2.0 + Gemma
57decima:guard-small
A. M. Madani
14.842.440.00.14026.6 ms1.2 GB140MApache-2.0
62clm:8b
Contrastive-LM
9.337.735.70.37294.5 ms15.3 GB8.2BApache-2.0
62largha:150mtrained on splitrejects 9 %
Phocinae
9.139.887.9 *0.02617.8 ms1011 MB144MApache-2.0
62coba:150m-ternarytrained on splitrejects 8 %
Ollaya
4.662.253.50.05315.4 ms1.1 GB164MApache-2.0
62coba:400m-ternarytrained on splitrejects 8 %
Ollaya
4.465.455.90.04719.6 ms1.6 GB421MApache-2.0
64coba:68m-ternaryParetotrained on splitrejects 8 %
Ollaya
1.660.348.30.07613.6 ms899 MB75MApache-2.0
–qwen3guard:0.6bbuilt-in questions
Qwen
––––30.2 ms3.6 GB0.6BApache-2.0

Score: chance-corrected accuracy over 14 tasks (0 is guessing, 100 is every answer right), with its 95 % interval; models whose intervals overlap share a rank. Public: Bespoke Labs’ public benchmark, 3,880 human-labeled questions. Typed: typed-decisions, 2,000 questions. ECE: calibration error, lower is better. Latency: median of 20 warm five-question requests over HTTP. * trained on that dataset’s train split, so the score leaves it out. How we measure.