choice: pick one of N
Each model reads a message a customer sent their bank and works out what they want, out of 77 possible requests, with a probability for each. All models get the same 300 messages, 5 times each, and we check every answer against human labels. The messages come from Banking77.
Score vs price
Up and to the left is better. The line joins the best value at each price.
System One LLM Top 2 for that metric (score, cost, speed)
Automation at 95% accuracy
Share of questions each model can answer alone and stay 95% right. It answers only above its confidence cut-off; the rest go to a person.
Ranking
System One LLM Baseline Top 2 for that metric (score, cost, speed)
| Model | Wins | 95% range of the Decision Score | ||||||
|---|---|---|---|---|---|---|---|---|
| 1 | Jev | 4 | 67.8 | 79.7% | 9.8 | $0.043 | 693 ms | |
Accuracy79.7%95% range 75.5–83.8% Calibration gap9.8 ptslower is better · 41.5% of answers at confidence 1.00 Automation at 95%54%of decisions, acting at confidence ≥ 0.98 (right 95.4%)shared gate ≥ 0.96: acts on 59%, right 94.3% Speed467 msmedian · p95 693 ms Stability2.7%of answers change on an identical rerun · 10.3% change when options are reordered Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupai-sdk experimental_evaluate ai@7.0.106 · native probabilities via Vercel AI Gateway · reasoning — · temperature — · 300 items × 5 runs · 2026-09-18 · served as typesafe-ai/jevmodel pagebanking77 run log | ||||||||
| 2 | Gemini 3.8 Flash | 3 | 74.1 | 84.6% | 3.2 | $1.4 | 3.9 s | |
Accuracy84.6%95% range 80.8–88.2% Calibration gap3.2 ptslower is better · 0.0% of answers at confidence 1.00 Automation at 95%45%of decisions, acting at confidence ≥ 0.89 (right 96.1%)shared gate ≥ 0.96: acts on 12%, right 99.4% Speed1.6 smedian · p95 3.9 s Stability3.0%of answers change on an identical rerun · 8.7% change when options are reordered Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Google AI Studio · reasoning low (mandatory) · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as google/gemini-3.8-flashmodel pagebanking77 run log | ||||||||
| 3 | DeepSeek V4.1 Flash | 2 | 63.9 | 75.9% | 2.8 | $0.14 | 1.3 s | |
Accuracy75.9%95% range 71.9–80.1% Calibration gap2.8 ptslower is better · 0.6% of answers at confidence 1.00 Automation at 95%39%of decisions, acting at confidence ≥ 0.86 (right 95.8%)shared gate ≥ 0.96: acts on 15%, right 99.1% Speed999 msmedian · p95 1.3 s Stability10.7%of answers change on an identical rerun · 30.2% change when options are reordered Valid output99.9%0.0% refused · 1,498 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · DeepSeek · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as deepseek/deepseek-v4.1-flashmodel pagebanking77 run log | ||||||||
| 4 | Mercury 2.5 | 1 | 54.0 | 70.3% | 10.4 | $0.036 | 1.5 s | |
Accuracy70.3%95% range 66.1–74.7% Calibration gap10.4 ptslower is better · 0.0% of answers at confidence 1.00 Automation at 95%—never reaches 95% accuracyshared gate ≥ 0.96: acts on 1%, right 100.0% Speed638 msmedian · p95 1.5 s Stability13.8%of answers change on an identical rerun · 36.1% change when options are reordered Valid output99.7%0.0% refused · 1,495 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Inception · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as inception/mercury-2.5model pagebanking77 run log | ||||||||
| =5 | GLM-5.3 | 0 | 66.8 | 78.9% | 6.2 | $0.72 | 3.2 s | |
Accuracy78.9%95% range 74.9–82.7% Calibration gap6.2 ptslower is better · 0.0% of answers at confidence 1.00 Automation at 95%38%of decisions, acting at confidence ≥ 0.83 (right 95.1%)shared gate ≥ 0.96: acts on 0%, right 100.0% Speed2.1 smedian · p95 3.2 s Stability9.7%of answers change on an identical rerun · 23.7% change when options are reordered Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Novita (fp8) · reasoning low (mandatory) · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as z-ai/glm-5.3model pagebanking77 run log | ||||||||
| =5 | Qwen3.8 Flash | 0 | 62.4 | 76.8% | 9.5 | $0.12 | 3.1 s | |
Accuracy76.8%95% range 72.6–80.9% Calibration gap9.5 ptslower is better · 0.0% of answers at confidence 1.00 Automation at 95%—never reaches 95% accuracyshared gate ≥ 0.96: acts on 2%, right 100.0% Speed1.7 smedian · p95 3.1 s Stability9.0%of answers change on an identical rerun · 22.1% change when options are reordered Valid output99.9%0.0% refused · 1,498 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Alibaba · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as qwen/qwen3.8-flashmodel pagebanking77 run log | ||||||||
| =5 | Mistral Medium 3.5 | 0 | 59.5 | 74.5% | 8.1 | $1.5 | 1.4 s | |
Accuracy74.5%95% range 70.1–78.9% Calibration gap8.1 ptslower is better · 0.0% of answers at confidence 1.00 Automation at 95%11%of decisions, acting at confidence ≥ 0.81 (right 99.4%)shared gate ≥ 0.96: acts on 0% Speed936 msmedian · p95 1.4 s Stability2.3%of answers change on an identical rerun · 22.2% change when options are reordered Valid output99.5%0.0% refused · 1,493 of 1,500 answered Setupjevals-llm 0383a0e3e592 · verbalized probabilities via OpenRouter · Mistral · reasoning none · temperature provider default · 300 items × 5 runs · 2026-09-18 · served as mistralai/mistral-medium-3-5model pagebanking77 run log | ||||||||
| — | Label prior | — | 0.0 | 1.3% | 0.0 | — | — | |
Accuracy1.3%95% range 0.3–2.7% Calibration gap0.0 ptslower is better · 0.0% of answers at confidence 1.00 Automation at 95%—no probabilitiesshared gate ≥ 0.96: acts on 0% Speed—median · p95 — Stability0.0%of answers change on an identical rerun · 0.0% change when options are reordered Valid output100.0%0.0% refused · 1,500 of 1,500 answered Setupconstant probabilities · reasoning — · temperature — · 300 items × 5 runsmodel page | ||||||||
Wins: a model wins a column when it is one of the two best in it: the highest score, the highest accuracy, the smallest calibration gap, the lowest price or the fastest speed. Ties count for everyone tied. The table is ordered by wins, then by score; models with as many wins share a place, so =3 means tied for third.
Release 2026-09-18 · suite 0.1.0 · board.json · permalink · explore every question · how we score