Evals for models that decide
Jev is the first System One model: it writes no text, it answers typed questions with probabilities, in one fast pass. Jevals checks how good those answers are, against human labels, next to the LLMs that try to decide the same way.
Boards
System One LLM Top 2 for that metric (score, cost, speed)
A yes/no question. The model returns P(yes), and your code acts on it with an if.
PubMedQA, 300 questions
- Jev69.0
- Gemini 3.8 Flash73.0
- GLM-5.360.6
Pick one option from a list. The model returns a probability for every option, like a match over its cases.
Banking77, 300 questions
- Jev67.8
- Gemini 3.8 Flash74.1
- DeepSeek V4.1 Flash63.9
Place the state on an ordered rubric. The model returns a probability for every level, ready to sort by.
HelpSteer2 helpfulness, 300 questions
- Jev9.2
- GLM-5.37.8
- Mercury 2.5−5.5
No model clearly beats guessing yet.
Open the score board →Decision Score: 100 = perfect, 0 = guessing the label base rates. Bars on a 0–100 scale.
GPT-6 Astra, Claude Fable 5.1, Grok 4.6, Qwen3.8 Max, Kimi K3, decider-2b, openjev, GLiClass, TF-IDF + logistic regression, logprob-based LLM rows.
Every row is run by us at list price. How we list models
Jevals is independent. It is not affiliated with TypeSafe AI or Jev and does not represent them.