jev-rs

System One judgments from any LLM, in one prefill. A Rust engine that answers typed questions about a piece of state — noul (yes/no), choice (one of N), score (ordered scale) — with probabilities, never generated text. Wire-compatible with TypeSafe's Jev POST /v1/systemone, and exposed to coding agents as an MCP tool.

Among the open Jev reimplementations, jev-rs is the only one that is not itself a model or a single technique: it is the harness around the whole ecosystem — a logprob scorer that reads any decoder, adapters for the trained specialists, the TypeSafe wire server, and one eval/calibrate metric stack that scores all of them side by side (tables below).

Claude Code / Codex / Grok Build / OpenCode ──MCP stdio──▶ jev ──▶ llama-server (any GGUF)
TypeSafe SDKs (TYPESAFE_BASE_URL) ──────────POST /v1/systemone──▶ jev serve ──▶ llama-server

curl -fsSL https://raw.githubusercontent.com/yijunyu/jev-rs/main/install.sh | sh

Installs a prebuilt jev into ~/.local/bin (macOS arm64/x86_64, Linux x86_64/arm64) or builds from source with cargo if no binary matches. With a Rust toolchain, cargo install jev-rs works too. Then start any GGUF model behind llama-server (macOS: brew install llama.cpp):

llama-server -hf Qwen/Qwen3-4B-GGUF --port 8089 -np 2 -c 8192

Try it:

jev --backend http://127.0.0.1:8089 ask \
  --state "We were billed twice for March. Refund the duplicate today or we cancel." \
  --choice "dept=Which team handles this?|billing:refunds and invoices,technical:bugs,sales:pricing" \
  --score  "urgency=How urgent?|not urgent,soon,blocking" \
  --noul   "churn=Does the customer threaten to leave?"

{
  "dept":    {"type":"choice","choice":"billing","probabilities":{"billing":1.0,"technical":0.0,"sales":0.0},"confidence":1.0},
  "urgency": {"type":"score","score":1.98,"legend":{"0":"not urgent","1":"soon","2":"blocking"},"probabilities":{"0":0.0,"1":0.02,"2":0.98},"confidence":0.98},
  "churn":   {"type":"noul","noul":1.0}
}

Set JEV_BACKEND_URL once to drop the --backend flag. Use --template gemma|llama3|raw for non-ChatML models.

Hosted or OpenAI-compatible backends. --backend-kind openai scores through chat/completions with logprobs (max_tokens: 1, top_logprobs: 20), so the DeepSeek API, vLLM, SGLang or llama-server's own /v1 endpoint work without raw prompt access:

export JEV_API_KEY=$DEEPSEEK_API_KEY
jev --backend https://api.deepseek.com/v1 --backend-kind openai --model deepseek-flash \
    eval examples/dev_tasks.jsonl

The server applies its own chat template, so answers depend on the model emitting the option letter as its first token; the raw llamacpp path is exact and preferred when you control the server.

jev mcp is an MCP server over stdio with one tool, judge. The agent sends a state and typed questions and gets probabilities back instead of prompting a big model to classify and parsing its prose. Typical uses inside an agent session: routing a task to a skill, triaging tool output, gating a risky command, ranking candidates, yes/no checks on a diff.

Claude Code

claude mcp add jev -e JEV_BACKEND_URL=http://127.0.0.1:8089 -- jev mcp

or in .mcp.json at the project root:

{"mcpServers": {"jev": {"command": "jev", "args": ["mcp"], "env": {"JEV_BACKEND_URL": "http://127.0.0.1:8089"}}}}

Codex (~/.codex/config.toml)

[mcp_servers.jev]
command = "jev"
args = ["mcp"]
env = { JEV_BACKEND_URL = "http://127.0.0.1:8089" }

Grok Build (.grok/mcp.json in the project, or ~/.grok/mcp.json)

{"servers": {"jev": {"command": "jev mcp", "env": {"JEV_BACKEND_URL": "http://127.0.0.1:8089"}}}}

or grok mcp add jev --command "jev mcp" --env JEV_BACKEND_URL=http://127.0.0.1:8089.

OpenCode (opencode.json)

{"mcp": {"jev": {"type": "local", "command": ["jev", "mcp"], "environment": {"JEV_BACKEND_URL": "http://127.0.0.1:8089"}}}}

Then tell the agent, in its instructions file, when to reach for it:

Use the judge tool for any classification, routing, triage or yes/no decision about text or tool output. Put the facts in state and ask one judgment per question. Trust answers with confidence ≥ 0.8; otherwise decide yourself.

TypeSafe SDK users. jev serve speaks the same wire format as api.typesafe.ai, so the official Python/JS SDKs, the jev crate and any Jev integration run against a local model unchanged:

jev --backend http://127.0.0.1:8089 serve --bind 127.0.0.1:8090
export TYPESAFE_BASE_URL=http://127.0.0.1:8090 TYPESAFE_API_KEY=local

Case-file line: {"state": ..., "questions": {...}, "gold": {"id": "key-or-level"}}.

1. The state is rendered once into a shared prefix; each question is a suffix ending in Answer: with options labelled A, B, C… so the backend's prompt cache prefills the state once per request.
2. The backend returns raw next-token log-probabilities for the labels. Nothing is decoded; usage.output_tokens is always 0.
3. Restricted softmax over the labels, divided by a fitted temperature per (question type, option count) bucket. confidence is TypeSafe's documented formula; score is the probability-weighted level.
4. --permutations k averages k rotations of the option order to control position bias.

Apple M1 Ultra 128 GB, llama-server via Homebrew, zero-shot, same prompts. examples/dev_tasks.jsonl: 25 shell commands × 3 questions (command class 8-way, safe to re-run, output volume 3-level), hand-labelled, 75 decisions, 0 failed requests for either model.

What changed with the larger models: the 8-way classification was already saturated at 4B; the gains are on the yes/no and ordinal questions and on probability quality. Both large models are close to calibrated out of the box (fitted temperatures near 1), so their confidence can gate about twice as many decisions at a 5 % error budget. The 4B model is far faster and its fitted temperatures of 3–6 say its raw probabilities should not be trusted without jev calibrate.

DeepSeek V4 Flash gives the best yes/no judgment of the three (0.92 on "safe to re-run") and the worst ordinal one: of its 13 wrong score answers, 12 guessed low and 11 of those by exactly one level, a systematic bias a per-model calibration or a coarser scale would absorb. Its latency is an artefact of the run, not the model: 156 GB of tensors on a 128 GB machine means experts stream from SSD on every request. With the model resident (a 192 GB or 256 GB Mac) the same engine prefills at hundreds of tokens per second.

Prompt caching depends on the architecture: Qwen3-4B re-evaluates only the question suffix after the first question (33–62 tokens), while the hybrid Qwen3.8-Next re-evaluates the full prompt for every question in llama-server, so its four-question example costs 3.0 s against 0.4 s.

Two further experiments against real Claude Code session logs, predicting output floods before a command runs and triaging tool output after it, are in docs/EXPERIMENTS.md; one is a clear win for the judge, the other a clear loss to a blind rule.

Hosted Jev has been independently measured at 236–276 ms p50 per request (jev-benchmarks, decision-model-benchmark).

Same examples/dev_tasks.jsonl (75 decisions, 0 failed), same machine, through jev eval so the metrics line up with the table above. The two open alternatives are wired as first-class backends: Laya (--backend-kind laya, typed-decisions) and AgentJev (--backend-kind agentjev, 0.6B). Both answer a whole request in one call; probabilities are converted to log-probs and scored with the same harness as the logprob path.

Neither specialist beats the logprob path on accuracy here: both sit near chance on noul and score. Laya is about 2× faster and, after a per-bucket temperature fit, has the best ECE of the three — useful if you only care about calibrated confidence on its own traffic. AgentJev is ~25× slower on this CPU host and worst on raw Brier; its fitted ECE is fine, but accuracy stays at chance. Keep the default llamacpp/openai path for judgments; use --backend-kind laya or --backend-kind agentjev when you specifically want those models.

The same three backends, scored on their home bench — Typed Decisions test (400 cases / 2 000 decisions, revision ea930645…) — converted to jev-rs case JSONL and run through jev eval. Datasets differ from the table above; each bench favors the model trained on it.

† Fit is done on the same rows jev eval reports (optimistic); treat as a ceiling, not a held-out calibration number. Laya and AgentJev answers are whole-case wall time (5 decisions per case); the llamacpp column is per-question next-token scoring.

The two tables are one conclusion, not a contradiction. Laya and AgentJev are specialists: they only know the 20 fixed Typed Decisions schemas they were trained on. On their home bench they win (0.77 / 0.80 vs 0.54); on dev_tasks.jsonl — shell-command questions they have never seen — they fall to chance (0.49 each). Qwen3-4B through raw logprobs is a generalist: zero-shot on any schema. It wins dev_tasks (0.71) and sits near the Typed Decisions prior baseline (0.54) when cold against a model fitted to that exact data. This is the specialist-vs-generalist split Typed Decisions itself documents — each path wins where it was built to win. Pick by traffic:

• fixed, known schema with a trained checkpoint → --backend-kind laya or --backend-kind agentjev
• open-ended or changing questions → the default llamacpp / openai logprob path

Members' logits never transfer across models — the logprobs of model X only exist in X's own vocabulary and hidden-state geometry. What can be reused is the probability vector over the shared option labels, so jev ensemble takes each member's jev eval --rows file, aligns the questions, and fits a multinomial logistic stacker (the meta-predictor: given X's readout, guess what Y would have said) on a case-level train split — no case leaks across the split. The held-out test split is reported next to every member and the uniform average, through the same eval::metrics harness:

jev eval examples/dev_tasks.jsonl --rows dev.rows.jsonl                       # per backend
jev ensemble --rows dev.rows.jsonl --rows laya.rows.jsonl --rows jev.rows.jsonl \
             --out ensemble.json
jev ensemble --model ensemble.json --rows dev.rows.jsonl --rows …             # apply later

The stacker beats every individual member on accuracy and Brier — the generalist's probability vector carries signal the two specialists lack (0.548 alone, yet the ensemble is above the best specialist). On dev_tasks.jsonl — 12 test questions, llamacpp at 0.833 — no ensemble can beat a member that is already right four times out of five on that tiny split; the win needs the larger bench. The offline report sums each member's recorded latency — the rows come from three separate passes, so that figure is a sequential cost, not what a live fan-out pays — and fit runs on the train split only, no optimistic in-sample numbers.

The same model also serves live. --ensemble fans each request out to every --member backend in parallel, rebuilds the trained features (per-member temperature-calibrated probability vectors, kind one-hot, option count) and answers from the meta-model — so ask, serve, eval and the MCP path all return one ensemble answer instead of three. Member names and order must match the fitted model:

jev --ensemble ensemble.json \
    --member 'llamacpp.rows=llamacpp,http://127.0.0.1:8089' \
    --member 'laya.rows=laya,http://127.0.0.1:8010,typed-decisions' \
    --member 'agentjev.rows=agentjev,http://127.0.0.1:8149' \
    ask --state '…' --noul 'churn=…'

Live latency is then the slowest member, not the sum. Measured on the same day (dev_tasks.jsonl, 75 cases): members at 64 / 41 / 492 ms p50 — 597 ms if run in sequence — versus 495 ms for the ensemble itself, i.e. the fan-out waits on the slowest member while the other two ride along. The 0.747 accuracy that same pass prints is in-sample (these are the rows fit trained on) and is not a held-out number; use the table above.

Open Jev solutions cluster into three groups. Wire compatibility matters here: everything that speaks POST /v1/systemone (or a thin variant) can be swapped in and out, so the groups are not walled gardens.

1. Servers that speak the wire format. Hosted Jev is the reference; jeff (GLiFormer 400M), SemIf (formerly OpenJev; frozen Qwen3.5-4B option-logit readout, MIT), jev-bridge (Rust, OpenAI-API → letter-slot logprob readout), djev-spark (DiffusionGemma, multimodal), and jev-rs itself all serve the same request shape. So do System One Lite (MLX, Apple silicon) and coreai-kit (decide-cli serve, Apple Core AI); the vendor-neutral system-one SDK reaches any of them through HTTPConfig(base_url=…). SemIf is the strongest open row on the published JevBench composite (#2, 0.7 behind Jev); jeff and jeff-class rebuilds trail further.

2. Trained specialist models. Laya (ModernBERT, Apache-2.0), AgentJev (0.6B) and NanoJev (Qwen3-0.6B + decision heads, 2–255 candidates) return full distributions over their trained schemas. jev-rs wires the first two as first-class backends (--backend-kind laya, --backend-kind agentjev); NanoJev is not wired yet.

3. Client-side adapters and alternate techniques. system-one-adapter (TypeSafe's own LLM-backed adapter), poorjev (local NLI + temperature scaling), AnyJev (Nokia), Jevlike, LocalJev, Nimble — these reach the same interface from a different mechanism or sit in front of it rather than behind it.

What is unique about jev-rs. Every other open project is one model (Laya, AgentJev, NanoJev, jeff), one readout technique (SemIf, jev-bridge, poorjev), or hosted (Jev, djev). jev-rs is the only project that is all of:

• a generalizing scorer — the same prefill + label-logprob readout works against any GGUF through llama-server or any OpenAI-compatible /v1 host, zero-shot on schemas it was never trained on;
• a hub for the specialists — the full-request backends (Laya, System One, AgentJev) plug into the same FullBackend trait, so their probabilities are scored by the identical harness;
• the wire server — jev serve is a drop-in for api.typesafe.ai, so the official SDKs and every /v1/systemone integration run against any of the above;
• one measurement stack — jev eval / jev calibrate report accuracy, Brier, top-label ECE, coverage and latency for every backend, which is how the two comparison tables above were produced, and an MCP server so coding agents get the same judgments in-session; jev ensemble then stacks any of them into a single meta-predictor that beats its best member.

In short: the ecosystem offers interchangeable parts; jev-rs is the chassis that mounts them and the dyno that measures them.

It is also built to be the judgment engine inside two Rust systems — PRECC, a Claude Code hook that saves tokens, and ds4-rs-metal / Local Mind, an on-device DeepSeek-V4 engine — where an in-process, KV-forking scorer is the point.

• At most 26 options per question (single-letter labels); Jev accepts 255.
• Scorer backends: llama-server (raw logprobs) and any OpenAI-compatible /v1 host. Full-request backends: Laya/systemone, AgentJev. In-process llama.cpp and ds4-rs scorers are next.
• Zero-shot only; no RLCD-style training.

MIT or Apache-2.0, at your option.