data/authored144.jsonl is von's authored144 set (github.com/wfzyx/von, Apache-2.0; project-authored, no copied text): 144 three-way decisions over one- or two-sentence states, 48 each of evidence interpretation (supported / insufficient / contradicted), rule application (permitted / prohibited / insufficient) and candidate selection (A / B / neither), labels balanced. data/perturbations108.jsonl re-asks 108 of them with the options reversed or the question rewrapped. The set is adversarial by design: every case turns on a negation, a scope word or an abstention, and half the trick is knowing when the evidence does not settle the question.

jolt -M bench/authored144.clj                              # english
jolt -M bench/authored144.clj --model typed-decisions
jolt -M bench/authored144.clj --model minicpm5 --thinking false   # a configured thinker (config.edn :thinkers, or --thinker PATH as `thinker`)
jolt -M bench/authored144.clj --model minicpm5                    # thinking
jolt -M bench/authored144.clj --file bench/data/perturbations108.jsonl --out results.jsonl

Each row is one choice question: the row's question as the instructions, its options as the criteria, its state as the state.

10-core M-series laptop. lev through bench/authored144.clj; von-1.0 through its own Python backend (BertaBackend, torch 2.14 on the CPU, the weights from wfzyx/von-1.0); MiniCPM5-2B (openbmb/MiniCPM5-2B-GGUF, Q8_0) through llama-server, one chat completion per case with the answer constrained by a grammar to the option ids, thinking on or off through the chat template.

The lev thinker is the same model and the same llama.cpp inside lev-server (lev.think): the prompt is built by lev, the thought is decoded greedily until </think>, then \n\nANSWER: is forced and each option is scored by the log probability of its tokens (plus the end-of-turn token), a softmax over those being the answer's probabilities — the same calibrated shape the encoders give, where the llama-server harness parsed a free-form final line. Greedy beats a sampled thought by four points here and is reproducible; the two points to the free-form run are prompt and decoding noise on seven cases.

localjev is a Bun bridge that turns the Jev request into a prompt asking an MLX-served chat model for a JSON probability vector (no thinking, temperature 0, retries on malformed JSON). Its probabilities are self-reported: 143 of the 144 answers here were one-hot, and all 13 wrong answers came with a probability of 0.9 or more, so its confidence says nothing. It needs oMLX and bun on Apple Silicon and a 16.6 GB model; it was slower than the 2.5B thinker in the lev binary and four points behind it.

The trio localjev's bake-off uses, 40 balanced cases each (bench/triad120.py builds the set from the public datasets; jolt -M bench/triad.clj <model> [thinking] runs it). Routing-style traffic, the kind the encoder's presets exist for:

Localjev's own numbers on the same three tasks (different samples, M5 Max): Gemma 4 26B-A4B 75.0% macro at 0.68 s, DiffusionGemma 26B-A4B 74.2% at 1.2 s, with a 2,048-word distraction dropping everything.

The encoder is the fast path: it beats the 2.5B model answering at once on the routing task by 15 points, at 125 ms on a CPU against 150 ms on a GPU (~560 ms on a CPU), one forward for a whole workflow instead of one prompt per question, with calibrated confidence. The thinker's thinking earns its seconds on the reading-comprehension and reasoning cases.

• Option-permutation averaging (the note's #3; the encoder's answer averaged over every rotation of the option list): 20 of 144 cases change their argmax under rotation (14%, the same share nibzard measured on Jev). Averaging moves english 61.1% → 63.9% and its ECE 0.101 → 0.082; typed-decisions 66.7% → 66.0%. Three forwards per question, one batch. Small and inconsistent.
• Neutral-state prior subtraction (jevmlx's correction; score the options against an empty state, subtract the log-odds): english 61.1% → 54.9%, and combined with averaging 62.5%. An empty state is out of the encoder's distribution; the correction hurts. Not worth it.
• Qwen3-4B-Thinking-2507 as the thinker (#5): 94.4%, the same as MiniCPM5-2B's 95.1%, at 13× the time (41 s a case, 1,883 thought tokens against 369). The 2.5B model is the escalation tier.
• The gate on in-distribution traffic (#4): the encoder's noul confidence is max(p, 1 − p), never below 0.5, and its BoolQ answers are over-confident — at a 0.5 threshold every BoolQ case stays on the encoder (72.5% right) where the thinker gets 90%; at 0.7 it still keeps 37 of 40. Choice and score confidences (1 − normalised entropy) behave: at 0.5 the gate keeps 36/40 AG News (the encoder's best task) and 13/40 SST-5 (its worst). The gate needs a per-type threshold or a refit noul temperature before it can be trusted on yes/no questions.
• Calibration on authored144: encoder ECE 0.101 (typed-decisions 0.133), von 0.124, the thinker 0.039 — but 97% of the thinker's answers come with confidence above 0.5, so its confidence cannot gate anything; the encoder's can (88% of these adversarial cases sit below 0.5, 12% of in-distribution AG News).

jolt -M:calibrate --labels bench/data/calib.jsonl (1,240 public cases from bench/calib_data.py: SST-2, AG News, dair-ai/emotion, 20 newsgroups, BoolQ, SST-5 — one per temperature bucket), fitted on the even cases, reported on the odd ones, english:

The shipped temperatures are sharp where the encoder is weakest (a 20-way choice at 35% accuracy reported ~1.0 confidence). On the trio the refit takes ECE 0.161 → 0.106 and the gate at 0.5 now escalates every SST-5 case (13/40 were kept before) while AG News keeps 38/40 at 97%. BoolQ does not move: the noul confidence floor is 0.5 and within the task confidence barely tracks correctness (kept accuracy 72% at ≥0.5, 79% at ≥0.9), so yes/no questions need their own, higher threshold — threshold {"noul": 0.9} — or, on traffic like this, escalating them outright (the thinker gets 90%).

--debias through the runner: english 63.9%, ECE 0.082, 199 ms a case (the rotations share one batched forward).

What the numbers say:

• The gap to a hosted generative decision API on hard cases is the thinking, not the encoder. Answering directly, the 2.5B decoder is no better than von (73.6% vs 76.4%) at 3.5x the CPU time; with ~300 tokens of reasoning it gets 140/144, at 3 s a case on the GPU and ~10 s on the CPU.
• von-1.0 is the same ModernBERT-large encoder this port runs, with an NLI head instead of the encoders' marker head, one pair sequence per option instead of one sequence per question. +15 points over the english encoder on this set for ~1.4x the time. The encoder under-predicts the abstaining option (insufficient 18 times against 36 expected); von's failures are concentrated in rule application under perturbation (10/36).
• The two encoders fail on different cases: 41 von-right/english-wrong, 19 the other way, 15 both wrong, and the thinker gets all 15 of those. A confidence gate over the encoder's top-2 margin at 0.5 sends 92 of 144 cases up to the thinker and lands at 88.9%; over von's at 0.5, 92 up and 93.1%. On this set almost everything is low-confidence; on routine traffic the gate would pass most cases through at encoder speed.

The Jev benchmark from the other session (24/64 for the encoder vs 60/64 hosted Jev) is the same shape: a generative model with a reasoning budget against a single-pass encoder.

bench/workflow.clj times the README's four email questions as one call (choice, score, noul, noul) on a short state (53 tokens) and a long one (1,631 tokens, cut to max_len 512 for every question), 30–40 iterations after a warmup, english on the M-series CPU. Before, encoder-forward tokenized the state once per question; after (laya-mlx's PrefixCache ported: lev.sequence/encode-state once a call, cached-prefix in a 128-entry LRU per agent), once per call, and a question asked before costs no tokenization at all. Answers are unchanged (golden/ pins them: prefix-and-state-assemble-into-build-sequence proves the ids byte for byte).

The long-state call drops 10%: three of its four 48 ms state tokenizations were redundant. The short-state case saves ~5 ms of a 300 ms call, under the run-to-run drift of the same code (288 vs 304 ms), so no claim there. The tokenizer itself runs at ~33 µs a token, slow in absolute terms (laya-jolt-edf); this removes the repetition, not the rate.

native/llama.cpp is now thecodacus/llama.cpp's parallel-decision branch (upstream b10435 + 617 commits, plus tools/parallel-decision/decision-engine.cpp, linked into lev_llm by native/lev_decision.cpp). With thinking off, lev.think hands every question of a call to that engine (lev.llm/jev): the chat up to the state is decoded once and kept across calls, the state continues it, and each question is a branch sequence forked from the state (unified KV cache, llama_memory_seq_cp), all of them in one llama_decode. Each branch is the rest of that question's direct prompt through ANSWER: , and its option ids (closed by <|im_end|>) are scored at every node where their tokens diverge. So the prompts are byte for byte the ones the per-question path scores (think-test proves it), the questions cannot see each other there either, and what changes is the time and the scoring. MiniCPM5-2B Q8, Metal, same machine:

Against the per-question path, Jev mode with lev's cut moves four cases (+3 -1) and a probability by 0.011 at the median: the same answers, normalised at the divergence nodes instead of over the whole vocabulary. The per-question path read 74.3% on llama.cpp b10435 and 73.6% on the fork: one case, the newer kernels. The engine's own cut, which tokenizes ANSWER: supported whole so the id is scored as supported (the token the model would write), moves 38 cases the wrong way on balance (+15 -23); on this model the id after a decoded space is better, so lev passes the engine's split_boundary. kev's zero-shot SemIf prompt (which took an instruct Qwen from 0.726 to 0.812 on kev's transfer suite) loses ten points here: MiniCPM5 reads lev's prompt better.

Where it pays is a call with several questions (bench/workflow.clj --model <thinker>, the four email questions, 20 iterations, p50):

The per-question path decodes the whole prompt four times; Jev mode decodes the system turn once for the server's life, the state once a call, and the four questions' ~40-token tails together. One question on a long state is the state's prefill either way. Through lev-server, three questions on a short ticket answer in 170 ms warm.

Several states against the same questions (/v1/systemone/batch, lev.agent/system-one-batch) go to the engine as one call. With this layout it saves little: 8 short states take 1,671 ms against 1,681 ms one call at a time. Every question's ~50-token tail is decoded again for every state, so the work is compute-bound either way. The catalog layout (:layout "catalog": the questions go in the cached text before the state, and each field is a short ANSWER <n>: ) is what batching is for. It takes 565 ms for the 8 states and 104 ms for four questions on one, but authored144 falls to 48.6% (balanced 43.3%). A batched state's probabilities match the same state answered alone to within 0.007 (llm-test): llama.cpp is not batch-invariant.

Same benches, thinking off, Metal. Qwen2.5-1.5B-Instruct is what harshatheg/Qwen-2.5-1B-RLCD runs: that repo holds no weights, only the parallel-decoding code over the stock model. Qwen3.5-4B is the model SemIf (github.com/TheoLeeCJ/SemIf, #2 on JevBench v1.3.0) uses. Both run as thinkers: Qwen2.5 with :thinks false, since it has no thinking mode, and Qwen3.5 with lev's ChatML (its template's no-thinking turn is the same <think>\n\n</think>\n\n).

The last three rows answer whether lev or the model was the limit. SemIf's own code on the same GGUF gets 116/144, what lev gets with its prompt, and what SemIf published from BF16. The SemIf row's balanced accuracy uses lev's metric (mean recall per label); SemIf reports 0.813 by its own. lev's prompt adds 14.5 points on the same model. Most of that is candidate_selection: 48/48 with lev's prompt, 32/48 with SemIf's. There the options are candidates named A and B, which SemIf relabels with its own letters. Qwen3.5-4B with thinking off matches MiniCPM5 with thinking (95.1%) at 174 ms a question instead of 3 s.

Jev mode and the per-question path agree case for case on Qwen3.5 (the lev-prompt rows), so llama_memory_seq_cp on its hybrid memory holds up. SemIf's llama.cpp backend avoids it and serializes whole states. What the hybrid costs is batching. Its recurrent layers need every sequence in a ubatch to be the same length, so llama.cpp splits the branches into separate passes: four questions take 533 ms against 174 ms for one. The catalog layout (short, equal branches) runs 8 states in 1,244 ms instead of 4,197 ms, but scores 72.9% on authored144, so it stays opt-in.

Escalation (lev.patterns/escalate), simulated from per-case results: the english encoder answers first, and the cases below a threshold on its confidence take the thinker's answer instead.

Qwen3.5-4B is the escalation model from here on (README, config examples). MiniCPM5-2B escalated with thinking gets 88.9% at 0.5, at seconds a case (above). Qwen3.5-4B with its own thinking on has not been measured.

Not ported from SemIf: its per-workload temperature scaling. lev.calibrate refits the encoders only. At ECE 0.034 Qwen3.5-4B needs it least.

bench/typed_decisions.clj on the test split of LocalLLaMA/typed-decisions: 400 JSON states from four workflows, five questions each, 2,000 decisions. Build the data with uv run --with pyarrow python bench/typed_decisions_data.py. The gold is the mean of three teacher samples, so accuracy (argmax against the gold's label) measures agreement with the teacher. The teacher agrees with itself 0.735 of the time, and the prior scores 0.470. Jev scores 0.727 on the dataset's card and 0.738 in Winnow's report. ECE uses 15 bins, as ollaya does.

The encoders reproduce ollaya's numbers: 0.766 for typed-decisions and 0.361 for english in Winnow's report. typed-decisions beats the teacher's own self-agreement, so it has learned the teacher's quirks along with the task.

The two prompts ported from ollaya's reference code reproduce its numbers on the fine-tunes they belong to. JevK5 scores 0.626 (ollaya 0.625, ECE 0.093). Winnow-E4B scores 0.724 (ollaya 0.722, ECE 0.022). Winnow-E4B, a Gemma 4 E4B fine-tune, is the best general model here, within 0.014 of Jev, and it is also the best calibrated.

Qwen3.5-4B, the escalation model, is well behind on this set. That is in line with ollaya's generic llm-logits reads of 4B instruct models (Qwen3-4B 0.559). At T 1 it puts nearly all its probability on one answer (KL 1.045, ECE 0.232). Temperatures fitted by jolt -M:calibrate --model qwen3.5-4b take ECE to 0.058 and KL to 0.287 without moving an answer. The fit used 200 of the train split's cases, fitted per type, with the other 200 held out: ECE went 0.363 → 0.129 for choice, 0.157 → 0.055 for noul and 0.238 → 0.076 for score.

The two general encoders answer yes to 68–77% of the noul questions, against a gold 51%. The two fine-tuned thinkers lean the same way less (58–61%), and Qwen3.5-4B does not.

Not recorded: latency. The GPU was shared with another session's bench during these runs.

Winnow-E4B Q8 (the winnow prompt, gemma4 template, T 1.2574, Jev mode, Metal) against Qwen3.5-4B Q8, the escalation model (lev prompt, numbers from the sections above).

The two models' authored144 escalation, simulated from per-case results: english answers first, and the cases below the gate take the thinker's answer.

Winnow-E4B gives up three authored144 cases and three trio cases to Qwen3.5-4B. It holds up far better on the output-blind perturbations of authored144 (+17.6 points) and on typed-decisions (+13.6). It is calibrated as shipped (ECE 0.025 on typed-decisions against Qwen's 0.232 before a fit). It is also nearly twice the file (8.0 GB against 4.5 GB), and its Gemma 4 sliding-window layers want a small :n-seq-max (8 here).

Latency is not compared: the GPU was shared with another session's bench (Winnow-E4B read 470 ms a case on authored144 under that load). So the escalation default stays Qwen3.5-4B until both are timed on a quiet machine, with Winnow-E4B the choice where robustness to rephrasing or structured states matters more than authored144.

Clef is Cloudflare's decision model: Qwen3.8-27B post-trained, with a joint schema head (a 128M-parameter transformer) in place of generation. The state and every question with its options go into one prompt. The head reads the backbone's final hidden state at every token, routes evidence to each option, lets the questions attend to each other and answers one logit per option of every question at once. No text is generated. In lev it is lev.clef, a thinker entry with :engine "clef":

python native/llama.cpp/convert_hf_to_gguf.py ~/src/models/clef --no-mtp --outtype q8_0 \
  --outfile ~/src/models/clef-backbone-Q8_0.gguf
;; config.edn :thinkers
"clef" {:engine "clef" :model ".../clef-backbone-Q8_0.gguf" :head ".../clef"}

The hidden states come from the fork's unmasked nextn embeddings (native/lev_clef.cpp). The head runs in f32 over cblas and is held to the release's torch head at 1e-4 (bench/clef_golden.py -> golden/clef). The prompt matches the release tokenizer's token for token. Answers are T 1, as shipped.

Against the escalation model, Winnow-E4B (sections above) and Clef's own base as a thinker answering at once (lev prompt, Jev mode, T 1). M1 Max, Metal, Q8 throughout.

authored144 escalation, simulated from per-case results as above (english first, the cases below the gate take the other model's answer):

Clef's post-training and head are worth a lot over its base. Against Qwen3.8-27B it gains 5 authored144 cases (rule application 75.0% -> 91.7%), 8 perturbations (to every one of them) and 0.027 typed-decisions accuracy. Its typed-decisions KL drops from 0.844 to 0.191 and its ECE from 0.149 to 0.024, with no refit. It gives two SST-5 cases back on the trio. Its confidence is informative: at >= 0.7 on authored144 it keeps 113 of 144 cases at 100%.

Against the 4B models, Clef ties Winnow-E4B on authored144 (93.1%) and on typed-decisions (0.722 / 0.724). It is ahead on the perturbations and on the trio, and it is the best-calibrated general model here. Qwen3.5-4B keeps two authored144 cases on it, and Clef is behind it in every escalation row.

Clef does not become the escalation model. It is 6x the memory of Qwen3.5-4B and 9x its time a case on authored144 (1.6 s against 184 ms, timed back to back on the same machine; 8.8 s a five-question typed-decisions state), and it loses authored144. Where it earns its cost is a decision model on its own: rephrased or structured inputs, several questions decided together, and probabilities that can be trusted as shipped. The README's Decision Index lead over Jev does not carry over at this scale of test: on typed-decisions Clef and Jev's published 0.727 are even.

TinyDecide is a 10.4M ELECTRA-small (12 layers, width 256, 4-bit weights, 6.2 MB) with typed heads. The state and each question are blocks of one sequence. A question attends to the state and to itself, so one pass answers every question and no answer depends on the others. It reads the first 127 tokens of the state. In lev it is lev.tinydecide, a thinker entry with :engine "tinydecide" whose :model is the release directory. Its full/ is the 13.8M build with the whole ELECTRA vocabulary. It runs on the C kernels, needs no llama.cpp, and is held to the release's JavaScript reference engine at 1e-4 on both builds. The parity fixtures are its conformance suite (bench/tinydecide_golden.py -> golden/tinydecide): 288 tokenizer cases, then token ids, truncation and answers for 49 requests. Answers are the release's own calibrated probabilities, T 1. M1 Max, CPU.

A choice's options go to it as their descriptions, or as the label when an option has none. The 10.4M build, label against description against "label: description":

The description wins three of the four and calibrates best. It gives back 5 of the 108 perturbations.

TinyDecide is the routing model the card describes and no more. On AG News, short everyday text with plain topics, it is 87.5% in 11 ms. It is at chance where the answer needs inference: authored144 is three-way and it scores 42%, and BoolQ is 52.5%. The card puts natural language inference near chance too. typed-decisions is below the prior (0.470): its states are JSON past the 127 tokens it reads, and it answers false to 98.5% of the nouls, where the gold says true to half of them. The 13.8M build is no better here. A typed-decisions state takes 34 ms for five questions. It does not replace an encoder, and it is no escalation target. It fits where 6 MB and 10 ms on a CPU matter more than accuracy: a pre-filter in front of an encoder, or short messages routed by topic.

A sequential before/after (two processes, one after the other) confounds the change with the machine's drift: above, the same code moved 288 → 304 ms between two runs, and laya-mlx's sequential pilot put mx.compile at 1.24x where its interleaved run found 1.03x. bench/paired.clj is that interleaved run, ported: every round takes one input (rotating over 8 state variants) and times every candidate on it in an order that rotates by one each round; the report is the median of the per-round baseline / candidate ratios with a percentile-bootstrap 95% interval (2,000 resamples), next to the p50s. An interval that includes 1 is no win. cpu,cpu is the noise floor, the same agent against itself:

jolt -M bench/paired.clj --candidates cpu,cpu --rounds 20
  short4 (4 questions)   cpu/cpu#2  paired speedup 1.000x  [0.998, 1.007]   per-round 0.989..1.207
  long4  (4 questions)   cpu/cpu#2  paired speedup 0.990x  [0.979, 1.016]   per-round 0.917..1.175
  long1  (1 question)    cpu/cpu#2  paired speedup 1.007x  [0.997, 1.021]   per-round 0.862..1.125

Single rounds swing by 20%; twenty paired rounds pin the ratio to ±2%. Anything that changes the forward (a backend, a pruned layer) is measured here before it lands.

jolt mlx builds the encoders' GPU engine (lev.mlx over mlx-c, the same forward as the C kernels as one MLX graph). Interleaved against the C kernels with bench/paired.clj --candidates cpu,mlx,mlx16 --rounds 20 on the M-series this tree is developed on, english:

Accuracy, the same runners as above:

f32 is the C kernels' answer to the fourth decimal (lev.mlx-test holds the golden batch to 1e-4 and the README answers to the oracle); f16 moves one authored144 case (in its favour, by luck) and probabilities by up to 1e-2. laya-mlx's own M3 Max numbers for the same architecture at f16 (13 ms a short single question, batch of one) are of the same order once its lighter per-call path is accounted for; lev's call carries the tokenizer (~1 ms a question), the calibration and the answer shaping. Not ported from laya-mlx, with its evidence: mx.compile (1.03x interleaved), a custom Metal GELU kernel (no consistent win over the compiled graph), 8/4-bit weights (no speedup at these shapes and 62/63, 50/63 fixture agreement), low-rank weights (84% Frobenius error at the rank a 10x needs).

Only each row's CLS and marker rows of the last head layer reach the scorer and the act head, so its out-projection, second norm and FFN (9 of the layer's 12 d² a row) run on those 1+k rows instead of all L (laya-mlx's selected_head; exact dependency pruning, the golden batch and the oracle answers unchanged). Interleaved against the whole layer, 20 rounds:

Small, every interval above 1, and it costs nothing: on by default on both backends (:selected-head false in the prepared config or, for lev.mlx, the loader's limits keeps the whole layer; bench/paired.clj --candidates fullhead,cpu / mlxfull,mlx is the comparison).

lev.agent/embed answers a state as one unit vector from the english encoder, for callers that need state similarity (cognitive-hydraulics' chunk recall). The encoder was trained for the decision head, not for similarity, so the pooling was chosen on a small check (bench/embeddings.clj): 12 short states in 4 groups (memory leak, bad config, capacity, billing), three phrasings each.

Mean pooling, raw, ships: centering widens the scale but does not move the nearest neighbour here and needs a corpus mean. The signal is real (8/12 against ~2/11 by chance) and modest, and the raw vectors are anisotropic: any two states sit between ~0.82 and ~0.95. Rank by cosine; a fixed similarity cutoff is not portable between domains. Nothing here touches an answer's numbers.