Small models.
Explicit decisions.
Give a model context and explicit questions.
Get typed probabilities over the choices you provide.
Can I get a refund?”
Watch complete games.
Selected successful episodes, from the opening state to the goal. Replayed actions from the original Qwen3.5-9B pilot.
A growing set of
inspectable decision tasks.
Prepared corpus inventory, including new retrieval and multilingual mailroom controls. These additions are independently audited data, not new training runs. Completed 2B/9B results below use the earlier release-v2 mixture; selected API probes are reported separately.
A small vocabulary.
A useful way to think.
Choose an action. Test a condition. Score a criterion. Ask independent questions over the same context, with outputs you can use directly.
{
"state": "My order arrived damaged. Please refund it.",
"questions": {
"route": {
"type": "choice",
"instructions": "Which team should handle this?",
"criteria": {
"billing": "Refunds and charges",
"engineering": "Software defects"
}
}
}
}The candidates are yours. The model assigns probabilities; your application decides what to do next.
Direct probability decisions, without autoregressive text generation or parsing generated JSON. Independent implementation inspired by Jev; no claim to reproduce its proprietary method or speedups.
Decisions in motion.
Browse the selected demos across our task categories. Successful examples and interface walkthroughs; see Results for the full evaluation.
Loading demo catalog…
No matching decisions.
Try another task name or category.
These clips make the interface inspectable. A replay of one response is not an end-to-end success rate; an interface walkthrough is not a model capability result.
Coverage still in progress
How fast is 2B?
Measured requests from the released Open-Jev-2B checkpoint. Inspect the workload, timing boundary and every raw attempt.
Loading the latency report…
All providers receive the same saved state and question semantics. The 2B numbers use a warm H100 with cache off and loopback HTTP. Hosted APIs use a fresh HTTPS connection per request, including Internet transit, TLS, routing and scheduling. These are client-observed request times, not a matched-hardware model speedup.
Swipe the table horizontally to compare providers →
Local timing boundaries and experimental cache diagnostics
| Workload | State tokens | Candidates | Cache off | Experimental cache | Validation |
|---|
For context, TypeSafe's launch article ↗ reports 70–500 ms end-to-end for Jev. That vendor range uses unspecified workloads and is separate from our measurements above.
Watch a measured request.
One recorded model response and the full timing distribution for that workload. Playback length is presentation time; the measured request time is printed on screen.
Exact request and recorded response
Quality, beside latency.
Compare categorical answers with fixed reference labels. Inspect the evidence and label limitations behind each result.
Coverage is a sample. These selected suites do not cover the full held-out inventory. Snapshot decisions are not closed-loop task or game win rates.
The same 76 cases are the direct comparison. The 2B and 9B results reuse historical predictions verified against exact row and checkpoint hashes. They add no new latency measurements. Hosted model results use new requests.
Loading the decision comparison…
Reference matches / evaluated hard-target decisions. Soft targets stay outside this count. Failed decisions count as mismatches; unattempted decisions remain pending. Ambiguous game references are disclosed in the method.
Swipe horizontally to compare all five providers →
* A probability-mass flag means the categorical answer was usable even though its probability vector failed strict validation. This table counts that answer separately as a decision; it does not validate or renormalize the probabilities.
Suite definitions and decision policy
Retrieval on TREC-DL
JevBench, public subset.
A new evaluation of the already released Open-Jev 2B and 9B checkpoints. Separate from the 76-case comparison above.
231 public tasks, out of 534 total. Original: 72 · Easy: 48 · Hard: 111. The other 303 private/judge tasks are unavailable; no full-benchmark score is claimed.
Candidate order differs on 119 of 139 Choice tasks. The native adapters preserve criteria-map order; the GPT adapter uses its task-label order. Open-Jev and Jev return native probabilities; GPT returns verbalized probabilities in constrained JSON, not token logprobs.
Timing is diagnostic. One observation per task, no warmups or retries. Open-Jev uses one local H100 with loopback HTTP and prefix cache off; hosted models use HTTPS. These heterogeneous quality requests do not establish a hardware-normalized speedup, throughput, or energy efficiency.
Loading the JevBench audit status…
| Model | Correct · Accuracy | Strict valid | Renormalized | P50 | P95 |
|---|
Strict probability-sum tolerance: 0.001. Vectors within the upstream 0.02 rounding tolerance may be normalized and scored. Score accuracy uses the most probable level; expected-value ordinal MAE is separate. Local compute cost is unmetered, not free.
Results by public tier and execution settings
| Model | Original | Easy | Hard |
|---|
Open-Jev: one NVIDIA H100 80GB HBM3, models run serially, candidate batch size 1, maximum length 16,384, prefix cache off. All streams: concurrency 1, no warmups, no retries. GPT-5.6 Luna: reasoning none. GPT-6 Astra: reasoning low. Both GPT models: maximum 4,096 completion tokens and temperature omitted.
A new 27B iteration on 148,639 frozen training rows has started optimizer updates. Its final results are pending; the table above uses the released 2B/9B checkpoints and the recorded hosted-model evaluations. A further 96,849-row community stage is tokenizer-checked and queued behind the current run; neither preparation nor training progress establishes an accuracy gain.
Show the work.
Keep the limits.
Full-data evaluation on every release-v2 test and OOD decision row. Independently audited, with failed, missing, and duplicate predictions checked.
Read the evaluation reportJF100 is a separate holdout.
100 questions × 3 option rotations, excluded from training. Final-model JF100 results remain pending.
| Model | Test | OOD | Status |
|---|---|---|---|
| Qwen3.5 2B | 94.71% | 86.02% | Audited |
| Qwen3.5 9B | 97.54% | 91.97% | Audited |
| Qwen3.8 27B | — | — | New run |
Each completed model: 26,452 held-out decision rows, including 25,492 hard-label rows used for accuracy. Test: 10,046 hard labels; OOD: 15,446.
These measure agreement with synthetic reference labels, not gameplay or workflow completion rates. A later coverage audit found equivalent game actions and omitted policy details; see the provider-comparison method. No full-data base-model comparison was run. Final-model closed-loop results remain pending.
Aggregate scores hide weak subgroups: 9B Wiki OOD expected accuracy is 32.74%, and reasoning OOD hard-label accuracy is 73.60%. T-Rex test has only 4 rows. All source-level results remain available.
Download verified resultsBuild a smaller
decision loop.
Inspect the contracts, follow the evidence, and make the next choice explicit.
Model packages require their pinned upstream Qwen base weights. The dataset card documents redistribution exclusions and reconstruction of the original training mixtures.