OpenJev

OpenJev answers the POST /v1/systemone request shape: a free-form state plus any number of questions, each a choice, a yes/no or a score. It handles general decisions such as triage, routing, policy checks and agent monitoring, and it handles browser-agent steps: which operation? and which element? It runs on your own GPU, or on a laptop.

New in v0.2 (OpenJev-4B): live-website tasks completed rise from 30 to 39–41 of 153, out-of-domain general accuracy from 73.8 to 81.4, and browser steps on unseen websites from 75.9 to 78.8; probabilities are calibrated (ECE 0.01–0.04) and a browser step takes 184 ms. OpenJev-8B and OpenJev-1.7B are still v0.1.

• 🌐 Browser steps on unseen websites: 79% right, where open decision models trained on general data reach at most 21%.
• 🧠 General decisions stay strong. OpenJev-4B scores 88.1 on Kev decision-v7 and 81.4 out of domain, and OpenJev beats Laya on every general benchmark.
• 🎓 Better than its teacher, locally. As the System One of an open browser agent, OpenJev-4B completes 39–41 of 153 held-out live tasks, against 27 for the qwen3-max teacher.
• ⚡ Fast. 10–40 ms for a general decision and 91–322 ms for a full browser page on one RTX 5090; 77 ms on a laptop.
• 📏 Calibrated. Probabilities you can threshold: raise the bar for DONE and early stops drop sharply.

Median latency on one RTX 5090 (bf16), one request at a time. OpenJev-4B v0.1 stays available as the v0.1 revision: wev.load("alanhuangya/OpenJev-4B", revision="v0.1"). OpenJev-4B v0.2 runs on a Qwen3.5 backbone and needs transformers 5; pip install "openjev-ai[serve,fast]" adds its fast kernels on CUDA.

pip install "openjev-ai[serve]"   # installs wev-ai, the package imported as wev

import wev

m = wev.load("alanhuangya/OpenJev-4B")   # downloads once from the Hugging Face Hub
out = m.predict(
    state="Refund request: order #4411 arrived damaged, customer attached photos, first refund this year.",
    questions={
        "action": {"type": "choice", "instructions": "What should support do?",
                   "criteria": {"refund": "Refund the order.", "replace": "Ship a replacement.",
                                "escalate": "Send to a human agent."}},
        "fraud_risk": {"type": "noul", "instructions": "This request looks fraudulent.",
                       "criteria": {"true": "Likely fraud.", "false": "No sign of fraud."}},
    },
)
out["answers"]
# {'action': {'type': 'choice', 'choice': 'replace', 'probabilities': {'refund': 0.38, 'replace': 0.51, ...}},
#  'fraud_risk': {'type': 'noul', 'noul': 0.02}}

Serve it over HTTP with the same request and response shapes as a System One API:

wev serve --model alanhuangya/OpenJev-4B --port 8009
curl -s localhost:8009/v1/systemone -H 'content-type: application/json' -d '{"state": "...", "questions": {...}}'

Use it in a browser agent. An agent written for a System One API, such as jev-ultrafast, can call http://127.0.0.1:8009/v1/systemone instead of the hosted endpoint. jev-ultrafast hard-codes that URL in jev_ultrafast/model.py (SYSTEM_ONE_URL), so change that one line.

Every model below receives the same requests and is scored the same way: per-question accuracy, with each model's most probable option taken as its answer. All numbers are on held-out test splits.

General typed decisions

Browser steps. Mind2Web test split: 873 requests on websites unseen in training. A step counts when both the operation and the target element are right.

End to end on live websites. jev-ultrafast ran 153 held-out tasks with each model as its System One. A task succeeds when the agent says DONE and an LLM judge, reading the final page, agrees.

Ranges are two passes of the LLM judge over the same episodes.

A frozen browser lane for decision models: 2,346 steps (Mind2Web test on unseen websites, audited NNetNav test with its stopping and giving-up steps, and teacher steps on tasks in no train split), stored as the bench config of alanhuangya/OpenJev-Data. scripts/wev_bench.py sends each stored request to any POST /v1/systemone endpoint and reports step success, operation and target accuracy, the premature-DONE rate, ECE and p50/p95 latency, overall and per subset. It needs only the standard library and pyarrow.

wev serve --model alanhuangya/OpenJev-4B --port 8009          # or any System One server
python scripts/wev_bench.py --url http://127.0.0.1:8009/v1/systemone --name wev-4b \
  --hardware "RTX 5090, bf16" --out results/wev-bench/wev-4b-v0.2.json

Add --revision <commit> to pin the dataset, --api_key_env NAME to send a bearer token from an environment variable, and --steps_out steps.jsonl to keep every prediction. Scoring follows JevBench (argmax accuracy, top-label ECE over 10 bins, multi-class Brier, interpolated latency percentiles; failed or malformed answers count as wrong), and the output file uses its per-system layout with a browser lane block, so it can be submitted to a JevBench-style leaderboard as is.

One RTX 5090 (bf16), queried serially from the same host; results/wev-bench/ has the full results.

Interface distillation. An agent talks to its System One through typed requests. We serve a prompted LLM behind that interface while the agent works on live websites, so every logged request and answer is already a training example in the decision model's own format. An LLM judge reads each episode's final page and keeps only the episodes whose outcome it confirms. It rejected 36% of the teacher's own DONE claims.

Data. We combine those episodes with Mind2Web and NNetNav steps converted to the same request format, and with general typed-decision corpora. An LLM-judge audit of NNetNav's stopping labels overturned 31% of its DONE labels.

Model. A Qwen3 base model with its vocabulary head removed, adapted with LoRA and merged into the released weights. Each question sees the state and itself only, so one forward pass answers every question in a request, and question order never changes an answer. A pointer head scores each option against the question and normalises the scores into probabilities.

<state> page text · elements · recent actions
<q> which operation? <opt> CLICK </opt> <opt> TYPE_TEXT </opt> … <opt> DONE </opt> <decide>
<q> which element to CLICK? <opt> [1] Search </opt> <opt> [2] Sign in </opt> … <decide>

The paper has the full method, ablations and failure analysis.

pip install -e ".[train,serve,dev]"

# data
python -m wev.mind2web --out data/m2w-v2 --dev_frac 0.15 --test_frac 0.1
python -m wev.nnetnav  --out data/nnetnav-v2
python -m wev.general  --out data/general --kev_repo path/to/kev
python -m wev.external --out data/ext
python -m wev.webchain --out data/webchain-v1 --max_traces 4000   # downloads WebChain's per-step HTML snapshots
# DONE relabelling of NNetNav by an LLM judge (OpenAI-compatible endpoint from WEV_JUDGE_BASE_URL / _API_KEY / _MODEL)
python scripts/judge_done.py --data data/nnetnav-v2/train.jsonl --out judge-train.jsonl
python scripts/apply_judge.py --data data/nnetnav-v2 --judge 'judge-{split}.jsonl' --out data/nnetnav-v3-clean

# OpenJev-4B v0.2: continue from Kev-4B on the v0.1 mix, then add the GLM teacher episodes (resumable: rerun the same
# line after a kill, on any number of GPUs), export, and fit the serving temperature on dev rows
torchrun --nproc_per_node 8 -m wev.train --init_from jaredpalmer/kev-4b --lr 5e-5 --global_accum 8 --epochs 1 \
  --batch_tokens 4000 --resume --out runs/wev-4b-a --data <the v0.1 mix below>
torchrun --nproc_per_node 8 -m wev.train --init_from runs/wev-4b-a --lr 3e-5 --global_accum 8 --epochs 1 \
  --batch_tokens 4000 --resume --out runs/wev-4b-v0.2 \
  --data data/teacher-v3-glm:3,data/teacher-v1,data/teacher-v2,data/m2w-v2,data/nnetnav-v3-clean,data/general/kev-v7,data/general/typed-decisions-train:2,data/ext/td-synth
wev export --run runs/wev-4b-v0.2 --out exports/wev-4b
python scripts/calibrate.py --model exports/wev-4b --data data/m2w-v2 data/nnetnav-v3-clean data/teacher-v3-glm data/general/kev-v7 data/general/typed-decisions-train

# the OpenJev-4B v0.1 recipe; dir:K repeats a training file K times. One GPU works too: drop torchrun.
# OpenJev-8B and OpenJev-1.7B use the same mix without data/teacher-v2.
torchrun --nproc_per_node 8 -m wev.train --base Qwen/Qwen3-4B-Base --head pointer --lr 1e-4 --epochs 1 \
  --batch_tokens 6000 --accum 1 --checkpointing 0 --out runs/wev-4b \
  --data data/m2w-v2,data/nnetnav-v3-clean,data/teacher-v1:3,data/teacher-v2:3,data/general/kev-v7:2,data/general/typed-decisions-train:8,data/ext/tasksource-jev,data/ext/jev-distill,data/ext/td-synth

wev evaluate --model runs/wev-4b --data data/general/kev-transfer-v4 --split dev
wev export --run runs/wev-4b --out exports/wev-4b --check data/general/typed-decisions/dev.jsonl
pytest -q tests

# the Hugging Face dataset (parquet configs + the OpenJev-Bench test set) and its PII scan
python scripts/build_hf_dataset.py --data data --tasks tasks-v1.jsonl --out hf/wev-data
python scripts/scan_pii.py --data hf/wev-data --out hf/pii-report.json

• English only. OpenJev makes decisions and does not write text; in jev-ultrafast, typed values come from its separate text model.
• Browser targets are scored among the candidates the agent lists (8–40 per step), not every element on the page, so the numbers are not comparable to the Mind2Web leaderboard.
• Rare operations are rarely predicted: BLOCKED and scrolling have low recall. On NNetNav, OpenJev-4B says DONE too early on 8–10% of steps unless you threshold it.
• On Kev's own suite, Kev is more accurate.
• OpenJev has not been compared with Jev itself, because we have no API access.

Jun Huang and Xin Ren contributed equally (University of Electronic Science and Technology of China). The paper describes OpenJev under its earlier name, wev.

@misc{huang2026wev,
  title     = {wev: Distilling LLM Browser Agents into Open, Local System-One Decision Models},
  author    = {Huang, Jun and Ren, Xin},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.22941164},
  url       = {https://doi.org/10.5281/zenodo.22941164}
}

Apache-2.0. The model code builds on kev (Apache-2.0) and uses instruction text from jev-ultrafast (MIT); the models build on Qwen3 base models (Apache-2.0). OpenJev is an independent project, not affiliated with TypeSafe AI, and does not use Jev. See NOTICE.