jev-rerank-bench

I gave TypeSafe's Jev thirty search results and asked it which ones were useful. Then I gave Cohere and ZeroEntropy the same passages. This repository contains the experiments, saved responses and scoring code.

The ranking average put Jev's rubric at 0.692 and Cohere Pro at 0.691, without establishing a winner. Giving every query equal weight instead puts Cohere ahead. Jev did better on the negation test than the rerankers; the open-weight Open-Jev 9B, run afterwards in its own block, reads negation better than Jev asked the same way (77% vs 71% of pairs). An open-source Qwen recipe improved substantially when I gave it one passage at a time, but still showed no clear gain over keyword ranking.

The main comparison covers eight English datasets. Five more BRIGHT subsets, NevIR negation pairs and MIRACL French are reported separately. Models start with the same thirty BM25 candidates, cut to 2,000 characters each; the duel variant only compares the first ten, and NevIR supplies its own two-passage pairs.

Measurements began September 16, 2026; the Qwen controls were added September 17. Jev calls used jev-latest, reporting version 1.13.0. The evidence viewer includes the original benchmark and the Qwen follow-up, with individual questions, scores and saved outputs.

• Ranking quality: Jev rubric minus Cohere Pro is +0.001, with a 95% interval of −0.009 to +0.012. This establishes neither a winner nor equivalence. With equal weight per query, Cohere scores 0.756 and Jev 0.738.
• Top-ranked passage: Jev Choice leads Cohere Pro by 3.1 percentage points (95% interval +0.7 to +5.6). This metric ignores the separate none option; it is not the accuracy of the answer Jev actually selected.
• Negation (NevIR, 1,383 pairs): Jev rubric 71% of pairs right, Cohere Pro 67% (gap +4.2, range +1.5 to +6.9), ZeroEntropy 61%; the chat-model baseline 17–22%, below the 25% of guessing.
• Reasoning (7 BRIGHT subsets, 307 questions): Jev rubric averages 0.493 and DeepSeek JSON 0.487; their difference remains unresolved. These subjects are a separate comparison, not added to the eight-dataset headline.
• Qwen's one-passage control reaches 0.471, up from 0.255 with thirty yes/no fields. BM25 scores 0.486; the Qwen-minus-BM25 interval crosses zero. Details and attribution are below.
• Dataset differences: Cohere Pro leads Jev's yes/no batch by 5.0 points on FiQA, its rubric by 3.9 on Natural Questions, and its Choice setup by 6.4 on French. zerank-2 leads the rubric by 2.1 points on TREC-COVID.

The intervals are exploratory paired bootstraps, without adjustment for the multiple comparisons. NevIR kept passage order fixed and reuses source passages across some pairs. Its 25% chance rate assumes independent random choices for the two questions.

Table latency is the mean of dataset medians, not a pooled median. API calls include network and serving time; Qwen timings are model calls on rented GPUs. Costs average the eight dataset rates over all 2,327 original queries, including queries without relevant candidates. API costs use saved usage and recorded rates; Qwen costs cover measured GPU time, excluding setup and idle time. These are not invoice-verified or equivalent deployment costs.

Full tables, per-dataset numbers, latency percentiles, usage totals and every check: results/summary.md, results/*.json, and the charts in results/.

Harsha Gundala's original recipe uses MLX on Apple Silicon. I tested Shreyansh Singh's Transformers/PyTorch port on rented RTX 4090s. The base model is Qwen2.5-1.5B-Instruct, from Alibaba's Qwen team. Despite “1B” in the recipe's name, it is a 1.5-billion-parameter model. The port adds serving code and presets, with no additional fine-tuning; it does not reproduce Jev's training or establish that their architectures match.

The tested run_parallel_generation processes the shared prompt once, then reuses its internal cache to evaluate the fields in a batch. It scores the permitted labels and assembles JSON in code. Prefill and field evaluation are separate model forwards, and some labels need continuation scoring. I did not test the port's separate tree mode. Allowed types and normalized label scores do not guarantee correct judgments or calibrated confidence.

With thirty passages together, Qwen scored 0.255 using yes/no fields and 0.340 using the rubric, both below BM25's 0.486. Jev's rubric led Qwen's rubric by 35.2 points (95% interval +33.2 to +37.4). Giving Qwen one passage per prompt raised it to 0.471: +21.6 points over its thirty-passage yes/no setup, but no demonstrated improvement over BM25 (difference −0.015; 95% interval −0.033 to +0.004). NevIR paired accuracy was 7% for the batch yes/no configuration, 19% for the rubric and 34% for the one-passage configuration.

Reversing the thirty passages changed Qwen's top-ranked passage on 1,498 of 1,617 queries (92.6%), compared with 400 (24.7%) for Jev Choice. As above, these are passage rankings, excluding Jev's none option. No repeated identical Qwen requests were run. This measures order sensitivity, not repeatability or proof that position alone determines the answer. The one-passage control also changes context length and prompt structure; it does not isolate parallel decoding as the cause of the quality difference.

rlcd_check.py provides the score diagnostics: the relevant-minus-irrelevant yes/no score gap was 0.046 for Qwen and 0.430 for Jev with the same wording; Qwen's mean score was 0.195 at slot 1 and 0.384 at slot 30. Under reversal, mean absolute score changes were 0.160 for Qwen and 0.013 for Jev Choice. Those last two values use different score scales—independent yes/no scores versus a distribution across choices—so they are not a relative stability measure.

This tests the port's ranking quality. It does not reproduce the original quantized Mac demo or test its advertised JSON-generation speedup.

• Ranking: nDCG@10, Top-1, Recall@5 and MRR@10 over questions whose BM25 top-30 holds at least one labelled relevant passage. nDCG@10 rewards useful passages near the top; 0.69 does not mean 69% of questions answered correctly. It uses linear relevance gains and all supplied relevance labels for the ideal ranking. Ties keep BM25 order; the report also breaks ties against each model as a sensitivity check.
• Latency: client-observed API timings at the stated concurrency, plus Qwen model-call timings on rented GPUs. network.py records connection and server-header diagnostics; it does not isolate equivalent model-only times across providers. Cohere used OpenRouter, and ZeroEntropy served many calls in its slower fallback mode.
• Cost: saved usage multiplied by the recorded rates, or OpenRouter's returned usage.cost. These fields were not checked against invoices. The Qwen estimate covers measured GPU time only.
• The "nothing relevant" test: for every question, a twin list with every relevant passage removed and refilled from further down BM25. AUROC of the top score, and the false-accept rate at 90% recall. For Jev also its built-in none option and its "does any passage answer it?" question.
• Calibration: ECE and reliability curves for probability-like scores. Normalizing scores does not establish that their confidence values match observed error rates.
• Uncertainty: significance.py uses 10,000 paired bootstrap resamples of queries within each dataset, then averages across the fixed dataset set. An interval crossing zero is inconclusive, not evidence of equivalence.
• Jev under the microscope: TypeSafe's confidence bands against real answers, order sensitivity (same 30 passages reversed), repeated requests and cold start after 1–15 minutes idle. These repeat tests were on Jev, not Qwen.
• Shared text: batching.py records one request per batch size with the same thirty SciFact passages. Each query adds two typed questions and roughly 443 input tokens. Forty queries cost 2.7 times one query in those observations. Only the first query's choice was tracked; complete batched answers were not retained. This does not demonstrate quality across all forty answers or equivalent-quality savings over separate requests.
• Negation: nevir_eval.py scores NevIR by paired accuracy: both questions must rank their correct passage strictly higher; ties fail. Fixed passage order, repeated source passages and unadjusted comparisons limit inference.

* questions whose BM25 top-30 holds at least one relevant passage.

Candidates. BM25 (bm25s, Snowball stemmer, language stopwords) top 30 per question over the whole corpus (for MIRACL, over the 100-passage pool MTEB ships per query). Models share the candidate lists and 2,000-character truncation; the duel's top-ten restriction and NevIR's two-passage pairs are listed separately. These are results for this candidate-generation pipeline, not official full-benchmark scores. Truncation can remove relevant text, and the source relevance labels may be incomplete.

The yes/no variants of Jev, DeepSeek and Qwen get the same wording: "Does the passage contain the information needed to answer or verify the query?" (rerankers/__init__.py). Cohere and ZeroEntropy take the query and the documents.

Laya (Convai Innovations, 421M, Apache 2.0) is the open-weight model with Jev's question shape: Choice, Score and yes/no answered with probabilities. Its window is 512 tokens, so it ran one passage per prompt (the only shape it allows), with Jev's 4-level rubric and with Jev's yes/no wording, on rented RTX 4090s; the like-for-like Jev row is "Jev yes/no per pair" (0.670). The three GLiNER2.5 models are span and label matchers, not built for this, listed for completeness. Rows copied from the render; never averaged into the headline.

• Laya rubric vs BM25: gap -0.3 points, range -2.3 to +1.6, within noise. NevIR pairs right: 36% (Jev rubric 71%).
• Laya's 512-token window cut 0.3% of passages on Natural Questions and 88.9% on BRIGHT StackOverflow (the cut counts are in results/small_models_headline.txt).
• GLiNER2.5 multi could not process 1 of 7,896 lists on a 24 GB card (a 19,341-character question); it is stored as failed, not scored. GLiNER ran out of GPU memory on 139 code-heavy lists at 30 passages per batch; they were redone in batches of 6 or 1 (same scores, slower).
• torch 2.4's fused attention gave NaN scores for Laya on mixed-length batches; fixed by zeroing NaNs at padded positions after each layer; batched scores match the library's own predict() within 0.004 (English) and 0.012 (multilingual). Libraries: laya 0.3.3, gliner2 2.0.0.
• Runner: small_models_runner.py (same cache rows as run.py, GPU seconds × the pod's hourly price).

Open-Jev (Zefan Cai, github.com/Zefan-Cai/Open-Jev; MIT code, Apache 2.0 adapters) is a LoRA adapter plus a scalar head on Qwen3.5-2B and Qwen3.5-9B trained to answer the same three question types through the same request format as Jev. It ran through this repo's own Jev request builders pointed at a local copy of its server (JEV_URL), one passage per prompt, Jev's exact yes/no wording and criteria, on rented GPUs (2B: RTX 4090s, four server copies per card; 9B: L40S, A100, H100, RTX PRO 6000, two to four copies per card). Run 2026-09-22.

• Open-Jev 2B vs BM25: gap +5.8 points, range +3.6 to +7.9, real.
• Open-Jev 9B vs BM25: gap +11.4 points, range +9.3 to +13.5, real.
• Jev yes/no per pair vs Open-Jev 9B, same shape: gap +7.0 points, 95% range +5.6 to +8.5, real. NevIR pairs right: 2B 73%, 9B 77% (Jev yes/no per pair 71%).
• Time and cost: GPU wall time of each call with several server copies sharing one card, so the per-list time carries contention and the dollar figure divides the pod price by the number of copies; both are rough. Passages longer than the servers' 4,096-token window (10 lists per size, all in BRIGHT robotics and StackOverflow) were redone on a 16,384-token server; every response is in cache/open-jev-*.
• Scripts: openjev/pod.sh, openjev/fix.sh, openjev/merge.py; the run's own tables: results/eval_run_sept22.txt, results/significance_run_sept22.txt.
• Setup: pip install -e '.[train]' of Open-Jev, adapters ZefanCai/Open-Jev-2B (rev 0c7aa49) and Open-Jev-9B (rev 47e9668), python -m jev.server --checkpoint models/Open-Jev-<N>/package/checkpoint --max-length 4096, then JEV_URL=http://127.0.0.1:8791/v1/systemone JEV_MODEL=open-jev JEV_GPU_RATE=<pod $/h ÷ copies> uv run run.py --model jev-noul-pair --cache-as open-jev-<n>-noul-pair --dataset all.

• Same candidate lists and truncation, with the duel and NevIR exceptions documented above; shared wording for the yes/no variants.
• Ties keep BM25 order for every model; eval.py also reports ndcg10_ties_against, tied_top_share, zero_score_share.
• The 8-dataset headline was fixed before the BRIGHT block and NevIR ran; they are reported separately, not averaged in.
• Public datasets may be in any model's training data. Stated, not fixable.
• API clients ran from Algeria; providers and routing paths differed. Qwen ran on rented GPUs.
• Ranking responses are in cache/, one JSONL line per query, gzipped. Additional diagnostics save the fields listed in their scripts.

uv sync
cp .env.example .env            # JEV_API_KEY, OPENROUTER_API_KEY, ZEROENTROPY_API_KEY
uv run candidates/build.py      # downloads the datasets, builds BM25 top-30 and the "absent" twins
uv run candidates/build_nevir.py
uv run run.py --model jev-score-batch --dataset all --workers 16
uv run run.py --model cohere-pro --dataset all --workers 12
uv run run.py --model zerank-2 --dataset all --workers 6
uv run eval.py                  # results/summary.md, results/summary.json, charts
uv run significance.py          # bootstrap ranges for every gap
uv run nevir_eval.py            # negation, paired accuracy
uv run rlcd_check.py            # signal and position drift of the Qwen RLCD scores
uv run network.py; uv run batching.py; uv run determinism.py; uv run coldstart.py 60 120 300 600 900

Runs resume: a query already in cache/ (plain or .gz) is skipped, and a failed row is redone on the next run. --limit N runs N uncached queries as a smoke test. To re-score saved results without model calls, run only eval.py, significance.py, nevir_eval.py and rlcd_check.py. The run.py, network, batching, determinism and cold-start commands above make new paid requests. Recorded API usage was about $61; that excludes GPU rental costs and has not been reconciled against invoices.

data/loaders.py        dataset download + one common shape
candidates/build.py    BM25 top-30 per query plus the "absent" twin  ->  candidates/<dataset>.jsonl (ids, queries, BM25 scores)
candidates/build_nevir.py
rerankers/             jev.py  cohere.py  zerank.py  llm_logprob.py  bm25.py
run.py                 one model on one dataset, both variants, caching every raw response
cache/<model>/         <dataset>.<variant>.jsonl.gz  saved ranking responses, scores, latency, usage, cost
eval.py                metrics, nothing-relevant test, calibration, charts  ->  results/
significance.py  nevir_eval.py  rlcd_check.py  network.py  batching.py  determinism.py  coldstart.py
rlcd_runner.py         the self-hosted Qwen RLCD recipe, run on a GPU pod, writing the same cache rows
small_models_runner.py the self-hosted Laya / GLiNER2.5 runs (one passage per prompt), same cache rows
openjev/               pod.sh (one GPU pod: install Open-Jev, adapter, server copies, sharded run.py), fix.sh (full-window rerun
                       of the long lists), merge.py (merge the pod shards, verify coverage, write cache/)
blog.py                renders an earlier experiment write-up from results/*.json
scripts/readme_header.py  draws the README header from saved scores and the website fonts

The full candidates/*.docs.jsonl passage files are not committed; build.py regenerates them from the datasets. Saved model responses may quote source text, and the evidence exporter includes candidate snippets with attribution.

The interactive evidence viewer lives in the personal website repository and is available at anessbelbati.com/lab/jev-reranking/. This benchmark repository contains the experiments, saved results, and the data exporter; it does not contain a separate website app. The viewer includes the Qwen batched yes/no, four-level rubric, one-passage and reversed-order runs. The reversed-order run is an order-sensitivity diagnostic over the original eight datasets. Missing runs remain marked as missing, including Qwen on MIRACL French. Qwen costs are GPU rental estimates and its timings are measured on the GPU host; neither includes client network time or pod setup. The source responses are also available in this repository's cache/qwen-rlcd-* directories.

With the local candidate passage files available, export the curated snapshot:

uv run scripts/export_evidence.py
# Or write directly to a website checkout:
uv run scripts/export_evidence.py --output <website>/public/lab/jev-reranking/data
uv run -m unittest scripts/test_export_evidence.py

The default output is exports/evidence/ (Git-ignored). The export contains allowlisted saved outputs, the candidate snippets used in the run, prompt templates, attribution, source hashes, and downloadable dataset archives. Regeneration reads local records and makes no paid model calls.

Usage totals per run (tokens, search units) are in results/summary.md so the cost can be checked against the vendor dashboards.

Code: MIT. Dataset content keeps its source licences (BEIR, MTEB/BRIGHT, CodeSearchNet, NevIR and MIRACL), including text quoted in saved responses or included in evidence exports. The exporter records source attribution; the code licence does not replace dataset licences.