Can JEV's zero-shot general “intelligence” match established specialist rerankers? We compare ranking quality, time and cost on fixed candidates, using MS MARCO passage and document benchmarks, with TREC-1 WSJ as a transfer test.
The interactive reranking simulator replays the fixed WSJ lexical ranking and completed JEV scores, with a separate MS MARCO passage comparison. It reads seeded Neon tables at runtime; WSJ article bodies stay outside both Git and the simulator database.
In the web app's WSJ view, each saved BM25 search has 1,000 results, but JEV
reranks only positions 1–100; positions 101–1,000 keep their BM25 order. The
page displays the top 10 and evaluates WSJ metrics with a rank-100 cutoff
(trec_eval -c -M100). Its nDCG@10 is 0.4640 → 0.6634 and MAP@100 is
0.1729 → 0.2263, while the WSJ table below reports full 1,000-result MAP of
0.2521 → 0.3055. The MS MARCO web view replays the completed experiment on all
supplied candidates, comparing monoBERT with original-text JEV; its benchmark
table below reports the same full-candidate scope.
Rerank 41,042 supplied query–passage pairs across 43 judged queries, with no new indexing or retrieval. The TREC DL 2019 paper defines NIST passage labels as 3: exact answer, 2: answer with limitations, 1: related but no answer, and 0: irrelevant. Our primary metric, nDCG@10, uses all grades; MAP, P@10 and recip_rank treat only grades 2–3 as relevant.
| Method | Source | nDCG@10 | MAP | P@10 | recip_rank | Query median (reranking) | Estimated API cost (USD) |
|---|---|---|---|---|---|---|---|
BM25, no reranking (bm25base_p) |
Paper, Table 4 | 0.5058 | 0.3013 | — | 0.7036 | — | — |
| monoBERT | Our local implementation | 0.7177 | 0.4488 | 0.6233 | 0.8717 | 22.46 s | $0 hosted API |
| JEV matched passage text | Our run | 0.6825 | 0.4748 | 0.6116 | 0.8594 | 37.46 s | $0.762991 |
| JEV original passage text | Our run | 0.6835 | 0.4729 | 0.6163 | 0.8447 | 38.08 s | $0.761027 |
IDST BERT (idst_bert_pr2) |
Paper, Table 4 | 0.7379 | 0.4565 | — | 0.8818 | — | — |
TU Vienna neural (TUW19-p3-re) |
Paper, Table 4 | 0.6746 | 0.4113 | — | 0.8568 | — | — |
Published rows come from Table 4 of the TREC DL 2019 overview
and are historical references, not local reproductions or part of our paired
tests. BM25 is a separate full-retrieval baseline before reranking, not the
verified original ranking of our candidates. The two published neural runs are
classified as reranking; their exact candidate identity has not been audited
against our manifest. recip_rank uses the paper's NIST MRR,
not its separate MS MARCO MRR. P@10 is not reported there. Bold marks the highest
reported effectiveness value in each column among the displayed methods.
P@10 measures precision in the first ten results; recip_rank averages the reciprocal rank of the first relevant result. Higher effectiveness is better; lower time and cost are better. API costs are per complete run, not per query; — means unavailable or not applicable. Local compute is not included in API cost. Against our local monoBERT, JEV improves MAP but has lower observed nDCG@10.
See the full results and audits.
- monoBERT was implemented and run locally in this repository using the pretrained
castorini/monobert-large-msmarcocheckpoint; we did not retrain the model. - JEV matched passage text uses the decoded token windows supplied to monoBERT.
- JEV original passage text uses the dataset passage with its original casing and spacing, not its source article.
All passages fit one BERT window, so this run compares text normalization rather than context coverage. MS MARCO is monoBERT's training domain; JEV's training exposure is unknown.
Matching windows controls JEV's advantage of seeing a long article at once. The WSJ experiment below separately compares matched windows with full articles. Pairwise JEV is deferred for this passage benchmark; no original duoBERT checkpoint result is claimed here. Benchmark 2 tests larger document windows.
See the protocol and commands and frozen input manifest. Reports retain raw evaluations, model versions, timing and usage. Dataset text, model downloads, credentials and response caches stay outside Git.
Rerank the official top 100 documents for each of 57 judged queries: 5,700 query–document pairs, covering 5,679 unique documents. JEV scores overlapping windows that cover the complete title, headings and body, then uses each document's highest window score (MaxP). No new retrieval or indexing is needed. Document grades 1–3 count as relevant for MAP, P@10 and recip_rank; nDCG@10 uses all grades. These labels and queries differ from Benchmark 1.
JEV achieves MAP 0.2790 and P@10 0.8930, up from 0.2126 and 0.6684 for the supplied ranking. Its observed scores also exceed the three selected published references below; those comparisons are descriptive, without paired significance tests against the published systems.
| Method | Evidence | nDCG@10 | MAP | P@10 | recip_rank | Query median (reranking) | Estimated API cost (USD) |
|---|---|---|---|---|---|---|---|
| JEV large-window MaxP | Measured | 0.7535 | 0.2790 | 0.8930 | 1.0000 | 4.81 s | $1.762336 |
PASH pash_doc_r3 |
Published | 0.7164 | 0.2672 | 0.8526 | 0.9772 | — | — |
BERT CIP_run2 |
Published | 0.6783 | 0.2478 | 0.8140 | 0.9373 | — | — |
BERT MaxP CIP_run3 |
Published | 0.6668 | 0.2457 | 0.8175 | 0.9567 | — | — |
| Supplied ranking | Locally evaluated | 0.5116 | 0.2126 | 0.6684 | 0.8367 | — | — |
Published references come from the TREC 2021 overview, Table 2 and official NIST summaries for PASH, BERT, and BERT MaxP. PASH is a multistage ensemble; the BERT runs use different training and passage aggregation. These are published systems, not locally reproduced checkpoints.
The primary paired comparison against the supplied ranking improves nDCG@10 by 0.2419. A recip_rank of 1.0000 means the first result is relevant for every query, not that every result has the highest relevance grade. JEV training exposure is unknown. See the full results and audit and protocol and reproduction.
This uses the TREC-1 WSJ collection and 50 topics, with our own JASSjr-derived BM25 and query-expansion retrieval step. Its saved candidates and stage-1 MAP 0.2521 baseline are fixed. Scores are separate from the passage benchmark.
| Method | MAP | P@10 | recip_rank | Query median (reranking) | Estimated API cost (USD) |
|---|---|---|---|---|---|
| JEV complete document | 0.3055 | 0.6340 | 0.8063 | 4.02 s | $0.328312 |
| JEV passage MaxP | 0.3053 | 0.6000 | 0.8457 | 14.98 s | $0.605227 |
| monoBERT passage MaxP | 0.2693 | 0.4960 | 0.6715 | 30.27 s | $0 hosted API |
| Stage 1: BM25 + query expansion | 0.2521 | 0.4460 | 0.6271 | — | — |
Pointwise methods rerank the top 100 documents. Passage MaxP gives JEV and monoBERT identical windows and uses each document's highest passage score. Both cover the article across windows; only complete-document JEV sees it all in one call.
See the paired report and implementation for usage and timing details.
The stage-1 baseline is lexical, with no dense retrieval or
reranking. Five-run medians are 11.03 s indexing and 0.41 s search for all 50
topics. Stage 2 reuses its saved candidates and index; see the
reranking methodology and research program.
Changes to main do not replace the fixed baseline; original archives remain
read-only history.
The experiment workflow draws on
- karpathy/autoresearch (MIT);
- the WSJ retrieval engine derives from andrewtrotman/JASSjr (BSD-2-Clause).
This project is MIT-licensed except for upstream-derived code under its respective licenses. See THIRD_PARTY_NOTICES.txt.