jev-test

A test harness for TypeSafe's Jev, a "System One" model: you send it a state plus typed questions (Choice / Score / Noul) and get back typed answers with confidence scores instead of generated text. Three posts' worth of tests live here:

1. Confidence audit — does Jev's own confidence score mean anything? (testcases/, below)
2. Hallucination judge — pointed at someone else's output, can Jev tell whether a claim is supported by a source passage, and how does it compare to an LLM judge? (hallucination/, further down)
3. Jev for RAG — three questions about using Jev for retrieval: do you need to tune chunking, does Jev add precision as a reranker, and can a chunk's text change its own rank (injection). (rag/, below)

Both share the same base harness: jev_harness/client.py wraps typesafe-sdk and always logs the raw HTTP response, so nothing here depends on how any particular SDK version parses a result.

Post 1 background and results: Stress-Testing Jev's Confidence.

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
cp .env.example .env   # fill in TYPESAFE_API_KEY, and ANTHROPIC_API_KEY for the LLM judge (post 2) and hard-negative drafting (post 3)

set -a && source .env && set +a
.venv/bin/python run.py testcases/off_menu.yaml
.venv/bin/python run.py testcases/calibration.yaml --model jev-1.13.0

Each run writes to results/<batch>/<timestamp>/:

• raw_responses.jsonl — the literal API request/response body for every call, one line per case
• results.csv / results.json — flattened per-question rows (case, question, type, value, confidence, expected, correct)
• summary.json — accuracy, confidence histogram, and confidence-bucketed calibration stats

jev_harness/
  client.py         wraps typesafe-sdk, always logs the raw HTTP response body
  testcases.py       loads YAML/JSON batch files into Batch/Case objects
  runner.py          runs a batch, writes raw log + CSV/JSON + summary
  stats.py           grading and aggregate stats (accuracy, calibration buckets)
run.py                CLI: python run.py <batch.yaml> [--model] [--out-dir]
testcases/
  off_menu.yaml       schema questions where none of the options fit, with/without an escape option
  ambiguous.yaml       tickets deliberately written to sit on a tone/urgency boundary
  calibration.yaml     90 synthetic cases with known-correct labels (generated, see below)
  generate_calibration.py   regenerates calibration.yaml deterministically (seed=20260921)

A batch file is YAML (or JSON): a state, one or more typed questions sent to the API as-is, and an optional expected map (harness-only bookkeeping — never sent to Jev):

batch: example
cases:
  - id: case_01
    state: "Customer message here."
    tags: [example]
    questions:
      department:
        type: choice
        instructions: "Which team should handle this?"
        criteria:
          billing: "Payment or subscription issues"
          technical: "Bugs or integration problems"
          shipping: "Delivery status, delays, lost packages"
    expected:
      department: billing

Can Jev catch hallucinations? Uses Jev as a Noul judge deciding whether a response's claims are supported by a source passage, measures accuracy/calibration against human labels from RAGTruth (MIT licensed), and compares it with an LLM judge (Claude Haiku 4.5 by default) on accuracy, cost, and latency.

Background and results: Can Jev Catch Hallucinations?.

scripts/download_ragtruth.sh                    # fetch RAGTruth into data/ragtruth/ (gitignored)
.venv/bin/python hallucination/build_sample.py   # stratified 600-case sample -> hallucination/results/
.venv/bin/python hallucination/error_types.py    # heuristic error-type tags + a spot-check CSV to review

set -a && source .env && set +a
.venv/bin/python hallucination/run_judge_batch.py            # dry-run: prints call/token/cost estimate, no API calls
.venv/bin/python hallucination/run_judge_batch.py --live      # both judges, full 600 cases
.venv/bin/python hallucination/analyze.py                     # accuracy/precision/recall, ECE, recalibration (300 random splits), cost -> CSV/JSON/PNG

.venv/bin/python hallucination/consistency.py --live           # 50 cases x 20 reps: run-to-run spread near p=0.5

Results: hallucination/RESULTS.md (regenerated by analyze.py / consistency.py, not hand-edited).

jev_harness/
  cache.py                content-addressed disk cache - both judges route every call through it
  llm_judge.py             LLM judge client (Anthropic by default; LLM_JUDGE_PROVIDER/_MODEL env vars)
  cost.py                  tiktoken-proxy token/cost estimation for --dry-run
  calibration_analysis.py  ECE, isotonic recalibration, learning curve, repeated-split summary (builds on stats.py)
hallucination/
  build_sample.py    stratified RAGTruth sample (3 task types x hallucinated/clean x 100, seeded)
  error_types.py      heuristic error-type tagging (numeric/date, negation, wrong-entity, unsupported-detail)
  run_judge_batch.py   runs Jev + the LLM judge over the sample; --dry-run default, --live to spend money
  analyze.py            accuracy/precision/recall, reliability diagrams, recalibration over repeated splits, cost/latency
  consistency.py         run-to-run consistency: repeated calls per case, spread vs. distance from p=0.5
  RESULTS.md              headline numbers (regenerated, not hand-edited)

RAGTruth's own dataset files and the sampled judge_sample.jsonl are not committed - data/ and hallucination/results/ are gitignored, since the passage text (CNN/DM, MS MARCO, Yelp) carries its own terms beyond RAGTruth's MIT license on the annotations. Everything regenerates deterministically from a fixed seed (build_sample.py's SEED), matching the same pattern testcases/generate_calibration.py uses in post 1.

LLM_JUDGE_PROVIDER / LLM_JUDGE_MODEL env vars swap the comparison judge; only the anthropic provider is implemented (see jev_harness/llm_judge.py to add another). Anthropic auth can be an API key (ANTHROPIC_API_KEY) or Workload Identity Federation - the client code doesn't change either way, since anthropic.Anthropic() resolves credentials from the environment automatically.

Background and results: Does Jev Work for RAG?.

Tests three questions about Jev in retrieval, all against a pinned jev-1.13.0. Every script is a dry-run unless you pass --live, caches every Jev response in cache/ (reruns cost nothing), and uses fixed seeds. Results are regenerated into rag/RESULTS.md by each --analyze; charts go to rag/results/ (gitignored).

scripts/download_rag_data.sh                       # SQuAD v1.1 dev + BEIR SciFact into data/ (gitignored)
set -a && source .env && set +a

# Test 1 - context rot: 50 SQuAD questions, documents of 500 / 2K / 8K / 30K tokens, answer at start / middle / end
.venv/bin/python rag/test1_context_rot.py                # dry-run (builds the documents, prices the calls)
.venv/bin/python rag/test1_context_rot.py --live         # 4 calibration calls, then 800 calls
.venv/bin/python rag/test1_context_rot.py --analyze      # AUC per length and position, charts, RESULTS.md

# Test 2 - reranker bake-off on SciFact: embeddings vs Jev Noul vs Jev Score vs a local cross-encoder
.venv/bin/python rag/test2_rerank.py --local             # first stage (Qwen3-Embedding-0.6B) + bge-reranker-v2-m3, local
.venv/bin/python rag/test2_rerank.py                     # dry-run;  add --batched for the 20-documents-per-request variant
.venv/bin/python rag/test2_rerank.py --live --batched
.venv/bin/python rag/test2_rerank.py --analyze           # nDCG/MRR/Recall, ties, overlap buckets, cost and latency

# Test 2b - hard negatives (Claude drafts, YOU review, then Jev runs)
.venv/bin/python rag/hard_negatives.py draft --live      # writes rag/results/hard_negatives_review.csv, then stops
#   edit the CSV: set status to approved / rejected for every row
.venv/bin/python rag/hard_negatives.py run --live        # only approved rows reach Jev

# Test 3 - injection: planted chunks in top-20 lists, controls, and the two guard questions
.venv/bin/python rag/test3_injection.py --local && .venv/bin/python rag/test3_injection.py --live
.venv/bin/python rag/test3_injection.py --analyze
.venv/bin/python rag/test3b_heldout_guard.py --local && .venv/bin/python rag/test3b_heldout_guard.py --live

jev_harness/
  metrics.py     nDCG, MRR, Recall, AUC, seeded bootstrap intervals (also paired differences)
  pairs.py       cached, threaded Jev requests (run_requests) used by every rag/ script
rag/
  common.py             pinned model, seeds, the Jev question wording (relevance Noul and Score, injection, guard)
  test1_context_rot.py, test2_rerank.py, hard_negatives.py, test3_injection.py, test3b_heldout_guard.py
  RESULTS.md            headline numbers (regenerated, not hand-edited)

Notes:

• Datasets: SQuAD v1.1 is CC BY-SA 4.0. SciFact is packaged by BEIR as CC BY-SA 4.0, but the original allenai/scifact card says CC BY-NC 2.0, so treat it as non-commercial. Nothing from either dataset is committed; data/ and rag/results/ are gitignored.
• Models run locally: Qwen/Qwen3-Embedding-0.6B (Apache 2.0) and BAAI/bge-reranker-v2-m3 (Apache 2.0), on the Apple GPU (mps) when available. First use downloads them from Hugging Face.
• Hard negatives are written by Claude (claude-sonnet-5-5) from the claims, so they track the claim's wording. That favours any non-embedding judge; read the Test 2b numbers with that in mind.
• The guard question was written after seeing the Test 3 attack strings, so Test 3 numbers for it are in-sample. test3b_heldout_guard.py tests it on paraphrases written afterwards.
• Dry-run estimates use a tiktoken proxy with a measured factor (see jev_harness/cost.py); real Jev token counts vary by content, so expect roughly +/-30%.

• Every run pins a model version (--model jev-1.13.0 / jev-1.13.0 in hallucination/run_judge_batch.py) rather than the jev-latest alias, for reproducibility.
• results/, data/, cache/, hallucination/results/, rag/results/, and .env are all gitignored - nothing tracked in this repo requires secrets to read.