jev-guardbench

AI agents often check every message with a second LLM, a "judge", before and after the main model answers. That check is slow and costs money on every turn.

This repo tests whether TypeSafe's Jev, a small model built for quick yes/no judgments, can do that job instead.

Jev is about 3× faster and 10–25× cheaper than the judges. It matches the best judge on PII and prompt injection and trails it on harmful content, so it misses the strict bar we set in advance. Against a Gemini Flash judge it is as good or better on three of four checks, and for many agents we think the speed and cost are worth the trade.

The full story, with the caveats, is in RESULTS.md.

• Same job for everyone: each guard got the same policy text and 7,925 rows from public datasets, across four checks.
• Rules first: what counts as "much faster" and "as accurate" was written down before any run (HYPOTHESES.md). Every later change is recorded there, with the reason.
• Fair split: thresholds were tuned on one part of the data and scored on another.
• Real agent: the guards also ran inside a google-adk agent, to measure the delay a user would feel.
• Stress tests: attacks on the guard itself, repeated runs, long inputs and 8 languages.

The guards:

1. RESULTS.md: what we found, in plain words.
2. HYPOTHESES.md: the questions and pass/fail rules, fixed in advance.
3. METHODOLOGY.md: datasets, labels and how everything was measured.
4. results/full/report.md: every table. The rows behind them are in results/full/ (no dataset text).

uv sync
uv run pytest

cp .envrc.example .envrc && $EDITOR .envrc && direnv allow   # credentials

uv run python -m guardbench freeze configs/full.yaml          # downloads data, checks it against the manifest
uv run python -m guardbench run configs/full.yaml             # resumable; failed rows retry on the next run
uv run python -m guardbench repeat configs/full.yaml          # consistency: 3 runs on 200 test rows
uv run python -m guardbench attack configs/full.yaml          # attacks on the guard: 200 violating rows
uv run python examples/adk_latency.py configs/full.yaml \
  --guards jev,gemini-flash,claude-haiku --messages 100 --out results/full/h3.md
uv run python -m guardbench report --results results/full --config configs/full.yaml
uv run --with matplotlib python scripts/figures.py           # the charts

The counted run took about 10 hours of compute and cost about $16, mostly the two judges. configs/smoke.yaml tests the pipeline for free against a local Kev server, which serves the same API as Jev. Kev is not part of the benchmark.

src/guardbench/
  tasks.py        one policy per check, rendered for every guard
  guards/         systemone (Jev), llm_judge (Gemini, Claude), cascade
  data.py         public datasets -> rows, stable dev/test split
  runner.py       async, resumable JSONL results
  metrics.py      F1, FPR, AUROC, ECE, latency percentiles, bootstrap CIs
  hypotheses.py   the pre-registered decision rules
  attacks.py      the attack texts, fixed before the attack run
  report.py       markdown report
  adk.py          guards inside google-adk callbacks
examples/         adk_latency.py, the in-agent latency measurement
scripts/          figures.py, the charts in docs/figures
configs/          guard definitions and sample sizes
manifests/        the frozen row set: ids, labels and text hashes, no content
results/full/     the counted run: every scored row, and the report