Every gate here is free unless the section says otherwise: no API key, no model in the loop, seconds to run. That is deliberate — a gate you skip because it costs money is not a gate.

Most are fully offline. The exception is distil suite, which fetches public benchmark rows over HTTP the first time and caches them; after that it runs offline too, and --offline forces cached-only. It still spends nothing, because grading is deterministic rather than judged by a model.

This file is HOW to run the evals. For WHY these are the metrics — why a compression ratio without a task-success delta is meaningless, why stale is reported apart from lost, what our own negative results showed — see EVALUATION.md.

(The two filenames were close enough to confuse; this one was EVALS.md.)

git clone https://github.com/dshakes/distil && cd distil
uv sync                      # or: pip install -e ".[dev]"
uv run distil --help

uv is the supported path; make setup wraps it. Everything below assumes uv run — drop it if you have the venv active.

Run all of them the way CI does:

make gate

Or individually:

uv run distil bench                        # 9-domain corpus, deterministic oracle
uv run distil conformal --runner anthropic # graded by a real model (needs a key, costs money)

Fails if compression changes a decision the oracle can detect. This is the non-inferiority gate.

uv run distil verify

Every Tier-0/Tier-1 compression must reverse to the original bytes. No flags; it either holds or it does not.

uv run distil retention                      # corpus, tri-state recall
uv run distil retention --json               # machine-readable
uv run distil retention --max-lost 0         # gate: fail on any unrecoverable fact
uv run distil retention --live               # YOUR traffic, content-free counts only
uv run distil retention --dataset hotpotqa   # graded against a PUBLIC answer key
uv run distil retention --dataset list       # what public datasets are wired up

Reports visible recall (in front of the model) separately from recoverable (one distil_expand away, verified against the handle's restore bytes). The headline is the macro average — per-domain mean, each domain counted once — because the fact-weighted number is set by whichever domain happens to carry the most probes.

uv run distil fidelity                  # all four probes over the corpus
uv run distil fidelity --json
uv run distil fidelity --max-silent 15  # gate on SILENT failures
uv run distil fidelity --no-propagation # gate on delayed damage

Four probes, described in EVALUATION.md §6:

• artifact state — is the file's final state right, or just its name present? Splits stale (present, wrong — silent) from lost (absent — loud).
• overclaim — did the value keep its uncertainty? "approximately 4200 ms" → "4200 ms" is a distortion every recall metric scores as perfect. Reversing a bound ("at least 3" → "at most 3") is reported separately as inverted, because a dropped hedge leaves a visible gap and a reversed one leaves a confident wrong number.
• continuation — does the agent still know what is left to do?
• propagation — does a loss at turn k show up as a change at turn k+n?

On --max-silent. It counts stale + overclaimed + inverted + dropped_work — the failures an agent cannot see. Loud loss is deliberately not gated here; retention --max-lost owns it, and gating one regression in two places obscures which property actually broke.

The shipped reversible tier currently reports 9 overclaims out of 171 hedged claims on the corpus, so --max-silent 0 fails today. That is a real measured property, not a bug in the gate — see EVALUATION.md §6.2. CI gates at the measured band.

--json does not emit bare metrics. It emits a record: the metrics plus the five things you need to reproduce or compare them.

{
  "schema": "distil.eval/1",          // so a consumer knows when the shape changed
  "run":     { "at": "...", "duration_ms": 218,
               "env": { "distil": "1.40.1", "python": "3.9.25", "platform": "darwin-arm64" } },
  "subject": { "compressor": "_ServingSurface", "module": "distil.compress.strategies" },
  "dataset": { "name": "corpus", "trajectories": 9,
               "domains": [...],
               "fingerprint": "sha256:051b836358932883" },
  "grader":  { "kind": "deterministic",
               "detail": "synthetic DECISION: oracle — NOT a model" },
  "metrics": { "artifact_state": {...}, "overclaim": {...}, ... },
  "gates":   [ { "name": "max_silent", "threshold": 15, "observed": 9,
                 "passed": true, "rationale": "..." } ],
  "passed":  true
}

Why each field earns its place:

• schema — a consumer can detect a shape change instead of silently mis-parsing it.
• dataset.fingerprint — a content hash of exactly what was graded, stable across load order. Adding a trajectory changes the numbers; without this that looks like a compressor regression. This distinction has already caught a test in this repo failing for a reason it was never written to detect.
• subject — "94.7% hedge fidelity" means nothing without knowing which compressor produced it.
• grader — follows the norm set by conformal.render_grader: a synthetic oracle is never reported as a model, because that conflation is what makes a result look stronger than it is.
• gates — threshold, observed value and outcome together. A gate whose bound is invisible cannot be audited, and "passed": true with no threshold beside it is not evidence. An empty gate list is explicitly not a pass: nothing was checked.

Under --json, failure diagnostics go to stderr, so stdout stays parseable at exactly the moment CI most needs to read it.

uv run distil fidelity --json --max-silent 15 | jq '.gates'
uv run distil fidelity --json --max-silent 15 | jq -r '.dataset.fingerprint'

uv run distil suite --tier 1              # tool-calling + retrieval (the evidence)
uv run distil suite --tier 1 --tier 2     # add the harder payloads
uv run distil suite --tier 3              # the thin-payload controls
uv run distil suite --only bfcl -n 50     # one benchmark
uv run distil suite --tier 1 --offline    # cached rows only, no network
uv run distil suite --json --max-lost 0   # gate on unrecoverable golds
uv run distil suite --min-answer-recall 0.95 --min-support-recall 0.95

A collapse always fails, flag or no flag. A rich benchmark that graded cases and recalled nothing is a broken run, not a low score, so the suite exits 1 on it even with no threshold passed. --max-lost counts unrecoverable golds of both kinds — answers and support facts — because benchmarks like SQuAD carry no support facts at all, and counting only those made the gate structurally unable to see an answer regression.

Twelve public benchmarks whose answer keys were written by someone else, so a stranger can falsify our numbers against data they already trust.

It costs nothing. Grading is deterministic recall against the answer key — no model in the loop, no API key, no spend. Suites that grade with an LLM judge cost real money per tier, which makes them something you run before a launch rather than before a merge. This one is wired into make gate and the CI gate job (tier 1, n=25).

Payload class is reported with every row, and it is the most important column.

A GSM8K case is a word problem; there is nothing in it to compress, so an unchanged score proves the compressor left it alone. That is a control, not a demonstration. Only rich rows can show compression quality, and a run that grades only controls exits 1 rather than reporting a clean sheet.

bfcl leads tier 1 deliberately. Berkeley Function Calling compresses the tool schema and checks that every name the gold call is built from — the function and each argument it passes — survives compression. That is the failure an agent proxy is most likely to cause and least likely to notice, because no QA benchmark ever asks the model to act: a schema can keep calculate_triangle_area and lose the base parameter, and the call cannot be formed at all.

Measured over 100 cases: 91.4% savings, 385 gold names, 4 lost (99.0% retained). Names are de-duplicated — the function name arrives from both the schema and the gold call, and counting it twice inflated an earlier 478/99.2% figure. Names nested inside a gold argument object are graded too — conditions={"department": …} contributes all three, not just conditions. Most survive as recoverable rather than visible (2.3% visible) — they sit behind expand handles, which is what the reversible tier is supposed to do.

The gold call itself is not graded as an answer. It never appears in the schema text, so a text-recall grader cannot see it, and checking whether a model still emits it would need a model in the loop — which would make this suite cost money and stop it running in CI. What is measured is stated exactly: the names the call depends on.

A benchmark that cannot be fetched is reported as a FAILED row and exits 1. It is never dropped: a suite that silently skips what it could not load reports a clean sheet for a run that measured less than it claimed.

Not wired: LongBench — upstream ships it only as a data.zip, with no rows API and no per-task files, so there is no stdlib-only path to it that keeps distil's zero-dependency promise.

uv run distil validate

Drives the compressor against huge, unicode, nested, malformed, marker-injecting and secret-looking inputs, asserting reversibility, reject-if-bigger, recency-exactness, fail-open and content-free telemetry on every one. This gate exists because a green unit suite kept coexisting with real-traffic bugs.

uv run distil certify           # conformal risk certificate at a chosen alpha
uv run distil shadow-stats      # live decision-equivalence from wrap --shadow
uv run distil certify-trajectories runs.jsonl  # offline trajectory-risk certificate
                                # runs.jsonl: {task_id, full_success, compressed_success} per line

certify runs Learn-Then-Test and Conformal Risk Control to pick the most aggressive compression level whose decision-change rate is provably bounded at α, distribution-free and finite-sample. It assumes exchangeability: the calibration traffic must resemble live traffic. distil.drift implements an anytime-valid sequential test for that assumption breaking — it runs inside the proxy rather than as a subcommand, so recalibrate on a rolling window when your workload shifts.

Nothing here uploads content. The live meters store counts only.

distil wrap --shadow 0.1 -- claude  # collect real decision-equivalence data (10% of turns)
distil retention --live        # fact recall on your traffic
distil shadow-stats            # what shadow mode measured

The decision-equivalence eval that chose the default live certifier (claude-sonnet-5-5 at effort=low) and graded the served strategy. Method, results and diagrams: https://dshakes.github.io/distil/ab.html#model-migration.

python benchmarks/model_migration_eval.py --fetch-tau-bench          # pinned, sha256-verified
python benchmarks/model_migration_eval.py --fetch-swe-agent
python benchmarks/model_migration_eval.py --cases real --flow .claude/hillclimb/decision-equivalence-real \
    --variant baseline --model claude-opus-4-8 --reps 4
python benchmarks/model_migration_eval.py --cases real --flow .claude/hillclimb/decision-equivalence-real \
    --variant v3 --model claude-sonnet-5-5 --effort low --reps 4
python benchmarks/model_migration_eval.py --cases coding --flow .claude/hillclimb/compression-coding \
    --variant baseline --model claude-sonnet-5-5 --effort low --reps 2
python benchmarks/model_migration_eval.py --savings-report savings.json   # free, offline
python benchmarks/model_migration_eval.py --status                        # cross-variant table
python benchmarks/model_migration_eval.py --regrade                       # no API calls
python benchmarks/model_migration_summary.py                              # the committed artifact

--fake oracle|null|flip checks the wiring offline first; --limit N runs a pilot. Run from a shell that is not distil-wrapped: a wrapped shell exports ANTHROPIC_BASE_URL to the local proxy, and an eval must talk to api.anthropic.com (the harness pins it). The live certifier is chosen per command with --model and --effort on certify, eval, benchmark, frontier and conformal; distil certify --strategy served certifies the serving adapter's digests of earlier tool outputs.

The SWE-bench task-outcome harness (benchmarks/swebench_outcome/, spec specs/swebench-outcome-eval.md) has committed runs under benchmarks/results/swebench-*; docs/scoreboard.html shows them. Each arm's system prompt now starts with a per-run cache namespace, so arms run together cannot read each other's prompt cache (that confounded the 2026-10-05 head-to-head; see docs/research/why-rtk-wins.md). To add arms to a committed run whose rows have no namespace, pass --no-cache-namespace and run them apart in time.

The probes are only as good as what the corpus makes them look at. Twice now a probe has reported a perfect score against nothing:

• the HTML transform compressed 0% because no trajectory carried HTML;
• the state probes saw 4 file operations and 0 obligations across all trajectories.

If you add a transform or a probe, add a trajectory that exercises it:

# generators live in benchmarks/ — edit these, not the JSON
uv run python benchmarks/gen_web_research.py     # HTML-bearing trajectory
uv run python benchmarks/gen_agent_worklog.py    # file ops + plan + hedges

# then register it
$EDITOR corpus/manifest.json
uv run pytest tests/test_fidelityprobes.py -k exercises   # coverage is pinned

test_corpus_actually_exercises_every_probe fails if the corpus stops carrying enough state transitions, hedged claims or plan items to grade. That test is the reason a green result means something.

All six gates run per-commit (make gate = bench, verify, retention, fidelity, validate, suite). The workflow is .github/workflows/ci.yml; the same commands work locally, which is the point — make gate before pushing and CI should tell you nothing new.

How the BFCL number is matched. BFCL's golds are bare code names, so they are scored as identifiers, not as prose: a name counts only where it appears as a quoted JSON token. The generic prose matcher anchors on non-word boundaries, which credited a lost argument unit for the word units elsewhere in the schema — on the n=25 sample that was 11 of 85 golds credited by accident ('a' matched inside "tool-schemas").

The escaping is the part that makes it work. A tool schema is JSON nested inside a JSON payload, so the schema's own quotes arrive escaped: the text contains \"base\", never "base". An earlier attempt required a bare "base" and read 2.9% on a payload where every name was plainly present — measuring the matcher rather than the compressor. The rule tolerates one level of escaping, and the result is 100.0% support recall (0 retained, 70 recoverable — the schema sits behind a restore handle, which is exactly what the reversible tier is for).

The 100% is TRUE recall, and the suite says so. Visible support recall on BFCL is 0% — the schema sits behind a restore handle, one distil_expand away. Against the matched-savings truncation baseline (89.3% vs distil's 90.1%) the contrast is total: truncation keeps 0 of 70 names with nothing to recover from, distil keeps all 70 but none on screen. The report prints visible → true support: bfcl 0%→100% rather than the flattering figure alone; a reader who assumes the model can see a schema it must first expand has been misled by numbers that are each individually correct.

Fifteen golds are excluded and counted. BFCL's gold calls genuinely name arguments a, b, c (quadratic coefficients). A one-letter token occurs in almost any English text, so it can be neither credited nor failed honestly — the same problem as HotpotQA's abstractive yes/no answers, and it gets the same treatment: excluded from the denominator and reported (support_unadjudicable), printed under the table rather than left implicit. A recall computed against a quietly reduced denominator is an inflated result.