Everything in this directory is measured. Nothing is estimated, extrapolated or projected. If a configuration was not run, it says "not run" rather than being quietly omitted.

benchmarks/
├── configs/     benchmark specifications (BenchmarkConfig YAML)
├── results/     measured results (BenchmarkResult JSON + Markdown)
├── schema/      the JSON Schema that every result file must satisfy
└── README.md    this file

# CPU, toy models, minutes
miniverl benchmark recipes/benchmark_calc.yaml --output runs/benchmarks

# One 16 GB GPU, the real Qwen3 pair, equal optimizer updates
miniverl benchmark benchmarks/configs/gpu_calc_hard.yaml --output runs/benchmarks

# Offline equivalent after downloading the pinned protocol adapter
miniverl benchmark benchmarks/configs/gpu_calc_hard_local_adapter.yaml \
  --output runs/benchmarks

Each run writes <name>.json and <name>.md into the output directory. The Markdown is for humans; the JSON is the artifact to submit.

The primary GPU config uses the public protocol-teacher adapter at immutable Hub revision 23323751318135484c06c043b1f9b9e7016dd89f. The local-adapter config is the explicit offline equivalent.

The v2 configuration declares budget_axis: optimizer_steps. Every arm receives, by construction:

• the same environment, difficulty and split_seed, so the same task instances;
• the same initial weights — one shared supervised cold start, loaded weights only so no Adam momentum leaks from the cold start into whichever arm happens to resemble it;
• the same number of optimizer steps and the same effective batch size (gradient_accumulation_steps trajectories per step);
• the same maximum trajectory length, turn limit and rollout bounds;
• the same held-out split, evaluation seed and greedy temperature.

This is an equal-optimizer-update comparison, not a generic matched-compute claim. Arms differ only in paths declared under allowed_differences; resolved leaf diffs are checked before any model is loaded. Four quantities cannot be matched by construction, so they are measured and reported per arm:

Listed so you can check that this repository does not.

• Unmatched steps. Give the favoured arm more optimizer steps. Countered by the shared budget above; optimizer_steps is in every row, so check it.
• A different starting point. Let one arm keep the cold start's optimizer state. Countered by loading weights only.
• Cherry-picking a seed. Countered by reporting every seed's row and the aggregate, and by flagging single_seed when there is only one.
• Claiming saved teacher FLOPs. Reducing selected positions reduces the LM head, the cache and the loss positions — not the teacher's forward pass. The field is named teacher_queried_position_ratio for that reason.
• Reporting a saturated task. If every arm scores near the ceiling, the benchmark measures nothing. Both saturated results in this repository say so in their notes.

All 18 completed arms start from one checksummed SFT checkpoint and retain the same 48 ordered final-test tasks within each of three preregistered seeds. The task-level JSONL contains 864 rows. DPO provenance includes the external pinned TRL 1.8.0 training cost; teacher-query ratio is explicitly a selected-position ratio, not teacher-backbone FLOPs.

The State × Supervision file is a measured signal diagnostic, not a separately trained hard-target result. The fresh soft target retains 0.0251% mean probability mass beyond argmax under matched states, teacher, budget, checkpoint and seeds; no soft-target quality advantage is claimed. All public Alignment Cards are under alignment-cards/alignment-lab-v1/.

This is a performance and numerical-equivalence experiment, not a task-quality comparison. The frozen JSON and profiler summary are generated by scripts/benchmark_consumer_runtime.py; the SVG is regenerated and byte-checked by scripts/publish_consumer_runtime_artifacts.py. Revision 1.2 of the public preregistration locks eager attention and NF4 weights with FP32 compute after two non-headline diagnostics. The final result records identical trajectory and teacher-target digests plus 12 passed loss/gradient/update equivalence checks.

RecoveryBench asks whether scoring fresh current-student states improves SQLite tool-error recovery over distillation on a fixed state set collected from the cold-start student. It is a mechanism study, not an alignment benchmark. The primary hypothesis was not supported, and fresh supervision was substantially more expensive in this implementation.

The three result JSON files are immutable source artifacts. The compact task-result JSONL, paired analysis and three SVGs are generated from them by scripts/publish_recoverybench_artifacts.py. The final run follows the public revision-1.3 preregistration; one earlier partial run with an oracle-schedule defect and one post-run aborted wall-budget replacement are preserved outside the publication set and are not used in any headline result.

results/rtx4080-calc-hard-matched.json is preserved byte-for-byte as a schema v1 measurement and is now superseded by the v2 experiment specification. Its continuation comparison remains valid in one important respect: every arm started from the same checkpoint. Its provenance/accounting fields do not mean what their names implied:

• the shared cold start was trained on calculator medium;
• continuation and held-out evaluation were on hard;
• the controlled block was copied from the base recipe rather than the resolved arm configs, so it incorrectly reports base values such as difficulty: medium, learning rate 1e-4, cosine scheduling and 48 test tasks instead of the continuation settings;
• selected_training_tokens is only the final cycle's count, not the run total;
• SFT's teacher-query ratio is not semantically meaningful because no teacher was queried.

Schema v2 makes the transfer design explicit, hashes the resolved cold/arm configs, sums accounting over the full run and records SFT teacher-query fields as null. Do not use the legacy controlled or token-total fields for a new comparison.

The historical peak_allocated_bytes values were measured during each arm. Historical peak_reserved_bytes may include CUDA allocator state retained from an earlier sequential arm. The result files are not rewritten for that memory caveat; future comparisons use the lifecycle-isolated harness.

The original schema-v1 config is readable at configs/gpu_calc_hard_legacy_v1.yaml. Current miniVERL deliberately does not execute it. Reproduce it only from immutable commit 3383f2b9a3c595e0fa143fecdc27522ab368b27f, where its original filename was configs/gpu_calc_hard.yaml.

The toy benchmark was run twice. The first attempt gave every arm a fresh cosine schedule at the base learning rate after the cold start, which restarted the learning rate and damaged the model; the second uses a constant, lower rate. Both facts are recorded here rather than only the second run being shown. Only the second is published, because the first measured a bug in the harness rather than a property of any arm -- but it happened, so it is written down.

The published run covers two seeds, and the two-seed spread is what makes the point:

Changing the seed moves a single arm by up to 58 percentage points. Changing the objective moves it by at most 4 points at a fixed seed, and the four OPD arms are indistinguishable from each other at both seeds. The between-seed variance is more than an order of magnitude larger than the between-arm variance, so no ranking can be read out of this table -- which is precisely what it is here to establish. What it does establish is that all seven arms, including exact_full_vocab, run to completion under identical budgets and produce well-formed trajectories.

One detail worth not mistaking for a defect: opd-exact reports 0.00 MiB of cache. Exact mode with a resident teacher rebuilds the full-vocabulary distribution one chunk at a time and deliberately never persists a [positions, vocab] tensor, so an empty cache is the designed behaviour.

miniverl train <recipe>
miniverl export-benchmark runs/<run-id> --out benchmarks/results/<gpu>-<recipe>.json

export-benchmark sanitizes the run: it keeps the GPU model, VRAM, capability, driver, OS family and library versions, and drops absolute paths and anything identifying. Read the file before you post it.

Validate it before opening the pull request:

python - <<'PY'
import json
from miniverl.evaluation.schema import BenchmarkResult
BenchmarkResult.model_validate(json.load(open("benchmarks/results/your-file.json")))
print("valid")
PY

Then open a pull request, or use the "Benchmark submission" issue template if you would rather not. State every deviation from the shipped recipe; a result from a modified recipe is welcome as long as it says so.

RecoveryBench runs write schema v3; current calculator harness runs write v2. The reader and generated JSON Schema continue to accept preserved v1 and v2 artifacts, and old measurements are never rewritten during migration. schema/benchmark-result.schema.json is generated from the Pydantic model, so the file and implementation cannot drift:

miniverl schema --out benchmarks/schema/benchmark-result.schema.json

CI regenerates it and fails if the committed copy differs.