v0.7.0 external benchmarks: pinned endpoints, judges and the frozen evaluation suite - #50
Merged
Merged
Conversation
Every revision, license and gating status in registry.yaml was read from the Hugging Face Hub API rather than remembered from a paper, and the ones that did not survive contact are recorded with the reason. HarmBench, StrongREJECT and AdvBench are all gated: a download attempt returned DatasetNotFoundError asking for access. Accepting dataset terms is an authorization the maintainer gives on their own account, and a gated source cannot be reproduced by a reader either, so the harmful-compliance category uses JailbreakBench behaviours instead -- MIT, ungated, 100 fixed behaviours with categories, and non-adaptive, which the compute contract requires anyway. Llama-Guard-3-1B is rejected for the same reason: gated manual, so a reader would need an approval this project cannot grant. The judge is Granite Guardian 3.0 2B, apache-2.0 and ungated. That endpoint is deliberately not called HarmBench. HarmBench's official classifier is a fine-tuned 13B model and StrongREJECT's reference judge is a paid API model; both exceed the contract, so borrowing the name would misdescribe what ran. No official IFEval scorer is installable from PyPI -- the `ifeval` name there is an unrelated package that evaluates Python if statements -- so this is an independent implementation of all 25 instruction types present at the pinned revision. Running it over the real 541 rows leaves 0 of 834 instructions unscored. Two verifiers are marked approximate and named in every result: sentence segmentation, where the reference uses nltk, and language identification via langdetect. Real data found a defect the synthetic cases missed: langdetect raises when text has no detectable features, which a loose variant hits after stripping the first and last line off a short answer. An empty response does not answer in Arabic, so that is a failed constraint rather than an unrunnable verifier, and it no longer propagates. langdetect is also unreliable on fragments -- "hola amigo" identifies as Somali -- which is recorded as an endpoint limitation. The alignment-benchmarks extra keeps datasets and langdetect out of the core: importing miniverl and the bridge doctor pulls in neither, nor torch.
The batch-equivalence test failed on real weights, which is what it was for. Batched decoding diverged from batch size 1 on the pinned Qwen3-0.6B student after roughly 15 agreeing tokens -- drift, not an immediate mismatch, so not a padding or masking bug. Measured on the RTX 4080: bfloat16 batch 1 vs 2..12 diverges 8192 generations: 19.2 GPU h at batch 1 float32 batch 1 vs 2..12 identical 8192 generations: 2.0 GPU h at batch 8 In bf16 a batched matmul reduces in a different order than the single-row path, shifting logits enough to flip an argmax where the top two tokens are nearly tied; greedy decoding then walks off. Keeping bf16 would force batch size 1 to stay deterministic and spend 40% of the whole compute budget on generation. float32 makes batching exact and nine times cheaper, at 3.0 GiB peak reserved. GenerationConfig now defaults to float32 and refuses a batch size above 1 in any other dtype. dtype joins the recorded config digest because it decides whether a throughput knob can change an output; batch_size stays out of it, because an OOM backoff must not change the recorded decoding rule. Unrelated, found by the same run: a hypothesis property test asserted log1mexp against log1p(-exp(x)), which cancels catastrophically for tiny |x|. At x = -2.06e-15 that reference is off by 7e-4 relative while the implementation matches a 60-digit value exactly. The reference is now log(-expm1(x)).
Alignment Lab v1 started from a policy already at 100% on its deterministic suite, so every continuation arm measured the same ceiling and no method could show anything. v0.7.0 has to start from a non-saturated policy, and the way that choice is made is what decides whether the study means anything. Two properties, both enforced here: Candidates are evaluated in a committed order and the FIRST one that clears the gate wins. Taking the best-scoring candidate instead would be selection on the outcome. Every candidate is still recorded, including the ones evaluated after the choice was made. Only the train/eval split is read. The final test is untouched, so the starting point cannot have been picked to flatter a later result. The gate requires headroom in both directions: at least two alignment endpoints inside [0.10, 0.90] and retained utility inside [0.20, 0.90]. The utility band matters as much as the alignment one -- a policy at 0.98 utility has no room to regress, so a utility cost could never be observed. A missing eval metric makes the gate undecidable rather than failed. Absent evidence is not evidence of saturation. Teacher qualification has the same shape, with ceilings for harmful compliance and over-refusal and floors for the rest. An unmeasured metric fails the gate: a teacher is qualified on evidence, not on the absence of it. If no teacher passes, that is the published outcome -- run the non-teacher baselines, keep the failed gates, have pilot recommend against OPD -- not a reason to lower a threshold after seeing the numbers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase B of v0.7.0: the external benchmark layer — pinned endpoints, evaluators,
the frozen evaluation suite and its command surface. No experiment runs here and
no scientific result changes; all eleven frozen artifacts stay byte-identical.
Endpoints, verified against the live Hub
Every revision, licence and gating status was read from the Hugging Face API on
2026-08-08, not quoted from a paper.
google/IFEval966cd89natolambert/xstest-v2-copyb71afe2JailbreakBench/JBB-Behaviors886acc3allenai/reward-bench168d848HarmBench, StrongREJECT and AdvBench could not be used. All three are gated;
a download attempt returned
DatasetNotFoundErrorrequesting access. Acceptingdataset terms is an authorisation the maintainer gives on their own account, and
a reader reproducing the study would hit the same gate.
Llama-Guard-3-1Bisrejected for the same class of reason —
gated: manual. The registry validatornow rejects any gated source or judge outright, so this cannot regress.
The harmful endpoint is therefore JailbreakBench behaviours judged by
granite-guardian-3.0-2b(Apache-2.0, 2B, ungated), and it is not reportedas HarmBench: HarmBench's official classifier is 13B and StrongREJECT's is a
paid API model, both outside the compute contract, so the name would misdescribe
what ran.
The measurement that changed the design
The batch-equivalence GPU test failed on real weights, which is what it was for.
Batched decoding diverged from batch size 1 after ~15 agreeing tokens — drift,
not an immediate mismatch, so not a padding bug.
In bf16 a batched matmul reduces in a different order, shifting logits enough to
flip an argmax at a near-tie. Keeping bf16 would force batch size 1 and spend 40%
of the whole compute budget on generation.
GenerationConfignow defaults tofloat32 and refuses batching in any other dtype.
dtypeis in the recordedconfig digest;
batch_sizedeliberately is not, so an OOM backoff cannot changethe recorded decoding rule.
Contracts that fail closed
official scorer exists on PyPI). Verified against the real 541 rows: 0 of 834
instructions unscored. Two verifiers are marked
approximateand named inevery result.
averages them — a policy that refuses everything scores perfectly on one and
uselessly on the other.
coin flip, and the disagreement rate is published beside every win rate.
unclearfrom the safety judge is excluded from the rate, not folded into"refused".
TaskRecordrefuses an evaluated row without a score and anon-evaluated row with one. Generated text is never stored, only its digest.
not_applicable.Suite and profile
alignment-suite preparefreezes the study: it resolves each endpoint at itspinned revision, selects tasks round-robin across strata from benchmark
metadata alone, and writes a digest-stamped manifest. Selection never sees a
model outcome. Verified end to end against the live Hub:
508 sits under the 512 ceiling; XSTest is the endpoint that had to be reduced.
validaterejects a result set whose revisions or task ids do not match itsmanifest.
reportemits per-endpoint means and no combined score.Also fixed
A hypothesis property test asserted
log1mexpagainstlog1p(-exp(x)), whichcancels catastrophically for tiny |x|. At
x = -2.06e-15that reference is offby 7e-4 relative while the implementation matches a 60-digit value exactly. The
reference is now
log(-expm1(x))— the test was wrong, not the code.Validation
2006 CPU tests pass (49 new), 3 GPU batch-equivalence tests pass on the RTX
4080, 6 network tests confirm the pinned sources still have the shape the
registry claims. Ruff, format, mypy, text integrity, strict docs build clean.
The
alignment-benchmarksextra keepsdatasetsandlangdetectout of thecore import path.