DaoyuanLi2816 · DaoyuanLi2816 · Aug 9, 2026 · Aug 8, 2026 · Aug 8, 2026 · Aug 8, 2026
diff --git a/PROJECT_STATE.md b/PROJECT_STATE.md
@@ -45,11 +45,71 @@ local HF cache; `google/IFEval`, `natolambert/xstest-v2-copy` and
 `allenai/reward-bench` reachable. No v0.7.0 tag exists, so the release number is
 uncontested.
 
+Phase A is complete and merged as `0be43b0` (PR
+[#49](https://github.com/DaoyuanLi2816/mini-verl/pull/49)).
+
+### Phase B progress — branch `v0.7.0-external-alignment`
+
+| item | state |
+| --- | --- |
+| endpoint registry | **done** (`c020b14`). `benchmarks/external-alignment/registry.yaml` pins four ungated endpoints with 40-hex revisions, licenses and redistribution decisions, every fact read from the Hub API on 2026-08-08 |
+| IFEval verifier | **done** (`c020b14`). Independent implementation of all 25 instruction types; 0 of 834 instructions unscored on the real 541 rows. `number_sentences` and `response_language` are marked `approximate` |
+| XSTest refusal classifier | **done**. Deterministic three-label classifier; over-refusal and appropriate-refusal reported separately, never averaged |
+| record schema | **done**. `TaskRecord` refuses an evaluated row without a score and a non-evaluated row *with* one; generated text is never stored, only its digest |
+| registry validator | **done**. Rejects a gated source, an unpinned revision, a judge over 3B, a gated judge, an unqualified model evaluator, or a missing category |
+| pinned-source network tests | **done**. Six checks against the real Hub confirm 541 IFEval rows, XSTest 250/200 across 18 types, JBB 100 behaviours, RewardBench 2985 across 23 subsets |
+| Granite Guardian harmful judge | **done**. `unclear` is excluded from the rate, never folded into `refused`; qualification floor required before the endpoint counts |
+| PairRM preference judge | **done**. Both orderings evaluated; an order-inconsistent pair is a tie, not a coin flip, and the disagreement rate is published beside every win rate |
+| deterministic generation | **done**. Measured on the RTX 4080: bf16 batched decoding diverges from batch size 1, float32 is byte-identical. float32 batch 8 costs 2.0 GPU h for all 8,128 final-test generations against 19.2 GPU h for bf16 at batch 1, so float32 is required and `dtype` is in the recorded config digest |
+| CLI surface | **done**. `alignment-suite prepare/validate/report`; verified end to end against the live Hub, manifest digest `e7c946b9be1bba4a8af21be342030450d5f8238dd9bc10021f88779cae3c0867`, 508 generation tasks per model |
+| evaluation profile | **done**. `profile-v1.yaml`: ifeval 128, xstest 252, jbb 64, jsonnav 64 generating (508, under the 512 ceiling) plus rewardbench 96 judged |
+| governance doc, THIRD_PARTY_NOTICES | **done** |
+| checkpoint and teacher gates | **done**. First candidate in a committed order that clears the gate, evaluated on train/eval only; a missing metric is undecidable rather than failed; no qualified teacher is a publishable outcome |
+
+Two defects reached CI that the local machine could not show, both now covered
+by a local gate:
+
+* the fake-model generation tests drove `torch.no_grad()`, so they belonged in
+  the torch-marked set rather than the torch-free subset;
+* `default_registry_path` walked up from `__file__` to find a repository root,
+  which resolves to `lib/python3.12` from site-packages. The registry now ships
+  inside the wheel, and a test asserts it resolves under any install layout.
+
+### Endpoint decisions and why
+
+HarmBench, StrongREJECT and AdvBench are all **gated**; a download attempt on
+2026-08-08 returned `DatasetNotFoundError` requesting access. Accepting dataset
+terms is an authorization the maintainer gives on their own account, and a
+gated source is not reproducible by a reader either. The harmful-compliance
+category therefore uses `JailbreakBench/JBB-Behaviors` (MIT, ungated, 100
+non-adaptive categorised behaviours). `meta-llama/Llama-Guard-3-1B` is rejected
+for the same class of reason: `gated: manual`. The judge is
+`ibm-granite/granite-guardian-3.0-2b` (apache-2.0, ungated, 2B).
+
+That endpoint is **not** reported as HarmBench. HarmBench's official classifier
+is a fine-tuned 13B model and StrongREJECT's reference judge is a paid API
+model; both exceed the compute contract, so reusing the name would misdescribe
+what ran.
+
+### Compute budget, measured rather than estimated
+
+| phase | projected | basis |
+| --- | --- | --- |
+| final-test generation | 2.0 GPU h | 0.89 s per 256-token float32 batch-8 response x 8,128 |
+| judge passes | ~1.5 GPU h | Granite Guardian 2B over 64 behaviours x 16, PairRM over 96 pairs x 16 x 2 orders |
+| teacher preparation and qualification | <= 8 GPU h | contract allocation, not yet spent |
+| checkpoint candidates and calibration | <= 4 GPU h | contract allocation, not yet spent |
+| continuation training | <= 28 GPU h | 5 methods x 3 seeds x 8 optimizer updates |
+
+The envelope is 48 GPU hours. Nothing above has been spent except the
+measurement runs that produced the generation figure.
+
 ### Next action
 
-Phase A 7.3: default new extension sidecars to schema version 2 binding
-`dataset_sha256`, `dataset_rows` and generator identity, while continuing to
-read valid public v1 sidecars.
+Merge PR B, then Phase C: run the SFT candidate checkpoints and the two teacher
+candidates on the eval split, apply the committed gates, and write
+`benchmarks/preregistration/alignment-external-v1.yaml`. The preregistration
+must merge and be public before any final-test access.
 
 ## v0.6.3 Security, artifact integrity and release-state hardening
 

diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md
@@ -73,6 +73,35 @@ both **Apache-2.0**, pinned by revision:
 No model weights are committed to this repository, and `.gitignore` refuses the
 common weight extensions so they cannot be added by accident.
 
+## External alignment benchmarks
+
+The v0.7.0 external alignment study evaluates against four upstream datasets and
+two upstream evaluator models. **None of their content is redistributed here.**
+The repository commits identifiers, pinned revisions, selected row ids, content
+digests and resulting scores; prompts and model generations stay upstream, and
+harmful-compliance generations are stored only as SHA-256 digests.
+
+| Asset | Licence | Pinned revision |
+| --- | --- | --- |
+| `google/IFEval` | Apache-2.0 | `966cd89545d6b6acfd7638bc708b98261ca58e84` |
+| `natolambert/xstest-v2-copy` | CC-BY-4.0 | `b71afe2a6d10e5a6254ea8bcb006c48b095a15d5` |
+| `JailbreakBench/JBB-Behaviors` | MIT | `886acc352a31533ffbcf4ef22c744658688086fc` |
+| `allenai/reward-bench` | ODC-By | `168d848cdbbea9764fae4a544dc9ca1e6cca4931` |
+| `ibm-granite/granite-guardian-3.0-2b` | Apache-2.0 | `e48b7b8acf438d24daa2271ada6df945b5b8895e` |
+| `llm-blender/PairRM` | MIT | `5b880cc73776ac75a835b3e0bd5169bcb5be013b` |
+
+The IFEval scorer in `src/miniverl/alignment_external/ifeval.py` is an
+independent implementation written from the published instruction semantics. No
+official scorer is installable from PyPI and none was copied; where it
+substitutes for a reference dependency the affected instruction types are
+marked `approximate` in every result.
+
+The XSTest refusal classifier is likewise independent, and is the paper's
+string-matching variant rather than its GPT-4 judge configuration. Neither
+endpoint is reported under a benchmark name whose official evaluator was not
+run — see
+[benchmark governance](docs/alignment-external/benchmark-governance.md).
+
 ## Other assets
 
 `docs/banner.svg` was drawn for this project. It uses no third-party artwork,

diff --git a/benchmarks/external-alignment/profile-v1.yaml b/benchmarks/external-alignment/profile-v1.yaml
@@ -0,0 +1,64 @@
+# The v1 external alignment evaluation profile.
+#
+# Task counts are the compute-bounded subset each model is evaluated on. They
+# were chosen against the measured generation cost -- float32 batch-8 decoding
+# of the pinned Qwen3-0.6B student runs at about 0.89 s per 256-token response
+# on one RTX 4080 -- and are frozen before any final-test access.
+#
+# The compute contract caps generation at 512 tasks per model. The four
+# generating endpoints below total 508. RewardBench is excluded from that
+# budget because its pairs are judged, not generated.
+#
+#   128  ifeval
+#   252  xstest
+#    64  jbb_behaviors
+#    64  jsonnav_utility
+#   ---
+#   508  generation tasks per model, x 16 evaluations = 8,128 generations
+#        projected at roughly 2.0 GPU hours
+#
+# XSTest is the endpoint that had to be reduced. Its full pinned split is 450
+# prompts, which alone would leave no room for the other three.
+
+id: alignment-external-v1
+selection_seed: 20260808
+
+endpoints:
+  - id: ifeval
+    # Stratified over the 25 instruction types so no verifier is
+    # over-represented. 541 upstream rows reduced to 128.
+    tasks: 128
+
+  - id: xstest
+    # 252 of 450, round-robin across all 18 types at 14 each. That keeps the
+    # safe/unsafe pairing the benchmark is built on -- 10 safe types give 140
+    # safe prompts, 8 contrast types give 112 unsafe -- rather than sampling
+    # uniformly and letting the ratio drift.
+    tasks: 252
+
+  - id: jbb_behaviors
+    # 64 of the 100 behaviours, stratified across categories.
+    tasks: 64
+
+  - id: rewardbench
+    # 96 fixed pairs stratified over the 23 subsets, each judged in both
+    # orderings. Scored from existing chosen/rejected pairs rather than
+    # generated, so these do not consume the generation budget.
+    tasks: 96
+    counts_toward_generation: false
+
+  - id: jsonnav_utility
+    # Retained miniVERL tool utility, measured with the existing deterministic
+    # environment rather than an external endpoint. Kept in the same frozen
+    # suite so the utility side of every Pareto comparison uses the same task
+    # ids as the alignment side.
+    tasks: 64
+    external: false
+
+generation_budget:
+  max_tasks_per_model: 512
+  planned_tasks_per_model: 508
+  note: >-
+    Set after the runtime pilot and frozen before final-test access. A later
+    change that needs more is an amendment to the preregistration record, not a
+    silently raised ceiling.
diff --git a/benchmarks/external-alignment/registry.yaml b/benchmarks/external-alignment/registry.yaml
@@ -0,0 +1,190 @@
+# Pinned external alignment endpoints.
+#
+# Every revision, license and gating status below was read from the Hugging
+# Face Hub API on 2026-08-08, not remembered from a paper. `revision` is the
+# dataset repository commit that the evaluation loads; a different revision is
+# a different benchmark and fails closed rather than being silently accepted.
+#
+# Nothing here redistributes prompt text. The repository commits identifiers,
+# revisions, selected row ids and content digests; the prompts themselves are
+# fetched from their upstream source by the person running the evaluation.
+
+schema_version: 1
+verified_at: "2026-08-08"
+verified_against: "https://huggingface.co/api"
+
+# Categories the study must cover. An endpoint that cannot be executed is
+# reported as not_applicable, never as a zero score.
+required_categories:
+  - instruction_following
+  - over_refusal
+  - harmful_compliance
+  - preference_reward
+
+endpoints:
+  - id: ifeval
+    category: instruction_following
+    name: "IFEval"
+    dataset: "google/IFEval"
+    revision: "966cd89545d6b6acfd7638bc708b98261ca58e84"
+    config: null
+    split: "train"
+    rows_upstream: 541
+    license: "apache-2.0"
+    gated: false
+    prompt_redistribution: "not redistributed; loaded from the upstream dataset at the pinned revision"
+    output_redistribution: "per-task scores and digests only; generated text is not published"
+    evaluator:
+      kind: "deterministic_rule"
+      implementation: "miniverl.alignment_external.ifeval"
+      model: null
+      independent_implementation: true
+      instruction_types_covered: 25
+      instruction_types_in_dataset: 25
+      notes: >-
+        IFEval scores verifiable instructions programmatically, so there is no
+        judge model, no qualification step and no order-bias concern. No
+        official scorer is installable from PyPI, so this is an independent
+        implementation of all 25 instruction types present at the pinned
+        revision, verified to leave 0 of 834 instructions unscored. Two
+        verifiers are marked `approximate` and named in every result:
+        `length_constraints:number_sentences`, which segments with a regex
+        where the reference uses nltk, and `language:response_language`, which
+        uses langdetect. An instruction type without a verifier would report
+        not_applicable, never False.
+    expected_vram_gib: 0.0
+    known_limitations: >-
+      Measures only checkable surface constraints such as length, casing and
+      keyword presence. It says nothing about whether the response is correct
+      or useful. Language identification is unreliable on very short responses
+      -- "hola amigo" identifies as Somali -- so the 31 language-constrained
+      prompts carry more evaluator noise than the rest.
+
+  - id: xstest
+    category: over_refusal
+    name: "XSTest v2"
+    dataset: "natolambert/xstest-v2-copy"
+    revision: "b71afe2a6d10e5a6254ea8bcb006c48b095a15d5"
+    config: null
+    split: "prompts"
+    rows_upstream: 450
+    license: "cc-by-4.0"
+    gated: false
+    prompt_redistribution: "not redistributed; loaded from the upstream dataset at the pinned revision"
+    output_redistribution: "per-task labels and digests only"
+    evaluator:
+      kind: "deterministic_rule"
+      implementation: "miniverl.alignment_external.refusal"
+      model: null
+      notes: >-
+        The XSTest paper reports both a string-matching refusal classifier and
+        a GPT-4 judge. Only the string-matching variant fits the compute
+        contract and needs no network, so that is what runs; the endpoint is
+        reported as "XSTest v2 prompts, string-match refusal classifier"
+        rather than as the paper's GPT-4 configuration.
+    expected_vram_gib: 0.0
+    known_limitations: >-
+      A string-match refusal classifier misses refusals phrased unusually and
+      can mislabel a compliant answer that merely mentions refusal words. The
+      measured disagreement against a held-out hand-labelled subset is
+      published with the results.
+    strata_field: "type"
+    strata_count: 18
+
+  - id: jbb_behaviors
+    category: harmful_compliance
+    name: "JailbreakBench behaviours, Granite Guardian judged"
+    dataset: "JailbreakBench/JBB-Behaviors"
+    revision: "886acc352a31533ffbcf4ef22c744658688086fc"
+    config: "behaviors"
+    split: "harmful"
+    rows_upstream: 100
+    license: "mit"
+    gated: false
+    prompt_redistribution: "never redistributed; harmful behaviour text stays upstream"
+    output_redistribution: >-
+      category-level rates and per-task labels only. Model generations for this
+      endpoint are hashed, never published.
+    evaluator:
+      kind: "classifier_model"
+      implementation: "miniverl.alignment_external.harmful"
+      model: "ibm-granite/granite-guardian-3.0-2b"
+      model_revision: "e48b7b8acf438d24daa2271ada6df945b5b8895e"
+      model_license: "apache-2.0"
+      model_gated: false
+      model_parameters_b: 2.0
+      requires_qualification: true
+      notes: >-
+        This is NOT HarmBench and is not reported under that name. HarmBench's
+        official classifier is a fine-tuned 13B model and StrongREJECT's
+        reference judge is a paid API model; both exceed the compute contract.
+        `walledai/HarmBench`, `walledai/StrongREJECT` and `walledai/AdvBench`
+        are also gated on the Hub and were not accessible, so they could not be
+        pinned even as prompt sources.
+    expected_vram_gib: 5.5
+    known_limitations: >-
+      100 fixed behaviours with no adaptive attack generation, judged by a 2B
+      classifier. This measures whether the policy complies with direct
+      harmful requests under one judge. It is not a jailbreak-robustness
+      result and not a safety guarantee.
+    strata_field: "Category"
+
+  - id: rewardbench
+    category: preference_reward
+    name: "RewardBench subset, PairRM judged"
+    dataset: "allenai/reward-bench"
+    revision: "168d848cdbbea9764fae4a544dc9ca1e6cca4931"
+    config: null
+    split: "filtered"
+    rows_upstream: 2985
+    license: "odc-by"
+    gated: false
+    prompt_redistribution: "not redistributed; loaded from the upstream dataset at the pinned revision"
+    output_redistribution: "pairwise outcomes, order-disagreement rate and digests only"
+    evaluator:
+      kind: "pairwise_model"
+      implementation: "miniverl.alignment_external.preference"
+      model: "llm-blender/PairRM"
+      model_revision: "5b880cc73776ac75a835b3e0bd5169bcb5be013b"
+      model_license: "mit"
+      model_gated: false
+      model_parameters_b: 0.4
+      requires_qualification: true
+      evaluate_both_orders: true
+      notes: >-
+        Both A/B and B/A orderings are evaluated and the position-disagreement
+        rate is reported. Inconsistent pairs stay inconsistent; no winner is
+        forced. A PairRM preference is a model preference and is never
+        described as human preference.
+    expected_vram_gib: 2.0
+    known_limitations: >-
+      A 0.4B pairwise ranker calibrated on a fixed RewardBench subset. Its
+      agreement with the RewardBench labels is measured and published; below
+      the preregistered floor the endpoint is reported as not_applicable
+      rather than scored.
+    strata_field: "subset"
+    strata_count: 23
+
+# Candidates that were evaluated and rejected, with the reason. Recorded so the
+# choice is auditable rather than looking arbitrary.
+rejected_candidates:
+  - id: harmbench
+    dataset: "walledai/HarmBench"
+    reason: >-
+      Gated on the Hub; a download attempt on 2026-08-08 returned
+      DatasetNotFoundError requesting access. Accepting dataset terms is an
+      authorization the maintainer must give on their own account, and a gated
+      source cannot be reproduced by a reader either.
+  - id: strongreject
+    dataset: "walledai/StrongREJECT"
+    reason: "Gated on the Hub; same access failure as HarmBench."
+  - id: advbench
+    dataset: "walledai/AdvBench"
+    reason: "Gated on the Hub; same access failure as HarmBench."
+  - id: llama_guard_3_1b
+    model: "meta-llama/Llama-Guard-3-1B"
+    reason: >-
+      `gated: manual` on the Hub. Manual per-user approval means a reader
+      cannot reproduce the evaluation without an approval this project cannot
+      grant, so it is unsuitable as a pinned evaluator despite fitting the
+      parameter limit.