v0.7.0 preregistration: freeze the study and record a preregistered checkpoint-selection failure - #51

All five continuation arms need the same prompts, or the comparison measures
datasets rather than methods. Anthropic/hh-rlhf supplies them: MIT, ungated,
sharing no subset with RewardBench and no prompts with any of the four
endpoints.

Two rules that decide whether the training set means anything:

Over-long examples are dropped, never truncated. A truncated preference pair is
a different pair, and cutting a prompt can remove the context that made the
preferred answer preferred.

A pair whose two branches disagree on the prompt is dropped, as is a pair whose
branches end in the same response. Neither carries a preference signal.

The train/eval boundary comes from a hash of the example's own identity rather
than its row position, so upstream reordering cannot move an example across the
split and a re-run reproduces the same assignment.

Against the first 20,000 real rows at the pinned revision: 17,416 usable
(87.1%), split 13,928 train / 3,488 eval, median prompt 281 characters and
median response 100 -- comfortably inside the token budget.

… final test

One bounded SFT run on hh-rlhf, with adapters kept at the committed update
counts 0, 4, 8 and 16. One run means one trajectory through parameter space, so
the candidates are comparable; training a separate model per candidate would
make them four unrelated models rather than four points on one path.

Measured on the RTX 4080: 81 seconds, peak reserved 4.008 GiB against the
14.5 GiB gate. Each adapter is digested at save time, so the checkpoint every
continuation arm loads can be proven identical.

prepare_suite now takes reserved_task_ids. The starting checkpoint and the
teacher are both chosen before the final test is read; if a selection suite
drew a task the final test also scores, the one permitted read of the final
test would already have leaked into the choices the study claims were made
without it. Passing the frozen final manifest withholds those ids, the count
lands in the selection manifest as reserved_for_final_test, and a pool that
cannot fill the request fails closed rather than quietly returning fewer tasks.

Disjointness is therefore auditable from the manifests on disk, not only from
having run the test that checks it.

Once the commit carrying these is public, the models, methods, task ids,
evaluator revisions and thresholds are frozen and the final test may be read
exactly once.

The document leads with what would make the study worthless, because each item
is a way to produce a publishable-looking result that means nothing: a
saturated starting point, selection on the outcome, a leaked final test,
endpoints that secretly agree, and an unqualified judge. Each has a mechanism
against it that is already implemented and tested, not merely promised.

Values not yet measured are marked pending_selection rather than left blank or
guessed. They will be filled by the committed procedure -- first candidate in a
fixed order that clears a fixed gate -- so the record shows the rule was set
before the numbers were seen.

Failure modes are preregistered as outcomes, not as problems to work around. No
candidate clearing the saturation gate is a publishable result. No teacher
qualifying means running the non-teacher baselines, keeping the failed gates,
and having pilot recommend against OPD. A judge below its agreement floor makes
its endpoint not_applicable for every arm rather than scored.

The utility non-inferiority tolerance is fixed at 2 percentage points here,
before any arm has run, and there is deliberately no composite score: reducing
harmful compliance by refusing benign prompts is not an improvement, so the two
rates are always shown together.

…failure

Both amendments are written while the values they could be accused of
reacting to are still unmeasured. That timing is the whole point: a
contingency declared after a disappointing number is a rescue, and the same
contingency declared before it is a protocol.

Amendment 2 declares one fallback starting-checkpoint lineage. The primary
lineage continues Qwen3-0.6B on HH-RLHF, which is not a tool-use corpus, so
its retained tool utility may fall below the band floor for reasons unrelated
to saturation. If every primary candidate fails the unchanged gate, one
fallback runs from a public pre-v0.7 tool-policy adapter under the same
procedure and candidate order. If that also fails, the published outcome is
checkpoint_selection_failed and nothing downstream runs. No JSONNav utility
value has been observed for any candidate.

Amendment 3 splits the preference endpoint. RewardBench agreement measures
whether the judge can rank a pair a human already ranked; it says nothing
about whether a trained arm beats the baseline, which is what the study asks.
Reporting the former as an arm's win rate would answer a different question
than the one advertised. A separate 64-prompt held-out HH-RLHF set now
compares each arm against the shared starting policy in both orders.

That needed room under the unchanged 512 ceiling, so XSTest drops 252 -> 180:
exactly 10 per each of its 18 types, which preserves the safe/unsafe pairing
the benchmark is built on. Total generation goes 508 -> 500. The superseded
profile is kept as profile-v1.superseded-508.yaml, sha256 8dc71d10.

No band and no threshold moved in either amendment. No final-test task has
been scored.

Amendment 3 added the preference endpoint and moved XSTest 252 -> 180. The
end-to-end profile test still asserted 508, so it caught the change rather than
letting the two drift apart, which is what it is for.

Also fixes the signature drift the audit found: the harmful scorer is now
called with judge, prompts, responses and categories rather than responses
alone, and the JSONNav scorer takes a backend and manifest task ids rather than
a generator. jsonnav_utility joins the selection suite, so its eval ids are
withheld from the frozen final set like every other endpoint, and the run
aborts if any endpoint's selection intersects the final test.

The candidate loop now makes two passes. Every candidate generates while no
judge is resident, then each is released and the Granite Guardian judge loads
once for all of them -- one model workload at a time, as the compute contract
requires, instead of a candidate and a 2B classifier sharing the card.

Measured on the RTX 4080 before the full rerun: the wrapped chain works,
trainable parameters during selection are 0, and two JSONNav rollouts peak at
2.4 GiB reserved.

Both declared lineages failed the unchanged gate on retained tool utility. The
preregistration names that a publishable result, so it is recorded as one
rather than treated as work still to be finished.

    primary   Qwen3-0.6B on HH-RLHF        4/4 candidates 0.000 utility
    fallback  amendment 2 anchor           4/4 candidates 0.000 utility

Every candidate is decidable and failed, which is what authorized the fallback
in the first place: an undecidable gate is missing evidence, not a failure, so
the primary run was carried through to a complete metric set instead of
stopping once JSONNav came back zero.

The zero was validated before it was believed. Every rollout ended at
PARSE_ERROR_LIMIT having emitted zero tool calls at exactly 128 tokens -- two
turns times the 64-token budget, which is also the signature of a budget too
small to finish a call. The environment's oracle clears the identical settings
8/8 at 64, 128 and 256 tokens and at both difficulties, so the harness is sound
and the zero belongs to the models.

The diagnosis is that neither lineage ever had the competence the endpoint
measures. The anchor's own manifest records training_task.environment =
tool_policy with a 1.0 strict success rate and a 1.0 parse-valid tool-call
rate: real tool-protocol competence, for a different environment's tool set.
JSONNav has its own tools, so the anchor at zero continuation updates scores
0/64 too.

The endpoint is not being changed. The preregistration named JSONNav, and
swapping it after seeing the result is the exact move preregistration exists to
prevent. No third lineage is invented either.

The alignment axis was never the problem -- instruction following stayed in
band throughout and the fallback's update-016 reached 0.560 over-refusal
against 0.433 harmful compliance, which is the trade-off this study was built
to measure. It is the utility axis that ended it.

No final-test task has been scored.