Follow-on to toy#161, which deliberately shipped without it.
Why
toy#161 runs both phases in ONE process, which is what guarantees every retrofit arm starts from a bit-identical backbone. That is the right default and it stays. But it means every arm re-pretrains — the 3-seed × 5-cell matrix in toy#161 spent well over half its wall time re-deriving the same backbone — and it makes the workflow the F12 spec actually asked for impossible:
# pretrain ONCE
toy train gtx --steps 1500 --policy chain,chain --ckpt-every 1500 --out $BB
# retrofit MANY arms from that one backbone
toy train gtx --retrofit --load-ckpt $BB/step_1500.gguf --adapter-policy dfa --lr 0.001
toy train gtx --retrofit --load-ckpt $BB/step_1500.gguf --adapter-policy chain --lr 0.003
toy train gtx --retrofit --load-ckpt $BB/step_1500.gguf --adapter-policy frozen
It also decouples the two phases for Tao: pretrain on one machine/spec, sweep retrofit arms elsewhere.
What to build
--ckpt-every K on gtx (env GTX_CKPT_EVERY): write the BACKBONE weights — the span toy#161 already tracks as @gx_backbone_count, i.e. everything up to and including the pretrain head — as a GGUF into the run dir, at step multiples of K and at the final step. Reuse the tnn_gguf_w_* surface franken-moe already uses (toy#131); no new shim.--load-ckpt PATH on gtx (env GTX_LOAD_CKPT): validate shape metadata against the live instrument and fail loud listing both sides on any mismatch, then overwrite each backbone tensor BY NAME. A backbone loaded under a different --d-model/--heads/--features/--types would otherwise produce confident garbage.--load-ckpt implies the pretrain phase is SKIPPED — the point is not to redo it. Require --retrofit alongside it and reject the combination otherwise, rather than inventing a second meaning for a loaded checkpoint on this lane.
Gate legs
- round trip: the
backbone: sig_pre= printed by a load equals the sig_post= the writing run printed — every weight came back bit-exact, not approximately.
- shape mismatch fails loud: a checkpoint written at one
--types/--d-model rejected by an instrument built at another, naming both sides.
--load-ckpt without --retrofit rejected; a missing file rejected; --ckpt-every with no run dir rejected (nowhere to write is not a silent no-op).- the plain lane stays byte-null — neither flag may perturb a run that does not pass them (toy#160's fixture already catches this and has caught two silent bugs during toy#161).
What this does NOT change
Not the toy#161 result, and not its one-process default: same-process remains the way the bar is measured, because bit-identical-by-construction is a stronger guarantee than a round-trip that itself has to be gated. This only removes the cost of re-pretraining for sweeps that do not need that guarantee.
Refs
toy#161 (the retrofit lane + the backbone_sig instrument this leans on), toy#131 (--ckpt-every/--load-ckpt on franken-moe — same GGUF surface, same fail-loud-on-mismatch discipline), toy#160 (the byte fixture that guards the inert path).
Follow-on to toy#161, which deliberately shipped without it.
Why
toy#161 runs both phases in ONE process, which is what guarantees every retrofit arm starts from a bit-identical backbone. That is the right default and it stays. But it means every arm re-pretrains — the 3-seed × 5-cell matrix in toy#161 spent well over half its wall time re-deriving the same backbone — and it makes the workflow the F12 spec actually asked for impossible:
It also decouples the two phases for Tao: pretrain on one machine/spec, sweep retrofit arms elsewhere.
What to build
--ckpt-every Kon gtx (envGTX_CKPT_EVERY): write the BACKBONE weights — the span toy#161 already tracks as@gx_backbone_count, i.e. everything up to and including the pretrain head — as a GGUF into the run dir, at step multiples of K and at the final step. Reuse thetnn_gguf_w_*surface franken-moe already uses (toy#131); no new shim.--load-ckpt PATHon gtx (envGTX_LOAD_CKPT): validate shape metadata against the live instrument and fail loud listing both sides on any mismatch, then overwrite each backbone tensor BY NAME. A backbone loaded under a different--d-model/--heads/--features/--typeswould otherwise produce confident garbage.--load-ckptimplies the pretrain phase is SKIPPED — the point is not to redo it. Require--retrofitalongside it and reject the combination otherwise, rather than inventing a second meaning for a loaded checkpoint on this lane.Gate legs
backbone: sig_pre=printed by a load equals thesig_post=the writing run printed — every weight came back bit-exact, not approximately.--types/--d-modelrejected by an instrument built at another, naming both sides.--load-ckptwithout--retrofitrejected; a missing file rejected;--ckpt-everywith no run dir rejected (nowhere to write is not a silent no-op).What this does NOT change
Not the toy#161 result, and not its one-process default: same-process remains the way the bar is measured, because bit-identical-by-construction is a stronger guarantee than a round-trip that itself has to be gated. This only removes the cost of re-pretraining for sweeps that do not need that guarantee.
Refs
toy#161 (the retrofit lane + the
backbone_siginstrument this leans on), toy#131 (--ckpt-every/--load-ckpton franken-moe — same GGUF surface, same fail-loud-on-mismatch discipline), toy#160 (the byte fixture that guards the inert path).