Consuming Tao experiment: P1a (specs/P1a-latent-reconstruction-ceiling.md). The make-or-break, DFA-independent first step of the diffusion-text-LM capstone (CAPSTONE-DESIGN.md). All BP — no DFA, no diffusion in this lane.
Why
The capstone needs a per-token continuous latent of 4–8 dims (F20's window where DFA beats BP) to carry text. A literature search (2026-08-12) found this exact regime — continuous, learned, per-token, <16 dims, open text — is UNRUN: the smallest reported is Diffusion-LM's d=16 (narrow domain only), open text sits at 32–64, and BitLM's 18-bit result is "preliminary." So before we spend anything on diffusion or DFA, one cheap question decides whether the whole path is alive:
Can a d-dim per-token latent be decoded back to its token UNDER THE NOISE a diffusion sampler leaves — and where does that collapse as d shrinks {4,8,16,32}?
If a 4–8-dim latent has no usable noise-margin, no downstream diffusion (however good) or DFA (however cheap) can rescue it. This is a plain BP autoencoder — the cheapest possible form of the test.
What to build: toy train ae
toy train ae --vocab byte --latent-dim {4|8|16|32} --context 256 \
--steps 4000 --seed S --noise-eval 0,0.25,0.5,1.0,2.0 --out $DIR
- Architecture: token embed → contextual encoder (reuse the toy#160/gtx attention body) → per-position d-dim bottleneck (the latent) → PER-POSITION decode head (latent_i → token_i logits, no cross-position context in the decoder — so reconstruction genuinely measures what the latent carries, not what context predicts) → token. Train BP reconstruction cross-entropy.
- Vocab: byte-level (256) — small vocab is a deliberate lever (fewer codepoints → looser packing → larger noise-margin at fixed d, and keeps the decode head small). A
--vocab bpe arm later would be a nice second axis.
--latent-dim: the bottleneck width, the swept axis {4,8,16,32}; add a roomy 64 as the ~no-bottleneck reference ceiling.
THE METRIC — noise-margin, not clean reconstruction (please read)
Clean reconstruction accuracy is VACUOUS at low d and must NOT be the headline. Packing 256 byte-codepoints into 4 continuous dims is trivial analog capacity — a d=4 autoencoder will reconstruct near-perfectly clean, telling us nothing. The decisive question is robustness under latent noise, because a diffusion model at generation time produces a latent estimate with residual error, and the decode head must still recover the token from it.
So emit, per run:
- clean reconstruction accuracy (sanity only).
- noise-margin curve: reconstruction accuracy when the latent is perturbed by Gaussian noise at each
--noise-eval SNR (noise scaled relative to the per-dim latent std, so it's scale-invariant across d). This is the read Tao keys on.
- Ideally the half-accuracy SNR (the σ at which accuracy crosses 50% of clean) as a single scalar per cell — the "denoisable margin."
Intuition to preserve: lower d → tighter codepoint packing → smaller noise-margin. If d=4–8 has a margin comparable to d=16–32, the path is alive; if the margin collapses above d=8, it's capped by representation.
Discipline (carry the program's rules)
- CONTROL-CAN-LOSE (tao#19-shaped): a shuffled / zeroed latent decode must fall to the unigram floor — proves the decode reads the latent, not a positional/frequency prior. Gate this as a precondition; if a zeroed-latent decode scores well, the lane measures nothing.
- Per-position decode only — no context leak into the decoder, or reconstruction becomes next-token prediction and the latent is bypassed.
- CPU-only, small (this is a tiny autoencoder). No
--policy/DFA and no diffusion here — those are P1b (add the denoiser) and P1c (DFA + cost), separate tickets.
Gate legs (suggested)
- precondition: zeroed-latent (or shuffled) reconstruction ≈ unigram floor (≪ the d=32 clean accuracy) — the control can lose.
- clean reconstruction near-ceiling at d=32 (the lane can learn at all).
- noise-margin monotone in d (d=32 ≥ d=16 ≥ d=8 ≥ d=4 at every SNR) — the expected physics; a violation means a bug.
- determinism: two identical runs byte-identical.
What this does NOT do
No diffusion (P1b), no DFA (P1c). It isolates ONE thing: whether a small per-token latent can carry a token decodably under noise. A clean cheap answer to an open question either way.
Refs
- Tao P1a spec; CAPSTONE-DESIGN.md (the literature landscape + the sharpened DFA-≤8-vs-open-text-≥32 tension).
- Literature: Diffusion-LM 2205.14217 (d=16 narrow / 128 open), SED 2211.04236 & TextLDM 2605.07748 (32–64, "lower helps"), BitLM 2605.11577 (18 bits, preliminary), image LDM 4ch/pos × 4096 (CompVis kl-f8).
- F20/toy#156 (why the latent must be ≤8 for DFA), F22/toy#160 (the attention body reused as the encoder), tao#19 (control-can-lose).
Consuming Tao experiment: P1a (
specs/P1a-latent-reconstruction-ceiling.md). The make-or-break, DFA-independent first step of the diffusion-text-LM capstone (CAPSTONE-DESIGN.md). All BP — no DFA, no diffusion in this lane.Why
The capstone needs a per-token continuous latent of 4–8 dims (F20's window where DFA beats BP) to carry text. A literature search (2026-08-12) found this exact regime — continuous, learned, per-token, <16 dims, open text — is UNRUN: the smallest reported is Diffusion-LM's d=16 (narrow domain only), open text sits at 32–64, and BitLM's 18-bit result is "preliminary." So before we spend anything on diffusion or DFA, one cheap question decides whether the whole path is alive:
If a 4–8-dim latent has no usable noise-margin, no downstream diffusion (however good) or DFA (however cheap) can rescue it. This is a plain BP autoencoder — the cheapest possible form of the test.
What to build:
toy train ae--vocab bpearm later would be a nice second axis.--latent-dim: the bottleneck width, the swept axis {4,8,16,32}; add a roomy 64 as the ~no-bottleneck reference ceiling.THE METRIC — noise-margin, not clean reconstruction (please read)
Clean reconstruction accuracy is VACUOUS at low d and must NOT be the headline. Packing 256 byte-codepoints into 4 continuous dims is trivial analog capacity — a d=4 autoencoder will reconstruct near-perfectly clean, telling us nothing. The decisive question is robustness under latent noise, because a diffusion model at generation time produces a latent estimate with residual error, and the decode head must still recover the token from it.
So emit, per run:
--noise-evalSNR (noise scaled relative to the per-dim latent std, so it's scale-invariant across d). This is the read Tao keys on.Intuition to preserve: lower d → tighter codepoint packing → smaller noise-margin. If d=4–8 has a margin comparable to d=16–32, the path is alive; if the margin collapses above d=8, it's capped by representation.
Discipline (carry the program's rules)
--policy/DFA and no diffusion here — those are P1b (add the denoiser) and P1c (DFA + cost), separate tickets.Gate legs (suggested)
What this does NOT do
No diffusion (P1b), no DFA (P1c). It isolates ONE thing: whether a small per-token latent can carry a token decodably under noise. A clean cheap answer to an open question either way.
Refs