[capstone P1b] Latent-diffusion byte-LM: does a small-latent diffusion LM generate coherent text, vs an AR baseline? (all BP, no DFA yet) #166

Consuming Tao experiment: P1b (specs/P1b-latent-diffusion-generation.md). The generation-quality ceiling of the diffusion-text-LM capstone (docs/CAPSTONE-DESIGN.md). All BP — no DFA in this lane (DFA is P1c). This is the biggest lane yet; it composes existing machinery.

Why

P1a (toy#165) established that a small per-token latent (operating point ~d=8, pending the converged re-measure) can carry a byte decodably under noise. P1b asks the next question, on which the whole capstone hinges:

Can a diffusion model generate coherent text through that small latent — beating a prior-decode floor and approaching a plain autoregressive byte-LM of matched size?

If it can't, the cost/memory edge is moot and the path is capped by generation quality, not by DFA (which never enters here). So we front-load it, still all BP.

What to build: toy train difflm (two-stage latent diffusion)

Composes toy#165 (ae encoder + decode head) and toy#156 (the diff denoiser):

1. Stage 1 — the autoencoder, but LIGHTLY REGULARISED so the d-latent is diffusable: a small KL/smoothness term (VAE-style), OR rely on self-conditioning to compensate. Encoder (contextual attention) → per-position d-latent → per-position BP decode head (byte). Freeze after stage 1.
2. Stage 2 — the diffusion denoiser over the per-position latents: attention across positions (so generated latents are coherent as a sequence), self-conditioning ON (the literature's low-dim lever — Analog Bits/SED). Trained to denoise the frozen encoder's latents.
3. Generate: sample latents from noise → denoise → decode via the frozen head → bytes.

toy train difflm --text data/ae_shakespeare --latent 8 --arm {ar-baseline|diff-selfcond|diff-plain|prior-floor} --steps 20000 --seed S --out $DIR

• Corpus: shakespeare (N~55, English — coherence is judgeable, unlike multilingual udhr). --latent defaults to 8 (P1a's operating point) but must be a flag — P1a's converged re-measure may pin d=4 or confirm d=8, and we may run both.

Arms

• ar-baseline — a small autoregressive byte-LM, matched size, same corpus. The yardstick / ceiling; its own samples define what "good" looks like on the metrics.
• diff-selfcond — the candidate: d-latent diffusion LM with self-conditioning.
• diff-plain — self-conditioning OFF (the ablation: how much does it buy at low d?).
• prior-floor — decode random-prior latents (no denoising). The MANDATORY control that must lose (tao#19) — incoherent bytes. If it doesn't lose, the metric can't discriminate.

Metric (matched scale, so "competitive" is measurable)

Report for every arm's samples:

• (a) reference-AR byte-NLL / bits-per-byte — score each arm's generated samples under the ar-baseline model. Fluent samples get low NLL; garbage gets high. (The AR baseline's own samples set the "good" reference; the prior-floor sets "bad".)
• (b) byte n-gram JS divergence (3- or 4-gram) between generated and held-out real text — distribution match, and the anti-degeneracy guard (reference-NLL alone rewards repetitive text).
• (c) qualitative sample dumps in the bundle.

"Somewhat competitive" (Ori's bar) = diff-selfcond's (NLL, JS) sits closer to the ar-baseline's than to the prior-floor's. Trailing AR is expected and fine (the literature's latent text-diffusion trails AR — LD4LG MAUVE .72 vs .79); near the floor is the no-go.

Discipline (carry the program's rules)

• CPU-only, all BP (DFA is P1c — do NOT wire --policy/DFA here).
• control-can-lose: prior-floor must lose decisively; gate it as a precondition (a metric where random-latent-decode scores near the AR baseline measures nothing — the F18/F9e trap on a generative lane).
• decode head + AR head stay BP (256-way = output-dim-law-hostile). P1c will attach DFA to the denoiser (output = the latent d), so keep the denoiser a separable module from the decode head.
• train/val disjoint spans (as toy#165 already does) — a memorising encoder must not inflate the reference-NLL.
• the P1a-margin cross-check: if samples degrade to the floor at the operating d, note whether that matches a too-thin P1a converged margin — the representation-caps-generation link.

What this does NOT do

No DFA (P1c), no cost/memory measurement yet (P1c, and it's analytic per tao#21). No Unicode/factorized-decode-head (tao#22 deferred — shakespeare is single-byte ASCII, so the sequence-length bill doesn't bite here). It answers ONE thing: can a small-latent diffusion LM generate coherent text, and how far below an AR baseline?

Refs

• Tao P1b spec; P1a/toy#165 (the operating latent + converged noise-margin = the sampler-residual budget); F20/toy#156 (the diffusion denoiser); docs/CAPSTONE-DESIGN.md (phase plan + literature: LD4LG 2212.09462 trails AR, self-conditioning Analog Bits 2208.04202 / SED 2211.04236); tao#19 (prior-floor control), tao#22 (deferred seq-length / factorized head).

Note: --latent should default to 8 but stay a flag — P1a's convergence re-measure (running now) may adjust the operating d before this lane runs its cells.