Hierarchical VAE with split head attention. First N heads: bidirectional (encoder) - Second H-N heads: causal (decoder) - Each layer is a VAE with binary latents - ladder prior

First draft of rotor / cryptographic inspired discrete normalising flow. Turing would be proud (enigma machine)

Juicing an imagenet vae. Wip

Initial explorations for lossless representation learning.

I am suprised this works. It's pretty cool.

A clean example of a conditional flow posterior and a prior flow based vae as well as prior based iterative refinement.

Action regularised ar-affline flows. Learns a regularised sde base distribution for a coupling base normalizing flow.

More experiments with temporal latents.

Better sde flows.

Exact path-likelihood latent sequence model with conditional observation flow and Ornstein–Uhlenbeck process prior. SDE mechanics without needing to solve the sde.

Good base for NLL for SDEs in latent space.

Experiments in discrete path flows for learning time conditioned functions.

Pool the 'shape' of the input, approximates a kernel mean. A hypernet learns 'directions' to observe the inputs.

CatFlow (Variational Flow Matching) from https://arxiv.org/abs/2406.04843

A plug‑and‑play layer for an ordinal integer.

Output head for full range bfloat prediction.

Stable robust loss varient

Just an idea. Entirely untested.

A novel (?) generative model abusing dimensionality collapse to play an adverserial game.

Minimal Implimentation of VCRec (2024) for collapse provention.

Untested, use at your own risk, might help with trainability and plasticity. Will run some experiments later.

A simple masking strategy that out performs random and block masking, and some semantic masking for training masking based self supervised models.

RankUp is a PyTorch extension that nudges gradient updates away from directions that would lower the rank of your model’s parameters or features. It can be useful if you need to preserve full-rank …

Rank Regularisation for Rank Collapse Prevention in Self Distillation Methods

Neat.

Off the dome quick jot, wrote it while waiting for an eval plot. Unverified and slow, probably buggy. Don't use.

It's a hyperspherical thingy for discrete diffusionishness.

Computes the Energy Sliced Wasserstein Loss between two distributions. An optimal-transporty-energyish vibe distribution matching loss/regulariser.

A simple implimentation of Bayesian Flow Networks (BFN)