Skip to content
alexigladPublic

About

Explorative Modeling + Representation Autoencoders (https://github.com/bytetriper/RAE)

Resources

Stars

4 stars

Watchers

1 watching

Forks

Repository files navigation

Explorative Modeling with Representation Autoencoders (XM-RAE)
Official PyTorch Implementation

This repository adds Explorative Modeling (XM) — best-of-K noise exploration — to Representation Autoencoders (RAE). For every training sample we draw K noise candidates, score each with the per-sample flow-matching loss, and train only on the lowest-loss candidate; K=1 recovers standard flow matching exactly. This codebase is a fork of the official RAE repository; everything not described below behaves as upstream.

We present Representation Autoencoders (RAE), a class of autoencoders that utilize pretrained, frozen representation encoders such as DINOv2 and SigLIP2 as encoders with trained ViT decoders. RAE can be used in a two-stage training pipeline for high-fidelity image synthesis, where a Stage 2 diffusion model is trained on the latent space of a pretrained RAE to generate images.

This repository contains:

  • A PyTorch implementation of RAE and pretrained weights.
  • A PyTorch implementation of LightningDiT, DiTDH and pretrained weights.
  • Training and sampling scripts for the two-stage RAE+DiT pipeline.
  • XM best-of-K training (src/stage2/xm.py, the xm: config block).
  • Precomputed-latent training (src/precompute_latents.py) and FDr-6 evaluation (src/eval/fdr6.py, scripts/eval_fdr.py).

Environment

Dependency Setup

  1. Create environment and install via uv:
    conda create -n rae python=3.10 -y
    conda activate rae
    pip install uv
    
    # Install PyTorch 2.8.0 with CUDA 12.9 # or your own cuda version
    uv pip install torch==2.8.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu129 
    
    # Install other dependencies
    uv pip install -r requirements.txt

Note (FDr-6 only): FDr-6 evaluation needs a newer timm (uv pip install "timm>=1.0.27"). The reference statistics and Fréchet-distance code from FD-loss (MIT) are vendored under external/FD-loss, so a plain git clone includes everything.

Data & Model Preparation

Download Pre-trained Models

We release three kind of models: RAE decoders, DiTDH diffusion transformers and stats for latent normalization. To download all models at once:

cd XRAE
pip install huggingface_hub
hf download nyu-visionx/RAE-collections \
  --local-dir models 

To download specific models, run:

hf download nyu-visionx/RAE-collections \
  <remote_model_path> \
  --local-dir models 

Prepare Dataset

  1. Download ImageNet-1k.
  2. Point Stage 1 and Stage 2 scripts to the training split via --data-path.

Precompute latents (optional, recommended)

Encoding ImageNet once removes the frozen-encoder forward from every Stage 2 training step:

torchrun --standalone --nproc_per_node=N \
  src/precompute_latents.py \
  --config configs/stage2/training/ImageNet256/DiTDH-XL_DINOv2-B-xm.yaml \
  --output-dir data/latents/rae_dinov2b_imagenet256 \
  --split train --hflip-mode both

Then set runtime.latent_cache_dir in the training config (or pass --latent-cache-dir). --hflip-mode both stores original + flipped views so horizontal-flip augmentation happens in latent space. Without a cache, training encodes online via --data-path as upstream.

Config-based Initialization

All training and sampling entrypoints are driven by OmegaConf YAML files. A single config describes the Stage 1 autoencoder, the Stage 2 diffusion model, and the solver used during training or inference. A minimal example looks like:

stage_1:
   target: stage1.RAE
   params: { ... }
   ckpt: <path_to_ckpt>  

stage_2:
   target: stage2.models.DDT.DiTwDDTHead
   params: { ... }
   ckpt: <path_to_ckpt>  

transport:
   params:
      path_type: Linear
      prediction: velocity
      ...
sampler:
   mode: ODE
   params:
      num_steps: 50
      ...
guidance:
   method: cfg/autoguidance
   scale: 1.0
   ...
misc:
   latent_size: [768, 16, 16]
   num_classes: 1000
xm:
   best_of_k: 2   # K noise candidates per sample; 1 = standard flow matching
training:
   ...
eval:
   ...
  • stage_1 instantiates the frozen encoder and trainable decoder. For Stage 1 training you can point to an existing checkpoint via stage_1.ckpt or start from pretrained_decoder_path.
  • stage_2 defines the diffusion transformer. During sampling you must provide ckpt; during training you typically omit it so weights initialise randomly.
  • transport, sampler, and guidance select the forward/backward SDE/ODE integrator and optional classifier-free or autoguidance schedule.
  • misc collects shapes, class counts, and scaling constants used by both stages.
  • xm configures best-of-K exploration (best_of_k, chunk_bs_mult, save_mem_mode); omit it (or set best_of_k: 1) for standard flow matching.
  • training contains defaults that the training scripts consume (epochs, learning rate, EMA decay, gradient accumulation, etc.).
  • eval contains settings for online evaluation during training (fdr_eval: true adds FDr-6 alongside gFID).

Stage 1 training configs additionally include a top-level gan block that configures the discriminator architecture and the LPIPS/GAN loss schedule.

Provided Configs:

Stage1

We release decoders for DINOv2-B, SigLIP-B, MAE-B, at configs/stage1/pretrained/.

There is also a training script for training a ViT-XL decoder on DINOv2-B: configs/stage1/training/DINOv2-B_decXL.yaml

Note: In the released training configs, optimization hyperparameters are tuned to adapt to GPU hardware. The updated configs yield slightly better gFID than the reported numbers.

Stage2

The XM training config (DiTDH-XL, DINOv2-B, XM-2) is configs/stage2/training/ImageNet256/DiTDH-XL_DINOv2-B-xm.yaml; the corresponding autoguidance sampling config is configs/stage2/sampling/ImageNet256/DiTDH-XL_XM2_DINOv2-B_AG.yaml. Upstream baseline configs are provided unchanged at configs/stage2/training/ and configs/stage2/sampling/.

Stage 1: Representation Autoencoder

Train the decoder

src/train_stage1.py trains the ViT decoder while keeping the representation encoder frozen. Launch it with PyTorch DDP (single or multi-GPU):

torchrun --standalone --nproc_per_node=N \
  src/train_stage1.py \
  --config <config> \
  --data-path <imagenet_train_split> \
  --results-dir ckpts/stage1 \
  --image-size 256 --precision bf16/fp32 

where N refers to the number of GPU cards available.

You need to specify the checkpointing folder by export EXPERIMENT_NAME="your_experiment_name" before launching the script. The checkpoints and logs will be saved under results-dir/$EXPERIMENT_NAME/.

Logging. To enable wandb, log in once (wandb login, stores credentials in ~/.netrc), then add the --wandb flag; --wandb-project / --wandb-entity (or the config's training.wandb_project) select the destination.

Resuming. If the checkpoint folder already exists (results-dir/$EXPERIMENT_NAME/), the script will automatically resume from the latest checkpoint.

Online Eval. The script supports online evaluation during training to monitor reconstruction quality. Paste the following block into the training config:

eval:
  eval_interval: 2500 # Eval interval by optimization step
  eval_model: true # By default only evaluates EMA model. Set to true to eval non-EMA model as well.
  data_path: 'data/imagenet/val/' # path to ImageNet val images
  reference_npz_path: 'data/imagenet/val_256.npz' # packed npz of ImageNet val images for FID calculation
  metrics: ['psnr', 'ssim', 'rfid'] # metrics to calculate

torch.compile. Use --compile flag to enable torch.compile for potentially faster training.

Sampling/Reconstruction

Use src/stage1_sample.py to encode/decode a single image:

python src/stage1_sample.py \
  --config <config> \
  --image assets/pixabay_cat.png \

For batched reconstructions and .npz export, run the DDP variant:

torchrun --standalone --nproc_per_node=N \
  src/stage1_sample_ddp.py \
  --config <config> \
  --data-path <imagenet_val_split> \
  --sample-dir recon_samples \
  --image-size 256

The script writes per-image PNGs as well as a packed .npz suitable for FID.

We also provide an individual script pack_images.py to convert a folder of images into a FID-ready .npz:

python src/pack_images.py <image_folder> 

Calculating Encoder Statistics

We use a batchnorm-like normalization layer in the latent space for calculating the mean and variance statistics of the encoder outputs. Specifically, we calculate the mean and variance of the same shape as the latent [C, H, W]. To run:

torchrun --standalone --nproc_per_node=N \
  src/calculate_stat.py \
  --config <config> \ # w/o pre-computed stats
  --sample-dir stats \
  --precision fp32/bf16 \
  --data-path <imagenet_train_split> \
  --image-size 256 \
  --tf32

We note that the results may differ slightly from the released statistics because we use a momentum-based update rule for the mean and variance. As a result, the outcome is not deterministic and depends on batch size and data shuffling.

As a reproduction, we ran the script to calculate the statistic of DINOv2-B on 4 GPUs with a batch size of 256 under fp32. The config is exactly the same as configs/stage1/pretrained/DINOv2-B.yaml except commenting out the pre-computed stats. The computed variance give around 0.008 L1 difference compared to the released statistics.

Stage 2: Latent Diffusion Transformer

Training

src/train.py trains the Stage 2 diffusion transformer using PyTorch DDP. Edit one of the configs under configs/training/ and launch:

EXPERIMENT_NAME=DiTDH-XL_DINOv2-B_XM2 \
torchrun --standalone --nnodes=1 --nproc_per_node=N \
  src/train.py \
  --config configs/stage2/training/ImageNet256/DiTDH-XL_DINOv2-B-xm.yaml \
  --latent-cache-dir data/latents/rae_dinov2b_imagenet256   # or --data-path <imagenet_train_split>

Although bf16 is supported, we recommend using fp32 for more stable training.

The logging and checkpointing behaviour is the same as Stage 1 training. Runtime knobs (precision, compile, wandb, latent cache) live in the config's runtime: section and are overridable from the CLI.

Online Eval: Paste the following block into training config to support online eval:

eval:
  eval_interval: 25000 
  eval_model: true
  data_path: 'data/imagenet/val/'
  reference_npz_path: 'data/imagenet/VIRTUAL_imagenet256_labeled.npz'
  fdr_eval: true   # optional: online FDr-6 (needs external/FD-loss, see Environment)

Sampling

src/sample.py uses the same config schema to draw a small batch of images on a single device and saves them to sample.png:

python src/sample.py \
  --config <sample_config> \
  --seed 42

Distributed sampling for evaluation

src/sample_ddp.py parallelises sampling across GPUs, producing PNGs and an FID-ready .npz:

torchrun --standalone --nnodes=1 --nproc_per_node=N \
  src/sample_ddp.py \
  --config <sample_config> \
  --sample-dir samples \
  --precision fp32/bf16 \
  --label-sampling equal

--label-sampling {equal,random}: equal uses exactly 50 images per class for FID-50k; random uniformly samples labels. Using equal brings consistently lower FID than random by around 0.1. We use equal by default. We recommend using fp32 when model FID is low.

Autoguidance and classifier-free guidance are controlled via the config’s guidance block.

Evaluation

ADM Suite FID setup

Use the ADM evaluation suite to score generated samples:

  1. Clone the repo:

    git clone https://github.com/openai/guided-diffusion.git
    cd guided-diffusion/evaluation
  2. Create an environment and install dependencies:

    conda create -n adm-fid python=3.10
    conda activate adm-fid
    pip install 'tensorflow[and-cuda]'==2.19 scipy requests tqdm
  3. Download ImageNet statistics (256×256 shown here):

    wget https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/256/VIRTUAL_imagenet256_labeled.npz
  4. Evaluate:

    python evaluator.py VIRTUAL_imagenet256_labeled.npz /path/to/samples.npz

FDr-6

python scripts/eval_fdr.py --npz <samples.npz> computes Fréchet distances across six representation backbones (Inception, ConvNeXt, DINOv2, MAE, SigLIP, CLIP); the required FD-loss code and reference statistics are vendored under external/FD-loss.

Results

ImageNet 256×256, DiTDH-XL on DINOv2-B latents (Table 1 of the XM paper; RAE baseline reproduced under the same setup). We report FDr-6 as the primary metric — gFID is saturated at this performance level and can misrepresent sample quality:

Method FDr-6 ↓ (unguided)
RAE: DiTDH-XL 4.42
XRAE (ours): XDiTDH-XL, XM-2 3.91

XRAE matches the RAE baseline's best FDr-6 with 6.2× less data and 4.1× fewer training FLOPs. For guided sampling, the shipped AG config uses scale 1.5 with the released DiTDH-S (epoch 14) guide; see the paper for guided results, scaling curves, and ablations.

Citation

@misc{gladstone2026explorativemodeling,
      title={Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation},
      author={Alexi Gladstone and Heng Ji and Yilun Du},
      year={2026},
      eprint={2607.27372},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2607.27372},
}

@misc{zheng2025diffusiontransformers,
      title={Diffusion Transformers with Representation Autoencoders},
      author={Boyang Zheng and Nanye Ma and Shengbang Tong and Saining Xie},
      year={2025},
      eprint={2510.11690},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2510.11690},
}

Acknowledgement

This code is built upon the following repositories:

  • RAE - the codebase this repository is forked from.
  • SiT - for diffusion implementation and training codebase.
  • DDT - for some of the DiTDH implementation.
  • LightningDiT - for the PyTorch Lightning based DiT implementation.
  • MAE - for the ViT decoder architecture.
  • FD-loss - representation backbones for FDr-6.

About

Explorative Modeling + Representation Autoencoders (https://github.com/bytetriper/RAE)

Resources

Stars

4 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages