This repository adds Explorative Modeling (XM) — best-of-K noise exploration — to Representation Autoencoders (RAE). For every training sample we draw K noise candidates, score each with the per-sample flow-matching loss, and train only on the lowest-loss candidate; K=1 recovers standard flow matching exactly. This codebase is a fork of the official RAE repository; everything not described below behaves as upstream.
We present Representation Autoencoders (RAE), a class of autoencoders that utilize pretrained, frozen representation encoders such as DINOv2 and SigLIP2 as encoders with trained ViT decoders. RAE can be used in a two-stage training pipeline for high-fidelity image synthesis, where a Stage 2 diffusion model is trained on the latent space of a pretrained RAE to generate images.
This repository contains:
- A PyTorch implementation of RAE and pretrained weights.
- A PyTorch implementation of LightningDiT, DiTDH and pretrained weights.
- Training and sampling scripts for the two-stage RAE+DiT pipeline.
- XM best-of-K training (
src/stage2/xm.py, thexm:config block). - Precomputed-latent training (
src/precompute_latents.py) and FDr-6 evaluation (src/eval/fdr6.py,scripts/eval_fdr.py).
- Create environment and install via
uv:conda create -n rae python=3.10 -y conda activate rae pip install uv # Install PyTorch 2.8.0 with CUDA 12.9 # or your own cuda version uv pip install torch==2.8.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu129 # Install other dependencies uv pip install -r requirements.txt
Note (FDr-6 only): FDr-6 evaluation needs a newer
timm(uv pip install "timm>=1.0.27"). The reference statistics and Fréchet-distance code from FD-loss (MIT) are vendored underexternal/FD-loss, so a plaingit cloneincludes everything.
We release three kind of models: RAE decoders, DiTDH diffusion transformers and stats for latent normalization. To download all models at once:
cd XRAE
pip install huggingface_hub
hf download nyu-visionx/RAE-collections \
--local-dir models To download specific models, run:
hf download nyu-visionx/RAE-collections \
<remote_model_path> \
--local-dir models - Download ImageNet-1k.
- Point Stage 1 and Stage 2 scripts to the training split via
--data-path.
Encoding ImageNet once removes the frozen-encoder forward from every Stage 2 training step:
torchrun --standalone --nproc_per_node=N \
src/precompute_latents.py \
--config configs/stage2/training/ImageNet256/DiTDH-XL_DINOv2-B-xm.yaml \
--output-dir data/latents/rae_dinov2b_imagenet256 \
--split train --hflip-mode bothThen set runtime.latent_cache_dir in the training config (or pass --latent-cache-dir).
--hflip-mode both stores original + flipped views so horizontal-flip augmentation happens
in latent space. Without a cache, training encodes online via --data-path as upstream.
All training and sampling entrypoints are driven by OmegaConf YAML files. A single config describes the Stage 1 autoencoder, the Stage 2 diffusion model, and the solver used during training or inference. A minimal example looks like:
stage_1:
target: stage1.RAE
params: { ... }
ckpt: <path_to_ckpt>
stage_2:
target: stage2.models.DDT.DiTwDDTHead
params: { ... }
ckpt: <path_to_ckpt>
transport:
params:
path_type: Linear
prediction: velocity
...
sampler:
mode: ODE
params:
num_steps: 50
...
guidance:
method: cfg/autoguidance
scale: 1.0
...
misc:
latent_size: [768, 16, 16]
num_classes: 1000
xm:
best_of_k: 2 # K noise candidates per sample; 1 = standard flow matching
training:
...
eval:
...stage_1instantiates the frozen encoder and trainable decoder. For Stage 1 training you can point to an existing checkpoint viastage_1.ckptor start frompretrained_decoder_path.stage_2defines the diffusion transformer. During sampling you must provideckpt; during training you typically omit it so weights initialise randomly.transport,sampler, andguidanceselect the forward/backward SDE/ODE integrator and optional classifier-free or autoguidance schedule.misccollects shapes, class counts, and scaling constants used by both stages.xmconfigures best-of-K exploration (best_of_k,chunk_bs_mult,save_mem_mode); omit it (or setbest_of_k: 1) for standard flow matching.trainingcontains defaults that the training scripts consume (epochs, learning rate, EMA decay, gradient accumulation, etc.).evalcontains settings for online evaluation during training (fdr_eval: trueadds FDr-6 alongside gFID).
Stage 1 training configs additionally include a top-level gan block that
configures the discriminator architecture and the LPIPS/GAN loss schedule.
We release decoders for DINOv2-B, SigLIP-B, MAE-B, at configs/stage1/pretrained/.
There is also a training script for training a ViT-XL decoder on DINOv2-B: configs/stage1/training/DINOv2-B_decXL.yaml
Note: In the released training configs, optimization hyperparameters are tuned to adapt to GPU hardware. The updated configs yield slightly better gFID than the reported numbers.
The XM training config (DiTDH-XL, DINOv2-B, XM-2) is
configs/stage2/training/ImageNet256/DiTDH-XL_DINOv2-B-xm.yaml; the corresponding
autoguidance sampling config is configs/stage2/sampling/ImageNet256/DiTDH-XL_XM2_DINOv2-B_AG.yaml.
Upstream baseline configs are provided unchanged at configs/stage2/training/ and
configs/stage2/sampling/.
src/train_stage1.py trains the ViT decoder while keeping the
representation encoder frozen. Launch it with PyTorch DDP (single or multi-GPU):
torchrun --standalone --nproc_per_node=N \
src/train_stage1.py \
--config <config> \
--data-path <imagenet_train_split> \
--results-dir ckpts/stage1 \
--image-size 256 --precision bf16/fp32 where N refers to the number of GPU cards available.
You need to specify the checkpointing folder by export EXPERIMENT_NAME="your_experiment_name" before launching the script. The checkpoints and logs will be saved under results-dir/$EXPERIMENT_NAME/.
Logging. To enable wandb, log in once (wandb login, stores credentials in ~/.netrc),
then add the --wandb flag; --wandb-project / --wandb-entity (or the config's
training.wandb_project) select the destination.
Resuming. If the checkpoint folder already exists (results-dir/$EXPERIMENT_NAME/), the script will automatically resume from the latest checkpoint.
Online Eval. The script supports online evaluation during training to monitor reconstruction quality. Paste the following block into the training config:
eval:
eval_interval: 2500 # Eval interval by optimization step
eval_model: true # By default only evaluates EMA model. Set to true to eval non-EMA model as well.
data_path: 'data/imagenet/val/' # path to ImageNet val images
reference_npz_path: 'data/imagenet/val_256.npz' # packed npz of ImageNet val images for FID calculation
metrics: ['psnr', 'ssim', 'rfid'] # metrics to calculatetorch.compile. Use --compile flag to enable torch.compile for potentially faster training.
Use src/stage1_sample.py to encode/decode a single image:
python src/stage1_sample.py \
--config <config> \
--image assets/pixabay_cat.png \For batched reconstructions and .npz export, run the DDP variant:
torchrun --standalone --nproc_per_node=N \
src/stage1_sample_ddp.py \
--config <config> \
--data-path <imagenet_val_split> \
--sample-dir recon_samples \
--image-size 256The script writes per-image PNGs as well as a packed .npz suitable for FID.
We also provide an individual script pack_images.py to convert a folder of images into a FID-ready .npz:
python src/pack_images.py <image_folder> We use a batchnorm-like normalization layer in the latent space for calculating the mean and variance statistics of the encoder outputs. Specifically, we calculate the mean and variance of the same shape as the latent [C, H, W]. To run:
torchrun --standalone --nproc_per_node=N \
src/calculate_stat.py \
--config <config> \ # w/o pre-computed stats
--sample-dir stats \
--precision fp32/bf16 \
--data-path <imagenet_train_split> \
--image-size 256 \
--tf32We note that the results may differ slightly from the released statistics because we use a momentum-based update rule for the mean and variance. As a result, the outcome is not deterministic and depends on batch size and data shuffling.
As a reproduction, we ran the script to calculate the statistic of DINOv2-B on 4 GPUs with a batch size of 256 under fp32. The config is exactly the same as configs/stage1/pretrained/DINOv2-B.yaml except commenting out the pre-computed stats. The computed variance give around 0.008 L1 difference compared to the released statistics.
src/train.py trains the Stage 2 diffusion transformer using PyTorch DDP. Edit
one of the configs under configs/training/ and launch:
EXPERIMENT_NAME=DiTDH-XL_DINOv2-B_XM2 \
torchrun --standalone --nnodes=1 --nproc_per_node=N \
src/train.py \
--config configs/stage2/training/ImageNet256/DiTDH-XL_DINOv2-B-xm.yaml \
--latent-cache-dir data/latents/rae_dinov2b_imagenet256 # or --data-path <imagenet_train_split>Although bf16 is supported, we recommend using fp32 for more stable training.
The logging and checkpointing behaviour is the same as Stage 1 training. Runtime knobs
(precision, compile, wandb, latent cache) live in the config's runtime: section and are
overridable from the CLI.
Online Eval: Paste the following block into training config to support online eval:
eval:
eval_interval: 25000
eval_model: true
data_path: 'data/imagenet/val/'
reference_npz_path: 'data/imagenet/VIRTUAL_imagenet256_labeled.npz'
fdr_eval: true # optional: online FDr-6 (needs external/FD-loss, see Environment)src/sample.py uses the same config schema to draw a small batch of images on a
single device and saves them to sample.png:
python src/sample.py \
--config <sample_config> \
--seed 42src/sample_ddp.py parallelises sampling across GPUs, producing PNGs and an
FID-ready .npz:
torchrun --standalone --nnodes=1 --nproc_per_node=N \
src/sample_ddp.py \
--config <sample_config> \
--sample-dir samples \
--precision fp32/bf16 \
--label-sampling equal--label-sampling {equal,random}: equal uses exactly 50 images per class for FID-50k; random uniformly samples labels. Using equal brings consistently lower FID than random by around 0.1. We use equal by default. We recommend using fp32 when model FID is low.
Autoguidance and classifier-free guidance are controlled via the config’s guidance block.
Use the ADM evaluation suite to score generated samples:
-
Clone the repo:
git clone https://github.com/openai/guided-diffusion.git cd guided-diffusion/evaluation -
Create an environment and install dependencies:
conda create -n adm-fid python=3.10 conda activate adm-fid pip install 'tensorflow[and-cuda]'==2.19 scipy requests tqdm -
Download ImageNet statistics (256×256 shown here):
wget https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/256/VIRTUAL_imagenet256_labeled.npz
-
Evaluate:
python evaluator.py VIRTUAL_imagenet256_labeled.npz /path/to/samples.npz
python scripts/eval_fdr.py --npz <samples.npz> computes Fréchet distances across six
representation backbones (Inception, ConvNeXt, DINOv2, MAE, SigLIP, CLIP); the required
FD-loss code and reference statistics are vendored under external/FD-loss.
ImageNet 256×256, DiTDH-XL on DINOv2-B latents (Table 1 of the XM paper; RAE baseline reproduced under the same setup). We report FDr-6 as the primary metric — gFID is saturated at this performance level and can misrepresent sample quality:
| Method | FDr-6 ↓ (unguided) |
|---|---|
| RAE: DiTDH-XL | 4.42 |
| XRAE (ours): XDiTDH-XL, XM-2 | 3.91 |
XRAE matches the RAE baseline's best FDr-6 with 6.2× less data and 4.1× fewer training FLOPs. For guided sampling, the shipped AG config uses scale 1.5 with the released DiTDH-S (epoch 14) guide; see the paper for guided results, scaling curves, and ablations.
@misc{gladstone2026explorativemodeling,
title={Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation},
author={Alexi Gladstone and Heng Ji and Yilun Du},
year={2026},
eprint={2607.27372},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2607.27372},
}
@misc{zheng2025diffusiontransformers,
title={Diffusion Transformers with Representation Autoencoders},
author={Boyang Zheng and Nanye Ma and Shengbang Tong and Saining Xie},
year={2025},
eprint={2510.11690},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2510.11690},
}This code is built upon the following repositories:
- RAE - the codebase this repository is forked from.
- SiT - for diffusion implementation and training codebase.
- DDT - for some of the DiTDH implementation.
- LightningDiT - for the PyTorch Lightning based DiT implementation.
- MAE - for the ViT decoder architecture.
- FD-loss - representation backbones for FDr-6.