TTOM: Test-Time Optimization and Memorization for Compositional Video Generation

TTOM: Test-Time Optimization and Memorization for Compositional Video Generation

Leigang Qu^1*, Ziyang Wang^1*, Na Zheng¹, Wenjie Wang², Liqiang Nie³, Tat-Seng Chua^1
1NExT++ Lab, National University of Singapore    ²University of Science and Technology of China    ³Harbin Institute of Technology (Shenzhen)
^*Equal Contribution

📑 arXiv   |   🌐 Project Page   |   🖥️ GitHub

[ICLR 2026] Official repository

• 2026-01: 🎉 TTOM has been accepted to ICLR 2026!
• 2025-12: 🔥 Released inference code.

TTOM is a training-free, test-time optimization and memorization framework for compositional video generation. It addresses the challenge of generating videos with multiple objects, attributes, and motions that faithfully follow complex text prompts — without any additional training or fine-tuning.

The framework operates in two phases:

1. Meta Extraction & Layout Generation – Uses GPT-4o to extract object metadata and generate spatial-temporal layouts from text prompts.
2. Video Generation with TTOM – Generates videos conditioned on extracted metadata and layouts, with iterative test-time optimization of cross-attention via LoRA.

Built on top of DiffSynth-Studio, an efficient diffusion inference engine.

Evaluation on T2V-CompBench across 7 compositional categories. Bold = best, underline = second best.

💡 TTOM is training-free — it optimizes at inference time on top of frozen pre-trained models without any additional training or fine-tuning.

TTOM/
├── generation/
│   ├── gen_cache.py              # Phase 1: Meta extraction and layout generation
│   ├── gen_benchmarks.py         # Phase 2: Video generation with TTOM
│   └── get_attnmap.py            # Attention map generation
├── utils/
│   ├── gdino_detection_video.py  # GroundingDINO object detection + SAM2 segmentation
│   ├── evaluate_miou.py          # mIoU evaluation between attention and masks
│   ├── visualize_attn_maps.py    # Attention map visualization
│   └── ...                       # Other utility functions
├── ttom/                         # Core TTOM implementation
└── scripts/                      # Batch processing scripts

• Python 3.10+
• CUDA-capable GPU (recommended ≥ 24 GB VRAM for Wan2.1-T2V-14B)
• pip and a virtualenv/conda environment

💡 Note: GroundingDINO and SAM2 are only required for attention-layout overlap evaluation. For basic video generation, skip them.

git clone https://github.com/LgQu/TTOM.git
cd TTOM
pip install -r requirements.txt
pip install -e .

pip install "huggingface_hub[cli]"
huggingface-cli download Wan-AI/Wan2.1-T2V-14B --local-dir ./models/Wan2.1-T2V-14B

💡 The download can be large. The model will be saved to ./models/Wan2.1-T2V-14B/.

git clone https://github.com/IDEA-Research/GroundingDINO.git
cd GroundingDINO && pip install -e . && cd ..

git clone https://github.com/facebookresearch/segment-anything-2.git sam2
cd sam2 && pip install -e . && cd ..

Set up your OpenAI API key for GPT-4o prompt processing:

export OPENAI_API_KEY="your-api-key-here"

On Windows PowerShell:

$env:OPENAI_API_KEY = "your-api-key-here"

python generation/gen_cache.py \
    --benchmark_source t2vcompbench \
    --benchmark_type 1_consistent_attr \
    --start_idx 0 \
    --end_idx 199 \
    --skip_if_exists

Outputs:

• cache/{benchmark_source}_{benchmark_type}-gpt_4o.json – enriched prompts, object metadata, and layouts.
• data/layout/boxes_{cache_name}/layout_{pid}.gif – layout visualization GIFs.

python generation/gen_benchmarks.py \
    --pid 0 \
    --cache_type t2vcompbench_motion_binding_gpt-4o \
    --guidance_type lora \
    --target_layers [3] \
    --max_iter 8 \
    --max_guidance_step 5 \
    --max_lora_step 5 \
    --target_modules "cross_attn.q,cross_attn.k,cross_attn.v,cross_attn.o" \
    --jsd_loss_weight 1.0 \
    --min_loss_value 0.06 \
    --save_lora_weight False \
    --skip_existed_prompt False \
    --prefix "test" \
    --strat_id 0

Outputs:

• data/benchmarks/{cache_type}/.../{pid}_{tag}.mp4 – generated videos.

Uses GPT-4o to enrich prompts, extract object instances/attributes, and generate spatial-temporal layouts:

python generation/gen_cache.py \
    --benchmark_source t2vcompbench \
    --benchmark_type 1_consistent_attr \
    --start_idx 0 \
    --end_idx 199 \
    --skip_if_exists

gen_benchmarks.py generates videos using Wan2.1 with TTOM-style test-time optimization and memorization:

python generation/gen_benchmarks.py \
    --pid 0 \
    --cache_type t2vcompbench_motion_binding_gpt-4o \
    --guidance_type lora \
    --target_layers [3] \
    --max_iter 8 \
    --max_guidance_step 5 \
    --max_lora_step 5 \
    --target_modules "cross_attn.q,cross_attn.k,cross_attn.v,cross_attn.o" \
    --jsd_loss_weight 1.0 \
    --com_loss_weight 0.0 \
    --min_loss_value 0.03 \
    --save_lora_weight False \
    --save_mask False \
    --skip_existed_prompt False \
    --prefix "" \
    --strat_id 0

bash scripts/run_attnmap_batch.sh

python utils/gdino_detection_video.py

python utils/evaluate_miou.py \
    --attn_dir data/attn_maps/wan21_lora/ \
    --dino_dir data/dino_results_batch/ \
    --output_dir data/miou_summary/

python utils/visualize_attn_maps.py \
    --pid 0 \
    --step_id 40 \
    --layer_id 3 \
    --inst_id 0 \
    --save

data/
├── attn_maps/            # Attention map files (.pt)
├── dino_results_batch/   # GroundingDINO + SAM2 detection/segmentation results
├── miou_summary/         # mIoU evaluation results and summary stats
├── benchmarks/           # Generated videos
└── layout/               # Layout visualizations (GIFs)

We thank the authors and maintainers of the following projects:

• Wan2.1 – Open and advanced large-scale video generative models.
• DiffSynth-Studio – Efficient diffusion model inference engine.
• GroundingDINO – Open-set object detection with language grounding.
• SAM2 – Segment Anything 2 for high-quality segmentation.

If you find TTOM useful, please consider giving this repository a star ⭐ and citing our paper:

@article{qu2025ttom,
  title   = {TTOM: Test-Time Optimization and Memorization for Compositional Video Generation},
  author  = {Leigang Qu and Ziyang Wang and Na Zheng and Wenjie Wang and Liqiang Nie and Tat-Seng Chua},
  journal = {arXiv preprint arXiv:2510.07940},
  year    = {2025},
  url     = {https://arxiv.org/abs/2510.07940}
}