This repository packages a depth video to RGB video pipeline on top of the VACE Wan 2.1 model family.
The main entry point is batch_run.py, which fans out large datasets of depth-driven editing jobs, manages prompt sampling, and optionally reorganises finished results for downstream consumption.

• Python 3.10+
• CUDA-capable GPU (the default configuration targets 14B Wan checkpoints; expect ≥48 GB VRAM)
• PyTorch 2.5.1 with CUDA 12.4 wheels (see requirements.txt for the full dependency list)

Install the environment:

pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt
pip install wan@git+https://github.com/Wan-Video/Wan2.1

If you need the preprocessing annotators used by auxiliary scripts, also run pip install -r requirements/annotator.txt.

batch_run.py loads the Wan 2.1 VACE 14B checkpoint through vace.vace_wan_modified.get_wan_model.
Place the unpacked model weights at:

ckpts/Wan2.1-VACE-14B/

You can obtain the files from:

• Hugging Face: https://huggingface.co/Wan-AI/Wan2.1-VACE-14B
• ModelScope: https://www.modelscope.cn/models/Wan-AI/Wan2.1-VACE-14B

The script expects the directory to contain the original Wan checkpoint layout (config.json, diffusion weights, text encoder weights, etc.).

batch_run.py traverses --input_root for object folders.
Each object folder must contain:

• Depth renderings: depth_jointXX_videoYYY.mp4 (or colour videos color_jointXX_videoYYY.mp4 when --use_color_direct is enabled)
• Optional masks: mask_jointXX_videoYYY.png (same stem as the depth clip)
• Prompts file: a tab-separated prompts.txt

Sample layout:

inputs/
  data_1105/
    obj0001/
      depth_joint00_video000.mp4
      mask_joint00_video000.png
      depth_joint01_video000.mp4
      mask_joint01_video000.png
      prompts.txt

Example prompts.txt line (tab separated, empty columns are allowed):

0\tjoint_0\tOriginal prompt text\tSimplified prompt text

• Column 0: integer joint id
• Column 2: full prompt
• Column 3: optional simplified prompt (used when --simplified is passed)

The script draws up to --max_prompts_per_video prompts per joint using a deterministic RNG seeded by (object_id, joint_id, video_idx) so replicas in separate batches stay aligned.

Basic invocation:

python batch_run.py \
  --input_root inputs/data_1105 \
  --output_root results/data_1105_batch0 \
  --batch_id 0 \
  --batch_n 8 \
  --max_prompts_per_video 2 \
  --simplified

• --batch_n and --batch_id split the workload across multiple processes or machines. batch_id ∈ [0, batch_n).
• Outputs (without --reorg) are grouped under results/data_1105_batch0/obj<id>/obj<id>_joint<jj>_prompt<ppp>_video<vvv>/ with out_video.mp4, the first frame snapshot, the prompt text file, and copied masks.

batch_launch.sh showcases how we submit the same command to a cluster scheduler (rjob) while cycling through batches.

• Depth-to-RGB generation: Default mode loads Wan VACE 14B once per run and generates RGB clips for each sampled prompt.
• Colour direct mode (--use_color_direct): Bypasses the Wan model entirely, copying pre-rendered RGB clips into the output structure while exporting masks and thumbnails. Use this for the ablation scenario where you want a dataset derived purely from the original renders (no VACE-powered data augmentation).
• Prompt handling:
  • --max_prompts_per_video clamps the number of prompts sampled per depth clip.
  • --simplified switches from column 2 (full prompt) to column 3 (simplified prompt) in prompts.txt.
• Batch partitioning: --batch_n/--batch_id chunk the task list so you can distribute work manually or via batch_launch.sh.
• Metadata dry run (--dry_run_meta): Scans existing outputs instead of generating them. With --reorg, it emits a CSV mapping each completed video to its prompt, frame, and mask.
  • --num_processes can be set to parallelise the metadata sweep (defaults to the CPU count).
  • --out_meta selects the CSV filename (default metadata.csv).
• Reorganisation mode (--reorg): Writes finished artefacts to a flat directory (<output_root>/obj*_joint*_prompt*_video*.mp4) and accumulates metadata rows; useful before uploading to object storage.
  • When enabled, the script initialises CephS3Storage using the environment variables HOSS_KEY, HOSS_SECRET, and ENDPOINT_URL. Ensure these are present if you plan to immediately sync to Ceph/S3.
• Resource hygiene: Each completed job clears cached GPU memory via torch.cuda.empty_cache() so long runs stay within a single process.

Standard layout (--reorg disabled)

results/<run_name>/obj0001/
  obj0001_joint00_prompt000_video000/
    out_video.mp4
    frame0.png
    mask.png
    prompt.txt

Reorganised layout (--reorg enabled)

results/<run_name>/
  obj0001_joint00_prompt000_video000.mp4
  obj0001_joint00_prompt000_video000_frame0.png
  obj0001_joint00_prompt000_video000_mask.png
  metadata.csv

The metadata CSV contains columns video, prompt, input_image, and ref_mask, making it easy to ingest results downstream or upload to storage services.

• Confirm that every depth (or colour) clip has a matching mask file; the script derives the mask path by replacing "depth"/"color" with "mask" in the filename.
• If model loading fails, double-check that ckpts/Wan2.1-VACE-14B/ exists and that the wan package can see GPU resources.
• Large batches benefit from setting CUDA_VISIBLE_DEVICES manually and ensuring the environment is launched from the correct conda environment (see base_command.sh for an example activation sequence).

For additional utilities (dataset preparation, metadata alignment, etc.) explore:

• find_depth_video.py – helper for locating the exact depth clip that produced a generated RGB video
• merge_data.py and split_videos.py – dataset management scripts

Feel free to adapt batch_run.py for other VACE tasks by adjusting the input discovery logic and the prompt sampling strategy.