Seungwook Kim · Seunghyeon Lee · Minsu Cho

POSTECH Computer Vision Lab

[Paper]

FreeAction presents training-free techniques to enhance the fidelity of trajectory-to-video generation for robotics. Built on top of IRASim, our method introduces action-aware noise truncation and action-aware classifier-free guidance to improve video generation quality without requiring model retraining.

Key Finding: Our proposed action-aware noise truncation outperforms existing inference-time techniques for trajectory-to-video generation.

• 🚀 Training-Free: Works with pre-trained IRASim models
• 🎯 Action-Aware Techniques: Adaptive noise truncation and guidance based on action magnitude
• 📊 Improved Fidelity: Enhanced video quality across multiple robotics datasets
• ⚙️ Flexible Configuration: Easy toggle for different techniques

Create a conda environment and install dependencies:

bash scripts/install.sh

This will:

• Install PyTorch with CUDA support
• Install diffusers, transformers, and other required packages
• Set up the irasim conda environment

Requirements:

• Python 3.8+
• CUDA 11.8+ (for GPU support)
• 8GB+ GPU memory recommended

To download the complete dataset (evaluation data + checkpoints):

bash scripts/download.sh

Note: You only need evaluation data and checkpoints for FreeAction (training data not required).

The datasets will be downloaded to opensource_robotdata/ with the following structure:

Total: ~452 GB (evaluation + checkpoints only)

For best fidelity across all datasets, use:

• ❌ CFG: Disabled
• ❌ Action-Aware CFG: Disabled
• ❌ Fixed Truncation: Disabled
• ✅ Action-Aware Noise Truncation: Enabled

Edit sample/components/config.py:

class GuidanceConfig:
    do_classifier_free_guidance: bool = False   # Disabled
    enable_dynamic_guidance: bool = False       # Disabled

class LatentConfig:
    enable_truncated_sampling: bool = False               # Disabled
    enable_action_conditioned_truncation: bool = True     # ✅ ENABLED

Generate videos for short trajectories (16 frames):

# Single GPU
python3 main.py --config configs/evaluation/rt1/frame_ada.yaml

# Multi-GPU (8 GPUs)
torchrun --nproc_per_node 8 main.py --config configs/evaluation/rt1/frame_ada.yaml

Replace rt1 with bridge or languagetable for other datasets.

Generate videos for long trajectories (autoregressive):

# Generate videos for RT-1
python3 evaluate/generate_long_video.py \
    --config configs/evaluation/rt1/frame_ada.yaml \
    --rank 0 --thread 0 --thread-num 1

# For parallel processing with multiple GPUs/threads
python3 evaluate/generate_long_video.py \
    --config configs/evaluation/rt1/frame_ada.yaml \
    --rank 0 --thread 0 --thread-num 8

Calculate metrics (PSNR, SSIM, FVD, FID):

# Short trajectory metrics
python3 evaluate/evaluate_short_script.py

# Long trajectory metrics
python3 evaluate/evaluate_long_script.py

All configurations are in sample/components/config.py. Here are common setups:

class GuidanceConfig:
    do_classifier_free_guidance: bool = False
    enable_dynamic_guidance: bool = False

class LatentConfig:
    enable_truncated_sampling: bool = False
    enable_action_conditioned_truncation: bool = False

class GuidanceConfig:
    do_classifier_free_guidance: bool = False
    enable_dynamic_guidance: bool = False

class LatentConfig:
    enable_truncated_sampling: bool = False
    enable_action_conditioned_truncation: bool = True  # ✅

class GuidanceConfig:
    do_classifier_free_guidance: bool = True   # ✅
    enable_dynamic_guidance: bool = False

class LatentConfig:
    enable_truncated_sampling: bool = False
    enable_action_conditioned_truncation: bool = False

Also set guidance_scale in configs/base/diffusion.yaml:

guidance_scale: 4.5  # Must be > 1.0 for CFG to activate

class GuidanceConfig:
    do_classifier_free_guidance: bool = True
    enable_dynamic_guidance: bool = True        # ✅ Action-aware

class LatentConfig:
    enable_truncated_sampling: bool = False
    enable_action_conditioned_truncation: bool = False

If you find this work useful, please consider citing:

@article{kim2025freeaction,
  title={FreeAction: Training-Free Techniques for Enhanced Fidelity of Trajectory-to-Video Generation},
  author={Kim, Seungwook and Lee, Seunghyeon and Cho, Minsu},
  journal={arXiv preprint arXiv:2509.24241},
  year={2025}
}

This work builds upon:

• IRASim - Base trajectory-to-video generation framework
• Latte - Transformer-based video generation architecture
• RT-1, Bridge, Language-Table - Robotics datasets

We thank the authors for open-sourcing their code and datasets.