FACT: Failure-Aware Causal Training
for World-Action Models

FACT is a causal world-action model: one causal diffusion transformer jointly denoises robot actions, a task-progress value, and future video, all conditioned on the executed action.

• Act, then imagine. Future video and value condition on the clean action, never the reverse — future prediction sharpens actions without leaking targets, and deployment decodes actions without generating video.
• Failures teach consequences. Failure rollouts skip the action-imitation loss but still supervise the observed failed future and a lowered value.
• Optional best-of-N scoring. The value head ranks sampled action candidates at inference.

This repository is the official implementation, containing the end-to-end RoboTwin pipeline: data prep → training → inference → closed-loop simulator evaluation.

bash setup_env.sh        # conda env `fact`, pinned deps, Wan2.2 + RoboTwin download
conda activate fact

Already have the checkpoint or dataset: SKIP_MODEL_DOWNLOAD=1 SKIP_DATA_DOWNLOAD=1 bash setup_env.sh

Equivalent of what setup_env.sh does:

# Wan2.2 base model
huggingface-cli download Wan-AI/Wan2.2-TI2V-5B-Diffusers \
  --local-dir ./models/Wan2.2-TI2V-5B-Diffusers

# RoboTwin demonstrations
huggingface-cli download Bariona/robotwin-v2 robotwin-v2.tar \
  --repo-type dataset --local-dir ./datasets
tar -xf ./datasets/robotwin-v2.tar -C ./datasets   # -> datasets/RoboTwin/{Clean,Randomized}/<task>/

Trained FACT checkpoint (Bariona/fact-wam):

huggingface-cli download Bariona/fact-wam --local-dir ./models/fact-wam

1. Norm stats + per-episode T5 embedding caches (required; subset via --dataset_glob 'Clean/*'):

python -m scripts.prepare_robotwin \
  --robotwin_root ./datasets/RoboTwin \
  --wan_model_path ./models/Wan2.2-TI2V-5B-Diffusers \
  --output_dir ./artifacts/robotwin

2. VAE latent cache — training reads it by default; export FACT_USE_CACHED_VAE_LATENTS=0 to train from raw video instead:

python -m scripts.compute_vae_latents --batch_size 32   # match training BATCH_SIZE_PER_GPU

3. Dataloader check (optional):

python -m scripts.test_dataloader --config world_action_model.configs.robotwin \
  --num_workers 0 --batch_size 2 --num_batches 2

Edit the USER SETTINGS block at the top of world_action_model/configs/robotwin.py, then:

python -m scripts.train --config world_action_model.configs.robotwin.config

Serve a trained checkpoint (to use the released one instead, pass --transformer_path ./models/fact-wam/transformer --stats_path ./models/fact-wam/norm_stats_delta.json):

python -m scripts.inference_server \
  --model_id ./models/Wan2.2-TI2V-5B-Diffusers \
  --transformer_path ./experiments/robotwin/models/<checkpoint>/transformer \
  --stats_path ./artifacts/robotwin/norm_stats_delta.json \
  --port 8093

Useful flags:

Set up the simulator in its own conda env following RoboTwin, then apply git -C <robotwin> apply <fact>/evaluation/robotwin/robotwin_test_num.patch so TEST_NUM takes effect (upstream hardcodes 100 episodes).

# settings live in evaluation/robotwin/launch_config.yml; point ROBOTWIN_PATH at your checkout
bash evaluation/robotwin/launch_server.sh                                # terminal 1
bash evaluation/robotwin/launch_client.sh beat_block_hammer demo_clean   # terminal 2
bash evaluation/robotwin/eval_all_tasks.sh demo_clean 50                 # ... or sweep all 50 tasks

Per-episode results land in RoboTwin's eval_result/; the sweep also writes per-task and average success rates to ./eval_runs/<config>_<timestamp>/.

If you find FACT useful, please consider citing:

@article{peng2026fact,
  title   = {FACT: Failure-Aware Causal Training for World-Action Models},
  author  = {Peng, Quanquan and Liang, Yutong and Yan, Rui and Hansen, Nicklas and Wang, Xiaolong},
  journal = {arXiv preprint},
  year    = {2026}
}

This project is released under the Apache 2.0 license.

FACT builds on Wan2.2, giga-world-policy, GigaTrain, and GigaDatasets; simulation experiments use the RoboTwin benchmark. Thanks to the authors for open-sourcing their work.