RoboSPA

🌐 Project Page · 📄 Paper · 💻 Code · 🤗 Dataset · ✨ ModelScope

Zhenxuan Fan¹, Bo Zhang², Yutong Lin¹, Yuqian Yuan¹, Juekai Lin¹, Liang Liang¹, Zhuoyi Huang³, Wenqiao Zhang^1,*, Juncheng Li^1,*, Siliang Tang¹, Jun Xiao¹, Yueting Zhuang¹

_{¹ Zhejiang University    ² University of Electronic Science and Technology of China    ³ South China Normal University}

_{* Corresponding authors}

RoboSPA (Robot Spatial-Procedural Assessment) is a large-scale robotic manipulation dataset and benchmark for diagnosing embodied reasoning in Vision-Language-Action (VLA) models. It is built on the RoboTwin 2.0 simulation and data-generation framework (SAPIEN + CuRobo).

RoboSPA focuses on two capability dimensions:

• 🔍 Fine-Grained Spatial Reasoning — identify the intended target among visually similar candidates
• 🧭 Long-Horizon Procedural Planning — complete multi-step procedures under ordering, repetition, and memory constraints

• 📚 Overview
• 🐣 News
• 🛠️ Installation
• 🚀 Quick Start
• ⚙️ Task Configurations
• 📦 Data Collection
• 💽 Released Dataset
• 🤖 Other Policy Baselines
• 🧩 Deploy Your Own Policy
• 📋 Task List
• 👍 Citation
• 📬 Contact
• 😺 Acknowledgement
• 🏷️ License

Existing VLA benchmarks mostly score binary task completion in clean, short-horizon settings. RoboSPA instead scales spatial ambiguity and procedural horizon in a controlled way, while keeping task semantics fixed.

Each of the 56 base tasks has five variants (_1 … _5):

• Spatial tasks add more plausible candidates
• Procedural tasks add more subgoals / longer sequences

Beyond Success Rate (SR), RoboSPA reports:

• ONTA (Object-Normalized Target Accuracy) for spatial tasks — chance-corrected target selection
• Progress Score (PS) for procedural tasks — fraction of completed subgoals

The paper evaluates RDT, GO-1, π0.5, and X-VLA in a single-task setting on clean-scene Aloha-AgileX data. At L5, every evaluated model stays below 25% overall success.

This repository reuses RoboTwin 2.0's expert-script data pipeline: task environments live in envs/, collection is driven by collect_data.sh, and policy evaluation goes through script/eval_policy.py.

• 📄 2026/09/04 — Paper on arXiv: arXiv:2609.05324
• 🎉 2026/08 — Accepted at EMNLP 2026 (Main Conference)

Please follow the RoboTwin 2.0 installation guide to set up the environment and download assets.

Train and evaluate π0.5 on Pick_Blocks_Size_1 (demo_clean). Put training HDF5 under data/ from the released dataset. π0.5 env setup (uv, CuRobo): RoboTwin π0.5 guide.

conda activate RoboTwin
cd policy/pi05
mkdir -p processed_data training_data

bash process_data_pi05.sh Pick_Blocks_Size_1 demo_clean 50
mkdir -p training_data/demo_clean
cp -r processed_data/Pick_Blocks_Size_1-demo_clean-50 training_data/demo_clean/
bash generate.sh ./training_data/demo_clean/ demo_clean_repo

Set repo_id="demo_clean_repo" on pi05_aloha_full_base in src/openpi/training/config.py, then:

uv run scripts/compute_norm_stats.py --config-name pi05_aloha_full_base
bash finetune.sh pi05_aloha_full_base demo_clean 0
#                train_config          model_name  gpu

Checkpoints: policy/pi05/checkpoints/pi05_aloha_full_base/demo_clean/. Default eval checkpoint_id is 30000 in deploy_policy.yml.

bash eval.sh Pick_Blocks_Size_1 demo_clean pi05_aloha_full_base demo_clean 0 0
#            task_name          eval_cfg  train_config         model_name seed gpu

100 rollouts, unseen instructions. Results go to eval_result/.

Train clean, test randomized:

bash eval.sh Pick_Blocks_Size_1 demo_randomized pi05_aloha_full_base demo_clean 0 0

YAML files live in task_config/. Create a new one with:

bash task_config/create_task_config.sh my_config

Shipped configs:

Important fields (defaults match demo_clean.yml):

Supported embodiment names: aloha-agilex, ARX-X5, piper, franka-panda, ur5-wsg.

Camera names are in task_config/_camera_config.yml (D435, L515, and larger variants). Domain-randomization keys match RoboTwin 2.0 configuration docs. Paths in this repo are task_config/ (not env_cfg/task_config/).

Example: collect on Franka instead of Aloha-AgileX:

embodiment: [franka-panda]

For each task variant you care about:

bash collect_data.sh ${task_name} ${task_config} ${gpu_id}

Examples:

# Spatial L1, clean Aloha-AgileX (default embodiment in demo_clean.yml)
bash collect_data.sh Pick_Blocks_Size_1 demo_clean 0

# Same task, domain-randomized scenes
bash collect_data.sh Pick_Blocks_Size_1 demo_randomized 0

# Procedural L5
bash collect_data.sh Place_Phone_Press_Stapler_5 demo_clean 0

In the paper:

• Expert scripts in envs/ are executed with CuRobo motion planning
• Each variant uses 50 successful trajectories
• For each base task, a policy is trained on all five levels (250 demos)
• Instructions: 60 templates per variant (50 train / 10 held-out), instantiated into 100 train / 100 eval utterances
• Clean vs randomized scenes share the same success conditions; randomization only changes visuals and clutter

task_name must match an envs/<task_name>.py class of the same name, e.g. Pick_Blocks_Size_1.

Collection is resumable. Existing seed.txt and data/episode*.hdf5 are skipped. If a run dies, run the same command again.

Templates already live in description/task_instruction/<task_name>.json. After HDF5 exists, you can regenerate episode-level instructions with:

cd description
bash gen_episode_instructions.sh Pick_Blocks_Size_1 demo_clean 100
cd ..

Pre-collected trajectories (527K, 5 embodiments, clean + randomized) are released at:

• Hugging Face: https://huggingface.co/datasets/zxfan/RoboSPA
• ModelScope: https://modelscope.cn/datasets/fffffzx/RoboSPA

Use the released dataset if you only need training data. Collect locally when you need a new embodiment, camera, or randomization setting.

These stacks are inherited from RoboTwin 2.0. π0.5 train / eval is in Quick Start. For other policies, follow that folder’s README, then run its eval.sh. Results land in eval_result/.

X-VLA training and evaluation follow the RMBench X-VLA README. Replace RMBench paths and task names with RoboSPA equivalents (Pick_Blocks_Size_1, etc.).

Paper eval setting: clean scene, Aloha-AgileX, 100 trials per variant, unseen instructions.

Official RoboTwin policy docs: https://robotwin-platform.github.io/doc/usage/index.html. Replace original RoboTwin task names (e.g. beat_block_hammer) with RoboSPA names (e.g. Hit_Blocks_Hammer_Order_1).

Copy policy/Your_Policy/ and implement three files:

1. deploy_policy.py — get_model, eval (call TASK_ENV.take_action), reset_model
2. deploy_policy.yml — checkpoint paths and instruction_type
3. eval.sh — CLI overrides passed into script/eval_policy.py

Action interfaces:

TASK_ENV.take_action(action, action_type="qpos")      # joints + grippers
TASK_ENV.take_action(action, action_type="ee")         # xyz + quaternion + gripper
TASK_ENV.take_action(action, action_type="delta_ee")

qpos layout: [left_arm_joints, left_gripper, right_arm_joints, right_gripper].

cd policy/Your_Policy
bash eval.sh Pick_Blocks_Size_1 demo_clean <ckpt_setting> 0 0

More detail: RoboTwin — Deploy Your Policy.

task_name = {BaseTask}_{level} with level ∈ {1,2,3,4,5}. Example: Pick_Blocks_Size_1, Place_Phone_Press_Stapler_5.

Environment files: envs/<task_name>.py. Instruction templates: description/task_instruction/<task_name>.json. Per-task eval step limits: task_config/_eval_step_limit.yml.

If you use RoboSPA, please cite:

@inproceedings{robospa2026,
  title     = {RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?},
  author    = {Zhenxuan Fan and Bo Zhang and Yutong Lin and Yuqian Yuan and Juekai Lin
               and Liang Liang and Zhuoyi Huang and Wenqiao Zhang and Juncheng Li
               and Siliang Tang and Jun Xiao and Yueting Zhuang},
  booktitle = {Proceedings of EMNLP},
  year      = {2026}
}

Questions or suggestions: zxfan@zju.edu.cn.

Simulation, expert data collection, embodiment assets, and the policy-eval interface are based on RoboTwin 2.0. Physics and rendering use SAPIEN. Motion planning uses CuRobo.

This repository is released under the MIT license. See LICENSE.