RoboWits: Unexpected Challenges for Robotic Creative Problem Solving

arXiv 2026

Chunru Lin^*, Hongxin Zhang^*, Fenghao Yu, Zhehuan Chen, Thomas L. Griffiths, Yejin Choi, David Held, Chuang Gan

RoboWits, a bi-manual robotic benchmark designed to systematically evaluate cognitive reasoning, creative tool use, and robustness to unexpected conditions.

• [2026-06-04] We release the Benchmark code, along with the dataset consisting of ~50 demonstrations on 24 seed tasks for fine-tuning!
• [2026-05-30] RoboWits is on arXiv! Check out our project website for videos.

Install uv if you haven't already.

uv sync
source .venv/bin/activate

Some assets come from BlenderKit and require an API key to download, which can be found in your profile. Some BlenderKit assets are only available in .blend format — the download script converts them to GLB automatically by invoking Blender as a subprocess.

Make sure Blender is installed and available on your PATH:

# macOS (Homebrew)
brew install --cask blender

# Or download from https://www.blender.org/download/ and add to PATH
echo 'export PATH="/Applications/Blender.app/Contents/MacOS:$PATH"' >> ~/.zshrc && source ~/.zshrc

# Ubuntu (headless) — download the latest tarball from https://www.blender.org/download/
wget https://mirrors.dotsrc.org/blender/release/Blender5.1/blender-5.1.2-linux-x64.tar.xz
tar -xf blender-5.1.2-linux-x64.tar.xz -C /opt
echo 'export PATH="/opt/blender-5.1.2-linux-x64:$PATH"' >> ~/.bashrc && source ~/.bashrc

# Verify
blender --version

bash assets/setup_assets.sh --api-key <BLENDERKIT_API_KEY>

Note: Some assets require a BlenderKit full-plan subscription. If your API key is a free-tier key, downloads for paid assets will be skipped automatically and those tasks will be unavailable.

The directory should look like:

assets/
  hf_assets/
    work_table.glb
    marvin_bimanual/
      ...
    worktable_texture/
      grained black plastic_Normal.jpg
      grained black plastic_Roughness.jpg
    ...
  blender_kit/
    <asset-id>/
      obj.glb
    ...

python scripts/robowits/examples/run_env.py 

import gs_gym

# List all tasks
print(gs_gym.list_tasks())

# List tasks by benchmark
print(gs_gym.list_tasks(benchmark="robowits"))

# List all benchmarks
print(gs_gym.list_benchmarks())

Configure via observation_mode parameter:

Configure via control_mode parameter:

Environment output:

• "agent_pos": (n_envs, D) numpy array — robot state for the policy; D depends on observation_mode
• "agent_pos_joint": (n_envs, 16) numpy array — JOINT state, always present regardless of observation_mode
• "agent_pos_ee": (n_envs, 14) numpy array — EE state, always present regardless of observation_mode
• "pixels": dict of (n_envs, H, W, 3) numpy uint8 arrays — camera images
  • "ego": Static ego-view camera
  • "wrist_right": Right wrist camera
  • "wrist_left": Left wrist camera

After preprocess_observation() + add_envs_task() (LeRobot format):

• "observation.state": torch.Tensor (n_envs, D) — policy input, mirrors agent_pos
• "observation.agent_pos_joint": torch.Tensor (n_envs, 16) — always recorded
• "observation.agent_pos_ee": torch.Tensor (n_envs, 14) — always recorded
• "observation.images.ego": torch.Tensor (n_envs, 3, H, W) — channel-first, normalized [0, 1]
• "observation.images.wrist_right": torch.Tensor (n_envs, 3, H, W)
• "observation.images.wrist_left": torch.Tensor (n_envs, 3, H, W)
• "task": list[str] (n_envs,) — natural language task description, added by lerobot-eval

RoboWits integrates with LeRobot via a local patched version in third_party/lerobot/. All scripts use lerobot-train / lerobot-eval and accept configuration via environment variables.

A dataset containing ~50 human demonstrations for ~24 robowits seed tasks is available on HuggingFace at XHRlyb2001/RoboWits_lerobot_dataset. Here are some example training script with the dataset.

# ACT (env vars: HF_DATASET, OUTPUT_DIR, NUM_PROCESSES, STEPS, SAVE_FREQ, VAL_FREQ)
bash scripts/robowits/train/train_act.sh

# Pi0
bash scripts/robowits/train/train_pi0.sh

# Pi0.5
bash scripts/robowits/train/train_pi05.sh

All training scripts use accelerate launch --multi_gpu with W&B logging enabled by default. Checkpoints are saved to checkpoints/<policy>_robowits/ by default.

# Evaluate on robowits-10 seed tasks (env vars: CHECKPOINT_PATH, TASK_IDS, CONFIG, OBSERVATION_MODE, N_EPISODES)
CHECKPOINT_PATH=/path/to/checkpoint TASK_IDS="01 02 03 04 06 09 13 16 17 25" bash scripts/robowits/eval/eval.sh

# Evaluate on mutation tasks
CHECKPOINT_PATH=/path/to/checkpoint TASK_IDS="01 02 03 04 06 09 13 16 17 25" bash scripts/robowits/eval/eval_mutation.sh

RoboWits-10 is a subset of 10 representative tasks for standardized evaluation. Each policy is trained jointly on all tasks with 50 demonstrations per task.

Format: success rate (%), progress score [0,1]. Seed/Mut = evaluated on seed episodes / mutation variants.

Robowits is built upon amazing open-source projects:

• Genesis World Provides the universal physics engine.
• LeRobot Provides training and inference infrastructures.

If you find our work useful, please consider citing:

@article{lin2026robowits,
  title={RoboWits: Unexpected Challenges for Robotic Creative Problem Solving},
  author={Lin, Chunru and Zhang, Hongxin and Yu, Fenghao and Chen, Zhehuan and Griffiths, Thomas L and Choi, Yejin and Held, David and Gan, Chuang},
  journal={arXiv preprint arXiv:2605.30326},
  year={2026}
}