XPolicyLab

English | 简体中文

A Unified Standard and Open Ecosystem for Robot Policy Evaluation and Deployment

Website | arXiv | GitHub | RoboDojo Leaderboard | RoboTwin Leaderboard

Connecting N policies to M evaluation environments — from O(N×M) down to O(N+M).

XPolicyLab is the shared layer between policy code and evaluation environments. Keep each model's dependencies, checkpoints, and training recipes under policy/<POLICY>/; XPolicyLab handles the parts that are boring but easy to get wrong — serving, observation/action contracts, and eval wiring. As of September 2026, the ecosystem integrates 45 robot policies spanning VLA, world-action, imitation-learning, and memory-augmented families, and the same adapters serve RoboTwin, RoboDojo simulation, and standardized real-robot evaluation.

Start here for repo-level concepts and integration steps. For install commands, checkpoint layout, and training details, jump to that policy's README — it is the source of truth for its model.

• What XPolicyLab Enables
• Supported Benchmarks And Infrastructure
• Integrated Policies
• Framework Overview
• Quick Start
• Common Workflow
• Deployment Flow
• Standard Data Formats
  • Decode only through decode_image_bit
  • Official LeRobot conversion
• Data And Checkpoints
• Add Your Own Policy
• Coding Agents
• Citation
• Contact

• Environment isolation: run the policy model in its own conda/uv environment while the simulator, benchmark, or robot client runs separately.
• Remote deployment: connect the policy server and environment client through websocket, either on one machine or across machines.
• A common adapter contract: use the same high-level lifecycle for installation, data conversion, training, serving, and evaluation.
• A large policy zoo: reuse adapters for VLA/WAM policies, imitation-learning baselines, and reference templates.
• Benchmark and infra integration: mount XPolicyLab into benchmark or simulator workspaces without coupling policy code to one environment.

XPolicyLab is benchmark-agnostic: any benchmark, simulator, or real-robot setup can plug in as an environment client against the same policy-side interface — one adapter per policy, one client per environment. The benchmarks below are already integrated; the RoboDojo and RoboTwin official leaderboards are powered by XPolicyLab submissions.

Cross-platform evaluation through a shared codebase and standardized serving interface.

Benchmarks

• RoboDojo: simulator-backed evaluation and RoboDojo-format data exports. The RoboDojo Leaderboard covers 42 simulation tasks across five capability dimensions (Generalization, Precision, Long-Horizon, Memory, Open) plus 18 real-robot tasks on three bimanual embodiments.
• RoboTwin: benchmark and data source through policy-specific adapters and conversion scripts. The RoboTwin 2.0 Leaderboard covers bimanual manipulation across 50 tasks under clean and randomized settings.
• RMBench: memory-dependent manipulation benchmark built on RoboTwin 2.0, with nine dual-arm tasks spanning levels of task memory complexity (paper, website). Its reference policy is integrated as Mem-0.

Infrastructure

• RLinf (coming soon): infrastructure target for policy development and deployment workflows.
• StarVLA: infrastructure and policy stack; see policy/starVLA.

45 policies are currently integrated, spanning VLA, world-action, imitation-learning, and memory-augmented families, plus demo_policy as the minimal reference adapter. Top-level adapters live in policy/; each policy README documents that model's paper/repo link, environment, data format, training entrypoint, and checkpoint layout.

Adding a policy of your own, or entering a leaderboard, both go through a PR — see Add Your Own Policy.

XPolicyLab separates model-side dependencies from environment-side dependencies, so each side retains its native stack and may run locally or remotely. One adapter serves benchmarks, simulators, and physical robots.

Infrastructure of XPolicyLab. One adapter serves benchmarks, simulators, and physical robots.

Policy environment                         Evaluation / benchmark environment
------------------                         ----------------------------------
policy/<POLICY>/model.py     <---ws--->    env client / simulator / robot
policy server                              environment client
deploy.yml deployment config               benchmark task and observation API

A typical adapter contains:

policy/<POLICY>/
├── README.md                    # policy-specific guide
├── INSTALLATION.md              # optional detailed setup notes
├── __init__.py                  # keeps XPolicyLab.policy.<POLICY> importable
├── install.sh                   # environment setup
├── process_data.sh              # optional data conversion
├── train.sh                     # optional training
├── eval.sh                      # same-machine evaluation
├── setup_eval_policy_server.sh  # policy-side server
├── setup_eval_env_client.sh     # environment-side client
├── deploy.yml                   # deployment config
├── deploy.py                    # deployment loop
└── model.py                     # model adapter, served by the policy server

model.py implements the model-facing API. deploy.py bridges environment observations to model-server calls. Use policy/demo_policy as the minimal adapter reference.

model.py should define a Model class with this shape:

The policy server decodes camera colors before update_obs / update_obs_batch, so obs["vision"][<camera>]["color"] always arrives as an image array — model.py never decodes.

The default policy-server protocol is websocket (protocol: ws in deploy.yml); legacy_tcp exists only for adapters that have not migrated yet. The transport handles reconnects, retries, keepalive, and long model-loading cold starts for you — a normal adapter never touches it.

Clone XPolicyLab as a normal Python project for adapter development, offline checks, training from prepared data, or your own environment client:

mkdir demo_env
cd demo_env
git clone https://github.com/XPolicyLab/XPolicyLab.git
cd XPolicyLab
pip install -e .

You do not need a simulator to start model-side development: the bundled downloader fetches prepared RoboDojo data — several simulator export versions plus HDF5 RoboDojo_real real-world data — for training and offline debugging. If you use XPolicyLab/ as a subpackage inside the RoboDojo repository, follow RoboDojo's own data download scripts instead.

Download a small Hugging Face demo bundle and keep the data next to XPolicyLab/:

# From demo_env/XPolicyLab
bash scripts/RoboDojo/download_robodojo_data.sh demo

This creates:

demo_env/
├── data/        # demo data, including a small 10-episode HuggingFace bundle
└── XPolicyLab/

The same script pulls the full exports — hdf5, lerobot_v3.0, lerobot_v2.1, and real (real-world HDF5) — each into its own ../data/ folder. The two LeRobot exports come from the official converters, so you can regenerate them for your own task subset or resolution.

With this setup, you can test data conversion, model loading, training scripts, and debug-mode evaluation before connecting to a simulator-backed benchmark.

export EVAL_ENV_TYPE=debug
cd policy/demo_policy
bash install.sh
bash eval.sh RoboDojo stack_bowls demo arx_x5 joint 0 0 0 base base

The template for any adapter is the same — swap demo_policy and the argument values:

export EVAL_ENV_TYPE=debug
cd policy/<POLICY>
bash eval.sh <bench_name> <task_name> <ckpt_name> <env_cfg_type> <action_type> \
  <seed> <policy_gpu_id> <env_gpu_id> <policy_env_or_uv_path> <eval_env_conda_env>

For RoboDojo simulation, mount XPolicyLab/ beside the simulator-side env_cfg/, scripts/, src/eval_client/, and task/ directories.

Most adapters expose the same top-level shape. Some policies add extra arguments, consume upstream-native datasets, or skip training support. Follow the policy README when it differs from this template.

cd policy/<POLICY>

# Install the policy environment.
bash install.sh

# Optional: convert or prepare policy-specific data.
bash process_data.sh <bench_name> <ckpt_name> <env_cfg_type> <action_type> [extra_args...]

# Optional: train.
bash train.sh <bench_name> <ckpt_name> <env_cfg_type> <action_type> <seed> <gpu_id> [extra_args...]

# Evaluate on one machine.
bash eval.sh <bench_name> <task_name> <ckpt_name> <env_cfg_type> <action_type> <seed> \
  <policy_gpu_id> <env_gpu_id> <policy_env_or_uv_path> <eval_env_conda_env>

When you run eval.sh, you are mostly answering: which benchmark family, which task to run now, which checkpoint to load, which robot setup, joint or end-effector actions, and which seed. The same names travel through process_data.sh, train.sh, and eval.sh, so you do not have to rename things at every step.

How ckpt_name resolves. Usually you pass the short nickname used during training, such as cotrain, and XPolicyLab combines it with the other args into checkpoints/RoboDojo-cotrain-arx_x5-joint-0/. You can also pass the full folder name, or a path — relative paths resolve from the policy directory, absolute paths work too. Some adapters honor explicit keys in deploy.yml (checkpoint_path, model_path, ...). When in doubt, check the policy README.

A concrete eval example:

cd policy/AHA_WAM
bash eval.sh RoboDojo stack_bowls cotrain arx_x5 joint 0 0 0 aha_wam robodojo
# loads checkpoints/RoboDojo-cotrain-arx_x5-joint-0/ and evaluates on stack_bowls

During evaluation, the policy server and the environment client talk over websocket. That split is what lets you keep Isaac Sim / robot drivers on one machine and a heavy VLA on another.

For same-machine evaluation, eval.sh is enough — it starts the server, runs the client, and cleans up when you are done.

For split-machine deployment, start the policy server on the GPU machine and bind to 0.0.0.0 so other machines can reach it. The client connects to the policy machine's real IP, not 0.0.0.0.

cd policy/<POLICY>
bash setup_eval_policy_server.sh \
  <bench_name> <task_name> <ckpt_name> <env_cfg_type> <action_type> <seed> \
  <policy_gpu_id> <policy_env_or_uv_path> <policy_server_port> 0.0.0.0

Then start the environment client on the simulator or robot machine:

cd policy/<POLICY>
bash setup_eval_env_client.sh \
  <bench_name> <task_name> <ckpt_name> <env_cfg_type> <action_type> <seed> \
  <env_gpu_id> <eval_env_conda_env> <additional_info> \
  <policy_server_port> <policy_server_ip>

<additional_info> is a comma-separated key=value string forwarded to the environment client. eval.sh builds it automatically as ckpt_name=<ckpt_name>,action_type=<action_type>, which is the right default for most adapters.

EVAL_ENV_TYPE selects the environment-side backend:

• unset or sim: real simulator-backed evaluation, when the integration is installed.
• debug: offline wiring check — no Isaac, no robot, just shapes and IO.
• real: real-robot client path, where the hardware integration exists.

XPolicyLab standardizes the observation and trajectory dictionaries passed between adapters, converters, and environment clients. Individual policies may convert this standard format into their upstream-native format.

Always decode through decode_image_bit, and always encode through encode_image_bit. Both live in XPolicyLab.utils.process_data. Decoding image bits yourself is unsupported, because they come in two byte formats and a hand-rolled decoder is right on one and reverses the channels on the other. decode_image_bit tells them apart and returns RGB for every version of the data, so its output never needs a channel swap. Offline conversion and training must go through it. Runtime observations arrive already decoded, so model.py must not decode at all.

Stored image bits come in two formats, and both decode to RGB:

Legacy data is never migrated — JPEG cannot swap channels losslessly — so the two formats coexist indefinitely and may appear in the same training run. The marker sits inside the buffer rather than in a file attribute so that a single buffer is self-describing, and COM is a standard segment that every decoder skips, so it costs 12 bytes and breaks nothing. If you must read these buffers without OpenCV, you can do it correctly: PIL surfaces the marker as Image.open(...).info["comment"], so check for b"XPL-RGB1" and reverse the channels yourself when it is absent. What you cannot do is skip the check — a PIL-based loader tested against fresh data looks perfect and then quietly corrupts older episodes.

All pose values use [x, y, z, qw, qx, qy, qz]. Images are RGB end to end — decode_image_bit hands back RGB and no channel conversion happens anywhere else in the pipeline. Note one naming quirk: runtime observations carry camera extrinsics as extrinsics_matrix, while trajectory files store extrinsic_matrix.

Observation Data Format
├── data_format_version                        string, optional
├── instruction / instructions                 string or list[str]
├── env_idx                                    int, optional for batched eval
├── additional_info/
│   └── frequency                              int, optional
├── vision/
│   ├── cam_head/
│   │   ├── color                              (H, W, 3) RGB, decoded by the server
│   │   ├── depth                              (H, W) or (H, W, 1), optional
│   │   ├── intrinsic_matrix                   (3, 3), optional
│   │   ├── extrinsics_matrix                  (4, 4), optional
│   │   └── shape                              (2,) or (3,), optional
│   ├── cam_left_wrist/                        optional
│   ├── cam_right_wrist/                       optional
│   ├── cam_wrist/                             optional for single-arm robots
│   └── cam_third_view/                        optional
└── state/
    ├── left_arm_joint_state                   (DOF,), optional
    ├── left_ee_joint_state                    (EEF_DOF,), optional
    ├── left_ee_pose                           (7,), optional
    ├── left_tcp_pose                          (7,), optional
    ├── left_delta_ee_pose                     (7,), optional
    ├── right_arm_joint_state                  (DOF,), optional
    ├── right_ee_joint_state                   (EEF_DOF,), optional
    ├── right_ee_pose                          (7,), optional
    ├── right_tcp_pose                         (7,), optional
    ├── right_delta_ee_pose                    (7,), optional
    ├── arm_joint_state                        (DOF,), optional for single-arm robots
    ├── ee_joint_state                         (EEF_DOF,), optional for single-arm robots
    ├── ee_pose                                (7,), optional for single-arm robots
    ├── tcp_pose                               (7,), optional for single-arm robots
    ├── delta_ee_pose                          (7,), optional for single-arm robots
    └── mobile/                                optional
        ├── base_pose                          (7,)
        └── base_twist                         (6,), [vx, vy, vz, wx, wy, wz]

Trajectory Data Format
├── data_format_version                        string, e.g. "v1.0"
├── instruction / instructions                 string, or JSON-serialized list[str]
├── subtasks                                   JSON-serialized annotations, optional
├── additional_info/
│   └── frequency                              int
├── vision/
│   ├── cam_head/
│   │   ├── colors                             (T, H, W, 3), uint8 RGB or encoded stream
│   │   ├── depths                             (T, H, W) or (T, H, W, 1), optional
│   │   ├── intrinsic_matrix                   (3, 3) or (T, 3, 3), optional
│   │   ├── extrinsic_matrix                   (4, 4) or (T, 4, 4), optional
│   │   └── shape                              (2,) or (3,), optional
│   ├── cam_left_wrist/                        optional
│   ├── cam_right_wrist/                       optional
│   ├── cam_wrist/                             optional for single-arm robots
│   └── cam_third_view/                        optional
├── action/                                    action targets, same key naming as state/ below
└── state/
    ├── left_arm_joint_states                  (T, DOF), optional
    ├── left_ee_joint_states                   (T, EEF_DOF), optional
    ├── left_ee_poses                          (T, 7), optional
    ├── left_tcp_poses                         (T, 7), optional
    ├── left_delta_ee_poses                    (T, 7), optional
    ├── right_arm_joint_states                 (T, DOF), optional
    ├── right_ee_joint_states                  (T, EEF_DOF), optional
    ├── right_ee_poses                         (T, 7), optional
    ├── right_tcp_poses                        (T, 7), optional
    ├── right_delta_ee_poses                   (T, 7), optional
    ├── arm_joint_states                       (T, DOF), optional for single-arm robots
    ├── ee_joint_states                        (T, EEF_DOF), optional for single-arm robots
    ├── ee_poses                               (T, 7), optional for single-arm robots
    ├── tcp_poses                              (T, 7), optional for single-arm robots
    ├── delta_ee_poses                         (T, 7), optional for single-arm robots
    └── mobile/                                optional
        ├── base_poses                         (T, 7)
        └── base_twists                        (T, 6), [vx, vy, vz, wx, wy, wz]

Useful converter helpers:

from XPolicyLab.utils.load_file import load_hdf5
from XPolicyLab.utils.process_data import (
    decode_image_bit,
    encode_image_bit,
    get_robot_action_dim_info,
)

decode_image_bit and encode_image_bit are the only supported codec for trajectory image bits (see above). They mirror each other's input handling — one frame or a sequence, in any of the containers the trajectory files use — and already-converted values pass through untouched. get_robot_action_dim_info(env_cfg_type) returns robot-specific arm_dim and ee_dim lists, so adapters do not need to hard-code action dimensions.

CONTRIBUTING.md states the RGB exceptions and how a new robot gets registered in both _robot_info.json files.

Many policies train on LeRobot datasets instead of the trajectory format above. scripts/transform_lerobot_v21_format.py and scripts/transform_lerobot_v30_format.py are the official converters — one per LeRobot dataset version, both emitting the same keys.

Prepared LeRobot exports published for a benchmark — such as the lerobot_v2.1 / lerobot_v3.0 sets in Quick Start — are produced this way, so a policy that consumes one needs no conversion step of its own.

A policy that trains on LeRobot data must say so in its own README, under Data Processing: the dataset version, and whether the keys are the ones above. If they are, name the converter and say whether process_data.sh is absent or only links and normalizes the dataset. If they are not — an upstream-native layout, extra keys, a latent tree, different camera names — state the differences and how to produce that layout. policy/RISE and policy/AHA_WAM are worked examples of the first case, policy/LingBot_VA of the second.

# One robot, every task under it.
python scripts/transform_lerobot_v30_format.py "<bench_name>.*.<env_cfg_type>" --repo_id my_dataset

# Several robots merged, 50 episodes per task/env, downscaled.
# Without --resolution the target size comes from the first source frame.
python scripts/transform_lerobot_v21_format.py "<bench_name>.*.*" \
  --max_episode 50 --resolution 240x320

Training and data prep usually name things predictably so eval can find them without guesswork:

<bench_name>-<ckpt_name>-<env_cfg_type>-<action_type>
<bench_name>-<ckpt_name>-<env_cfg_type>-<action_type>-<seed>

So if you trained with bench_name=RoboDojo, ckpt_name=cotrain, env_cfg_type=arx_x5, action_type=joint, seed=0, the run lands in checkpoints/RoboDojo-cotrain-arx_x5-joint-0/. How ckpt_name maps back to these folders at eval time is covered in Common Workflow.

Policies may also use upstream-native layouts or explicit paths in deploy.yml. Check the policy README before assuming a naming convention. For a small local dataset to play with, see Quick Start.

Community policies are welcome — open a PR that adds policy/<POLICY>/. A PR is also required to enter the official RoboDojo and RoboTwin leaderboards, together with the checkpoint that reproduces your results.

How to scaffold an adapter, what a submission must contain, the checks to run before a PR, and the PR template all live in CONTRIBUTING.md. Start from policy/demo_policy, or bash scripts/create_policy.sh <POLICY_NAME>.

Two skills under .agents/skills are loaded by Cursor, Claude Code, and Codex (via .cursor/skills and .claude/skills symlinks). AGENTS.md is the always-on rule set.

• xpolicylab-model-integration — build an adapter. A prompt like Integrate <POLICY_NAME> into XPolicyLab is enough.
• xpolicylab-adapter-check — audit one before a PR (Check policy/<POLICY_NAME>).

The paste-in checklist for agents that load none of these is in CONTRIBUTING.md.

If XPolicyLab helps your research, please cite:

@article{community2026xpolicylab,
  title={{XPolicyLab}: A Unified Standard and Open Ecosystem for Robot Policy Evaluation and Deployment},
  author={Community, XPolicyLab and Chen, Tianxing and Chen, Yue and Nian, Tian and Cai, Zijian and Chen, Guangyu and Lin, Wenwei and Liang, Qiwei and Xiang, Peicheng and Su, Kailun and others},
  journal={arXiv preprint arXiv:2608.09892},
  year={2026}
}

Tianxing Chen (project lead): chentianxing2002@gmail.com

A collaborative open-source project led by MMLab@HKU and THU.

Core Lead Authors: Tianxing Chen, Yue Chen, Tian Nian, Zijian Cai, Guangyu Chen, Wenwei Lin, Qiwei Liang.

The full contributor list — spanning every integrated policy — lives on the project website.