StarVLA-WBC： A Generalizable Codebase for Humanoid Whole-Body Control

StarVLA-WBC is a focused StarVLA codebase for humanoid whole-body control. It connects visual and language observations to long-horizon, high-dimensional actions while keeping robot-specific control, safety, and real-time execution in the external WBC stack.

The first release concentrates on four complementary paths:

The repository intentionally removes unrelated manipulation benchmarks, non-humanoid robot examples, and third-party VLM families from the broader StarVLA platform.

camera + instruction + robot state
                 │
                 ▼
         StarVLA-WBC policy
       Qwen VLM / WM4A backbone
                 │
                 ▼
    humanoid joint-reference chunk
                 │
                 ▼
 external whole-body controller
 balance · constraints · tracking · safety
                 │
                 ▼
          humanoid robot / simulator

StarVLA-WBC provides the learned policy and protocol adapters. The external WBC system remains responsible for balance, contact constraints, collision handling, actuator limits, and the real-time control loop.

VLA and WM4A models should predict where the humanoid joints and floating base should move, without embedding one controller's recurrent observation layout or deployment runtime into the model-facing API. StarVLA-WBC therefore separates the stack into four contracts:

1. an action codec decodes a model action chunk;
2. HumanoidCommand carries named joint positions/velocities and the associated floating-base motion;
3. a controller adapter translates that command into the exact native input of a controller runtime;
4. a ControllerBackend returns a controller-neutral HumanoidJointTarget to the simulator or robot bridge.

The controller runtime remains responsible for proprioceptive observations, history buffers, policy inference, interpolation, PD targets, balance, and safety. This boundary lets the VLA/WAM and controller evolve independently and prevents low-level timing details from leaking into training datasets.

The initial interface includes four controller families:

Two action layouts are retained. simple_psi0 preserves existing 36D checkpoints for AMO, SONIC, HOMIE, and the zero-training BFM-Zero encoder route. New controller-neutral training uses g1_joint_reference: 43 named G1 body/hand joints followed by planar velocity, yaw rate, base height, and target yaw (48D total). BFM-Zero's latent remains an internal controller representation produced by its official backward encoder; the compatibility latent-bank route continues to require the complete 48D layout.

Each result uses 10 episodes per task and level. Our rows use each task's native maximum episode length and success criterion, its canonical language instruction, one left-head image, the 32D state, and the original 36D action layout. The upstream Psi0 row is transcribed from the SIMPLE official benchmark; the other official baselines and per-task results are listed in the SIMPLE integration guide. The official rows are contextual published references, not a local reproduction under a proven-identical max-step/success-threshold profile.

The full-policy rows use SIMPLE's native backend assignment: SONIC for eight Teleop tasks and AMO for two MP tasks. The QwenPi XMovePick result uses the canonical-language rerun (10/10, 10/10, 9/10).

Both local checkpoints were trained with the canonical LayerwiseFM Action DiT and interleave_self_attention: true. QwenPi has 36 blocks (18 layer-wise cross-attention, 18 self-attention) and forwards an encoder attention mask. WanPi has 30 blocks (15 cross, 15 self), and its Wan conditioning path does not provide a language/vision mask.

The following controller-backend study freezes the same QwenPi-100k checkpoint, model-facing contract, task reset, canonical language, task-native episode cap and success criterion, and 50 Hz VLA call rate. Each replacement backend was evaluated on all 10 tasks, all three levels, and the 10 official serialized instances per task/level (300 episodes per backend). All 90 task-level units completed with exit code zero and no FAILED marker.

The native row is not a pure-SONIC ten-task result: it uses SONIC for the eight Teleop tasks and AMO for the two MP tasks. Replacement-backend results measure how the frozen policy's learned 36D command distribution aligns with each adapter/runtime; they are not controller-only rankings.

See the detailed task tables and controller alignment audit.

The public names use the StarVLA product family. Historical names remain registered so existing YAML files and checkpoints continue to load.

The Qwen loaders retain Qwen2.5-VL, Qwen3-VL, and Qwen3.5 checkpoint compatibility.

WM4A is also retained for StarVLA's own world-model-for-action research path. Its primary names are StarVLA-WM-Cosmos-{OFT,GR00T,PI} and StarVLA-WM-Wan-{OFT,GR00T,PI}; the original CosmoPredict2* and Wan* registry names remain compatibility aliases.

Python 3.10 or newer and a CUDA-enabled PyTorch environment are required.

git clone https://github.com/starVLA/starVLA-WBC.git
cd starVLA-WBC
python -m pip install -r requirements.txt
python -m pip install -e .

Download or link a supported Qwen checkpoint under playground/Pretrained_models/, or override framework.qwenvl.base_vlm with a Hugging Face model ID or local path.

Prepare the ten-task SIMPLE training data:

export SIMPLE_TASKS="G1WholebodyXMovePickTeleop-v0,G1WholebodyBendPickMP-v0,G1WholebodyHandoverTeleop-v0,G1WholebodyLocomotionPickBetweenTablesTeleop-v0,G1WholebodyTabletopGraspMP-v0,G1WholebodyXMoveBendPickTeleop-v0,G1WholebodyCloseDoorTeleop-v0,G1WholebodyOpenOvenTeleop-v0,G1WholebodyOpenFaucetTeleop-v0,G1WholebodyPickAndPlaceAndHugContainerTeleop-v0"
bash examples/simBenchmarks/SIMPLE/train_files/download_simple_data.sh playground/Datasets

Train the default StarVLA-PI policy:

data_mix=simple_psi0_all_ten_tasks \
NUM_PROCESSES=8 \
bash examples/simBenchmarks/SIMPLE/train_files/run_simple_train.sh

Use Framework_name=StarVLA-OFT with the OFT YAML when a direct MLP baseline is preferred. See the SIMPLE guide for the simulation evaluation contract.

The retained RoboCasa path uses 29-dimensional actions, 58-dimensional state, and a 16-step horizon:

bash examples/simBenchmarks/Robocasa_tabletop/train_files/run_robocasa.sh

Dataset preparation and evaluator setup are documented in the RoboCasa tabletop guide.

Convert an EgoVerse source into LeRobot v2.1 without modifying the source dataset:

python examples/modelExtensions/human2robots/train_files/convert_egoverse_eva.py \
  --source-root /path/to/egoverse/eva \
  --output-root playground/Datasets/Human2Robots/EgoVerseEVA \
  --profile eva_command

Launch the homogeneous 14D EVA command configuration:

DATA_ROOT=playground/Datasets/Human2Robots \
NUM_PROCESSES=8 \
bash examples/modelExtensions/human2robots/train_files/run_egoverse_eva_train.sh

The human observed-motion configuration uses homogeneous 36D hand-motion targets. Robot command data and human pseudo-actions remain separate training mixtures so their semantics are not silently conflated.

The Unitree example is organized as an explicit workflow:

1. collect demonstrations through an external teleoperation/WBC stack;
2. convert the recordings into the registered LeRobot schema;
3. train a StarVLA policy;
4. expose the policy through the WebSocket or GR00T-compatible ZMQ server;
5. let the external controller validate and execute the returned action chunk.

Start with the Unitree G1 whole-body guide. Never connect an unvalidated checkpoint directly to hardware. Verify action ordering, units, normalization, joint limits, emergency stop behavior, and timing in simulation before enabling actuators.

starVLA/                       core dataloader, Qwen/WM4A models, and trainer
deployment/model_server/       WebSocket and GR00T-compatible policy serving
examples/simBenchmarks/        SIMPLE and RoboCasa humanoid simulations
examples/realRobots/           Unitree G1 whole-body integration
examples/modelExtensions/      Human2Robots conversion and training
tests/                         configuration and deployment contract tests

StarVLA-WBC preserves the registry, configuration override, dataset discovery, checkpoint, and policy-server contracts of StarVLA. Use the main repository when you need its broader benchmark collection, non-Qwen VLM integrations, or general-purpose robot examples.

The repository is released under the MIT License. Components derived from third-party projects retain their original headers and license terms.