[DeMiAn] How to Instruct Your Robot: Dense Language Annotations Power Robot Policy Learning

DeMiAn is instantiated on two policy architectures and two simulators:

Policy architectures:

• DeMiAn-VLA — openpi 0.5 / Pi0.5 backbone. Code: openpi + molmobot_pi0.
• DeMiAn-WAM — Cosmos-Predict 2.5 video DiT + action head. Code: demian_wam (overlay onto external Cosmos-Predict 2.5).

Environments:

• RoboCasa 365: 17 atomic + composite tasks
• MolmoSpaces: bench-v2, 9 benches

(image, task description) ──► Instructor (Qwen3.5-2B) ──► annotation
                                                              │
              task_description + annotation ──► Policy (DeMiAn-VLA / DeMiAn-WAM) ──► actions

demian/                             # repo root
├── README.md                       # ← you are here
├── docker/                         # Dockerfile (VLA env) + Dockerfile.instructor
├── openpi/                         # vendored Pi0.5 PyTorch backbone (no jax model code)
│                                   #   + scripts/eval (RoboCasa harness, serve_policy, instructor server)
├── molmobot_pi0/                   # DeMiAn-VLA policy/dataset code + Hydra eval entrypoint
├── demian_wam/                     # DeMiAn-WAM overlay onto Cosmos-Predict 2.5 (action head, egoverse, eval)
├── config/eval/                    # Hydra eval config
├── runs/{molmobot,robocasa}/       # instructor SFT and per-task annotation routing tables
├── scripts/                        # scripts/slurm
├── docs/                           # GitHub Pages project page (index.md)
└── data/README.md                  # dataset download (HF)

You can build the environments from the provided Dockerfiles:

docker build -f docker/Dockerfile            -t demian-vla:latest .       # VLA train + eval (openpi 0.5 / Pi0.5)
docker build -f docker/Dockerfile.instructor -t demian-instructor:latest . # Qwen3.5-2B instructor server (async eval)

• docker/Dockerfile — torch 2.7.1 + JAX (CPU tree-utils only) + transformers==4.53.2 with the PaliGemma/Gemma patches pre-applied; covers VLA post-training and the policy side of eval.
• docker/Dockerfile.instructor — newer transformers for the qwen3_5 instructor (kept separate from the policy's pinned 4.53.2; the two co-host at eval over HTTP).
• DeMiAn-WAM uses the upstream Cosmos-Predict 2.5 environment (build it from that repo's own Dockerfile), then apply the overlays in demian_wam/ — see demian_wam/README.md.

• openpi — vendored Pi0.5 PyTorch backbone, adapted from Physical Intelligence's openpi (no separate clone needed; PYTHONPATH=openpi/src:openpi/packages/openpi-client/src:., where . = repo root).
• molmobot_pi0 — DeMiAn-VLA policy / dataset / eval code, adapted from AllenAI's MolmoBot (MolmoBot-Pi0).
• Simulators: molmospaces and robocasa + robosuite.
• Cosmos-Predict 2.5 for DeMiAn-WAM.

The dataset and checkpoints are hosted on Hugging Face. See data/README.md for exact commands. In short:

The launcher scripts are listed in each component's README:

• DeMiAn-VLA (train + eval): molmobot_pi0/README.md
• DeMiAn-WAM (train + eval): demian_wam/README.md
• Instructor (train): scripts/instructor/README.md

The annotation-aspect ablations (paper Table 1) are trained with scripts/slurm/train_demian_vla.slurm, which fine-tunes the Pi0.5 PyTorch backbone (openpi, see its README) via molmobot_pi0/train_openpi_pytorch.py. The aspect is passed as an argument and selects the appended-caption type; the policy, visual data, and action targets are held fixed across aspects.

# node count / GPUs-per-node / partition / account are passed at submit time
# Set DATA_ROOT, PRETRAINED_CKPT, CONTAINER via env first.
sbatch --nodes=2 --gpus-per-node=8 scripts/slurm/train_demian_vla.slurm mix5

Defaults: 40K steps (the checkpoint evaluated in Table 1), global batch 256.

The RoboCasa365 VLA uses the same Pi0.5 backbone via a parallel launcher, scripts/slurm/train_demian_vla_robocasa.slurm, fine-tuning on the RoboCasa365 atomic split (data/robocasa365_v1; see data/README.md to obtain it and reproduce the packed images). Unlike the MolmoBot recipe, the annotated aspects add the auxiliary LM loss (λ_LM = 0.1, config pi05_robocasa_lm); the baseline uses pi05_robocasa with no LM loss. The aspect argument selects both the config and the appended-caption key:

# Set DATA_ROOT (RoboCasa365 LeRobot root, e.g. data/robocasa365_v1),
# PRETRAINED_CKPT, and CONTAINER via env first.
sbatch --nodes=1 --gpus-per-node=8 scripts/slurm/train_demian_vla_robocasa.slurm reasoning

• DeMiAn-VLA — eval launchers (RoboCasa, RoboCasa + instructor, composite tasks, MolmoSpaces + instructor) in molmobot_pi0/README.md; the RoboCasa eval (websocket policy server + sim) is detailed in openpi/README.md.
• DeMiAn-WAM — eval launchers and the RoboCasa eval in demian_wam/README.md.

@misc{kim2026instructrobotdenselanguage,
      title={How to Instruct Your Robot: Dense Language Annotations Power Robot Policy Learning}, 
      author={Bosung Kim and Ruiyi Wang and David Acuna and Jaehun Jung and Alexander Trevithick and Brandon Cui and Yejin Choi and Prithviraj Ammanabrolu},
      year={2026},
      eprint={2605.17077},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2605.17077}, 
}