Predicted Futures Are Not Enough: Learning Executable Goals for Robot Manipulation

Paper · Project page · Checkpoints

A 3D trace world model predicts one future per episode; the Entity-Level Goal Readout turns that prediction into a single executable goal in SE(3), and a shared Pose-Native Executor runs it closed loop at 20 Hz. Five ManiSkill3 tasks: PickCube, LiftPegUpright, PegInsertionSide, StackCube, PushCube. This repository holds the code, the frozen goal banks and the evaluation records the paper's numbers were read from.

Frozen T5, DINOv3 and SigLIP encoders supply a shared representation. The 3D Trace Planner (73.5M trained parameters) predicts 400 query points over 32 future steps; the Entity-Level Goal Readout (3.05M) reads one object-relative SE(3) goal out of the same representation, fixed for the episode. The Pose-Native Executor (804,002 parameters) closes the loop at 20 Hz on object-pose feedback, emitting seven joint increments and a gripper target without rerunning the planner. The Rigid Readout baseline recovers the same goal by weighted Kabsch inside RANSAC on the predicted tracks instead of predicting it.

Three planner checkpoints and three executors are released; which one produced which table cell is in docs/REPRODUCTION.md.

Docs: GLOSSARY (paper terms to code identifiers) · REPRODUCTION (every reported number, and how to re-run it) · PROTOCOL · MODULE_MAP · KNOWN_ISSUES

The project page and the weights are live; <ARXIV_ID> is filled in once the preprint is online.

One conda environment, one GPU, 70 MB of downloads. No planner, no DINOv3 licence, no TraceGen checkout -- the goal banks are frozen, so the planner does not have to run.

git clone https://github.com/Claire0730/executable-goals && cd executable-goals
conda create -n maniskill python=3.11 -y
conda run -n maniskill pip install -r requirements/maniskill_freeze.txt
export PM=$(conda run -n maniskill which python)

hf download Claire0730/executable-goals --local-dir checkpoints --include "student/*" --include "teacher/*" --include "SHA256SUMS"      # 12 MB of weights
hf download Claire0730/executable-goals --local-dir . --include "banks/*"                    # 58 MB of frozen goals
bash scripts/00_link_checkpoints.sh
ROW=final bash scripts/verify_main_table.sh 999        # ~10 min on one GPU

It runs 5 tasks x 256 episodes with the executor that produced that row and compares each task with the shipped record in paper_results/table2/ within +-0.04. ROW also takes k1, k4, k4pick and oracle; the seed also takes 997 and 998. The full installation, which adds the planner, is below.

scripts/ is numbered in dependency order. Each stage reads what the previous one wrote, is idempotent per task and seed, and skips outputs that already exist; the seed argument defaults to 999. Only 20, 30 and 52 need the planner environment (PG); everything else runs on the simulator environment (PM) alone.

Setup

Inference chain — from a scene to a success rate

Training chain — everything from scratch

The two chains meet at the executor: 70 produces it, 40/41/verify_main_table.sh evaluate it.

In addition to the quick start. Needs Linux x86_64, an NVIDIA GPU (sm_75+, CUDA 12.8 wheels; the freezes were taken on an RTX 5090) and a Vulkan-capable driver for SAPIEN off-screen rendering. Every stage waits for free VRAM through nvidia-smi; without it, set GPU_WAIT=0.

# planner environment (the quick start only builds `maniskill`)
conda create -n trace_gen python=3.10 -y && conda run -n trace_gen pip install -r requirements/trace_gen_freeze.txt
export PG=$(conda run -n trace_gen which python)

# upstream TraceGen at the pinned commit, plus the local patch (not vendored here)
git clone https://github.com/jayLEE0301/TraceGen third_party/TraceGen
git -C third_party/TraceGen checkout $(cat third_party/TRACEGEN_COMMIT)
git -C third_party/TraceGen apply ../tracegen_local.patch

hf download Claire0730/executable-goals --local-dir checkpoints     # now the planners too (~0.95 GB)
# banks/ as in the quick start; regenerating them instead is 10 -> 20 -> 30
bash scripts/00_link_checkpoints.sh

Both freeze files start with --extra-index-url https://download.pytorch.org/whl/cu128, which pip needs for the +cu128 wheels. Building a planner downloads the frozen encoders from the Hugging Face Hub, and DINOv3 is gated: accept its licence and run hf auth login once. ManiSkill demonstration packages are needed only for 50_collect_data.sh.

The planner files hold the trained parameters only. Every frozen-encoder tensor was bitwise identical to the published Hub weights and was removed, together with the optimizer and gradient-scaler state; the encoders are re-created from the Hub when the planner is built, and the reduced files reproduce the production predictions bitwise (checked on 16 scenes). Digests: checkpoints/SHA256SUMS.

Each planner file also carries the configuration of the run that trained it. Its data.dataset_dirs, data.cache_dir and checkpoint_dir are released as [path-to-the-repository-root-here]/...: the authors' absolute paths were replaced, the relative part kept. These fields are metadata — prediction and evaluation never read them — but 52_train_planner.sh resuming from a released checkpoint needs them set to real directories, so edit them (or pass the equivalent flags) before you resume training.

The quick start runs ROW=final. The same script takes the other rows and the other two evaluation seeds:

ROW=k1     bash scripts/verify_main_table.sh 999   # Rigid Readout K=1
ROW=k4     bash scripts/verify_main_table.sh 999   # Rigid Readout K=4
ROW=k4pick bash scripts/verify_main_table.sh 999   # the PickCube cell of the K=4 row as printed (seed 999 only)
ROW=oracle bash scripts/verify_main_table.sh 999   # Oracle Goal references (Fig. 5a)

Success rates in percent, PickCube / LiftPegUpright / PegInsertionSide / StackCube / PushCube. "Printed" is the paper; "recorded" is the mean over seeds 999, 997 and 998 of the shipped records (768 episodes per task).

Per-seed values, the bank and record behind every cell, Table I, Figs. 6 and 7, the full inference chain and the training chain: docs/REPRODUCTION.md.

Nothing in checkpoints/ is a third-party weight. Every external model the code touches is listed here with its source and licence; each is obtained by the user, not by this repository.

The frozen encoders are fetched from the Hub when the planner is built. DINOv3 is gated: accept its licence with the account you will use and run hf auth login once, or the planner cannot be built. verify_main_table.sh builds no planner and therefore needs no Hub session, no DINOv3 licence, no TraceGen checkout and no SAM 2.

• Real-robot deployment and the CoTracker3 online-tracking loop are not part of this release; the Table II records use the simulator's per-step object correspondence for the executor's pose feedback.
• The goal chain places object queries with simulator segmentation, which at test time is a privilege of the simulated setting.
• verify_main_table.sh reproduces each row within ±0.04, not bitwise; PegInsertionSide differs between identical runs. All results are single-training-seed; the three evaluation seeds vary scenes, not training.
• Retraining mix5_t2k_gmap uses goal-map labels v2 and is not bitwise comparable to the released checkpoint. Numbered caveats: docs/KNOWN_ISSUES.md.

Code and released checkpoints: Apache-2.0 (LICENSE, NOTICE). The released checkpoints contain only parameters trained by the authors; no third-party weights are redistributed. Cite the paper via CITATION.cff.