Skip to content
xmz111Public

About

No description, website, or topics provided.

Resources

Stars

18 stars

Watchers

0 watching

Forks

Latest commit

 

History

7 Commits

Folders and files

Repository files navigation

NowWAM

Beyond Future Prediction:
Denoising as Generative Adaptation for Robot Control

Zanyi Wang, Yuheng Lei, Dengyang Jiang, Ping Luo, Mengdi Wang, Zhixuan Liang, and Shilong Liu

Paper on arXiv Project page Model weights

Paper PDF · Quick start · Evaluation · Training · Citation

NowWAM adapts pretrained generative models to robot control through current-observation denoising, without a separate future-target branch. We provide model weights, data preparation, training and evaluation code.

📦 Klein weights · 🏋️ LIBERO & RoboCasa (100-shot) training · 🤖 LIBERO-Plus & RoboCasa evaluation

Checkpoints

Quick start

Use Python 3.11 on x86-64 Linux with an NVIDIA GPU and a working CUDA driver. On Ubuntu/Debian, install the system packages first:

sudo apt-get update
sudo apt-get install -y git build-essential python3.11-venv ffmpeg libgl1 libegl1 libglib2.0-0 libmagickwand-dev

Accept the model access terms for the FLUX.2 VAE if required, then install and authenticate:

git clone https://github.com/xmz111/NowWAM.git
cd NowWAM
python3.11 scripts/install.py Klein-LIBERO --dependencies-only
.venv/bin/hf auth login
python3.11 scripts/install.py Klein-LIBERO
source .venv/bin/activate
python -m eval Klein-LIBERO --limit 4

The installer downloads the weights and assets, prepares the simulator, and runs a one-episode test. Replace Klein-LIBERO with Klein-RoboCasa-100shot to evaluate RoboCasa. Use all with the installer to prepare both checkpoints; evaluate each by name. Simulator environments are selected automatically. Do not reinstall during a run.

Evaluation

Full evaluation, with optional parallel lanes:

python -m eval Klein-LIBERO
# Or use two GPUs with two lanes each:
python -m eval Klein-LIBERO --gpus 0,1 --lanes 2

Results are saved under outputs/<model>. Re-run the same command to resume. Short tests use a separate output directory. Each parallel lane holds its own policy; start with one lane if GPU memory is limited. Keep the installed environment and assets unchanged while resuming. For results from an older evaluator, use a new directory with --output outputs/new-run.

Training

Run from the repository root. Use four GPUs with DDP (tested on 4 × H200, 141 GB each) and allow 500 GB for data, assets and checkpoints. Keep training in its own environment; do not run pip install -e . inside it. Accept any required base-model access terms and the VAE terms above before downloading.

python3.11 -m venv .venv-train
source .venv-train/bin/activate
pip install -r requirements/train.txt
hf auth login

Download the base models once; both training recipes use them:

Base-model download
hf download black-forest-labs/FLUX.2-klein-base-4B flux-2-klein-base-4b.safetensors \
  --revision a3b4f4849157f664bdbc776fd7453c2783562f4d --local-dir assets/FLUX.2-klein-base-4B
hf download black-forest-labs/FLUX.2-dev ae.safetensors \
  --revision 26afe3a78bb242c0a8bb181dcc8937bb16e5c66c --local-dir assets/FLUX.2-dev
hf download Qwen/Qwen3-4B --revision 1cfa9a7208912126459214e8b04321603b3df60c \
  --include '*.json' '*.safetensors' 'merges.txt' 'vocab.json' 'tokenizer.model' --local-dir assets/Qwen3-4B

[ -d assets/sources/flux2/.git ] || git clone --no-checkout https://github.com/black-forest-labs/flux2.git assets/sources/flux2
git -C assets/sources/flux2 checkout --detach 50fe5162777813d869182b139e83b10743caef15
python -m training.prepare

LIBERO

Download the training data and train:

hf download yuanty/LIBERO-fastwam --repo-type dataset \
  --revision ee018b997c430bb12b5bf3c892d744798c5a2f91 \
  --include 'libero_*_no_noops_lerobot.tar.gz' --local-dir data/archives
for suite in spatial object goal 10; do
  tar -xzf "data/archives/libero_${suite}_no_noops_lerobot.tar.gz" -C data
done
OMP_NUM_THREADS=4 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  torchrun --standalone --nproc-per-node=4 -m training

RoboCasa (100-shot)

Prepare the RoboCasa simulator, then download and process the training data from .venv-train:

python3.11 scripts/install.py Klein-RoboCasa-100shot
python -m training.robocasa_data
OMP_NUM_THREADS=4 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  torchrun --standalone --nproc-per-node=4 -m training.robocasa

Preparation downloads the GR1 demonstrations, uses episodes 0–99 per task, and prepares observations, actions, statistics and text features under data/robocasa.

Both recipes use global batch 64 and 42,210 updates. Repeat the training command to resume, or append a new output directory for a new run. EMA policies are exported under outputs/<benchmark>/exports/. After evaluation setup, run:

.venv/bin/python -m eval outputs/libero/exports/step_032830 --gpus 0,1 --lanes 2
.venv/bin/python -m eval outputs/robocasa/exports/step_037520 --gpus 0,1 --lanes 2

Citation

@article{wang2026nowwam,
  title   = {Beyond Future Prediction: Denoising as Generative Adaptation for Robot Control},
  author  = {Wang, Zanyi and Lei, Yuheng and Jiang, Dengyang and Luo, Ping and Wang, Mengdi and Liang, Zhixuan and Liu, Shilong},
  journal = {arXiv preprint arXiv:2609.28339},
  year    = {2026}
}

Acknowledgements

We thank FastWAM and ImageWAM for their open-source work, and TRC for compute support.

About

No description, website, or topics provided.

Resources

Stars

18 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages