From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning

S2E is a robot learning framework for building visual navigation models — pretrain on web videos with imitation learning, then finetune with reinforcement learning in URBAN-SIM. We provide a 3DGS-based navigation benchmark, a pretraining pipeline, an RL finetuning pipeline, and a model zoo with unified inference interfaces.

To ensure the best experience with URBAN-SIM, please review the following hardware guidelines:

• Recommended Hardware:
  • OS: Ubuntu 22.04; 24.04.
  • GPU: Nvidia GPU with at least 16GB RAM and 12GB VRAM.
    • Tested GPUs: Nvidia RTX-4080, RTX-5080, RTX-5090, L40S.
  • Storage: Minimum of 50GB free space.

# Clone the repository (includes third_party submodules)
git clone -b main --recurse-submodules https://github.com/VAIL-UCLA/S2E.git ~/S2E
cd ~/S2E
# If you already cloned without --recurse-submodules:
# git submodule update --init --recursive

# Install Isaacsim 5.x from github repo: https://github.com/isaac-sim/IsaacSim
suppose that you install it in:
# ${HOME}/IsaacSim/
and the builded software in:
# _build/linux-x86_64/release
Then try:
cd ~/S2E
ln -s ${HOME}/IsaacSim/_build/linux-x86_64/release ./_isaac_sim
# Isaac Sim root directory
export ISAACSIM_PATH="${HOME}/IsaacSim/_build/linux-x86_64/release"
# Isaac Sim python executable
export ISAACSIM_PYTHON_EXE="${ISAACSIM_PATH}/python.sh"
# note: you can pass the argument "--help" to see all arguments possible.
${ISAACSIM_PATH}/isaac-sim.sh

# Bundled at third_party/isaaclab-extension
cd third_party/isaaclab-extension
# install the isaaclab following the doc

# Bundled at third_party/gsplat
pip install ninja numpy jaxtyping rich
cd third_party/gsplat
pip install -e . --no-build-isolation -v
# For the Vid2Sim rasterizer, use third_party/vid2sim-rasterizer-gsplat instead

pip install -e NavbenchGS
python NavbenchGS/scripts/download_scenarios.py --output-dir data/NavBenchGS

After download, data/NavBenchGS/ is laid out as:

data/NavBenchGS/
├── vid2sim_raw/                              # Raw per-scene captures
│   ├── 0000/, 0001/, 0003/, ..., 0044/       # 4-digit scene IDs (non-contiguous)
│   ├── old-ped/                              # Legacy pedestrian captures
│   ├── invalid_list.txt                      # Scene IDs to skip
│   └── upload.sh
├── vid2sim_torch/                            # PyTorch-formatted per-scene data
│   └── 0000/, 0001/, 0003/, ...              # same scene IDs as vid2sim_raw
├── vid2sim_starting_ending_position.json     # Start/goal positions per scene
├── gs.gif                                    # Preview animation
└── README.md

Note: the dataset is gated — accept the terms on HuggingFace and log in (huggingface-cli login or pass --token) before downloading.

Unified waypoint-to-action debug for both robots — pure-pursuit drives COCO (Ackermann) or Go2 (JIT locomotion policy):

# COCO — Ackermann car on a circle
${ISAACSIM_PYTHON_EXE} NavbenchGS/scripts/run_waypoint_to_action.py --enable_cameras \
    --robot coco --trajectory circle --num_steps 3000

# Go2 — JIT policy; defaults to robots/go2/unitree_go2_actor.pt
${ISAACSIM_PYTHON_EXE} NavbenchGS/scripts/run_waypoint_to_action.py --enable_cameras \
    --robot unitree_go2 --trajectory circle --forward_speed 0.6

Robot articulation configs and action terms live in NavbenchGS/navbench_gs/robots/ (coco.py, unitree_go2.py). USDs + Go2 policy ship under robots/.

The web-pretrained S2E navigation model is released in the UCLA-VAIL/Navigation-Model-Zoo-Public model zoo — exported to ONNX behind a unified inference interface (shared with GNM, ViNT, NoMaD, CityWalker, MBRA, and MIMIC).

⚠️ Pretrained (BC) weights only. The released checkpoint is the behavior-cloning, web-pretrained model (the IL pretraining output), not the RL-finetuned policy from the paper. Use it as a navigation prior / BC baseline; RL-finetuned weights will be released separately.

1. Download the pretrained model

pip install -U "huggingface_hub[cli]"
hf download UCLA-VAIL/Navigation-Model-Zoo-Public --include "S2E/*" --local-dir ./nav_model_zoo

2. Run inference — requires onnxruntime-gpu (or onnxruntime), numpy, torch, opencv-python, pyyaml:

import numpy as np
from S2E.inference import S2ENavigator        # run from ./nav_model_zoo

nav = S2ENavigator(device="cuda")             # device="cpu" if no GPU
# obs: the robot's last 11 RGB frames, (1, 11, 3, H, W) float32 in [0, 1]
obs = np.random.rand(1, 11, 3, 256, 256).astype(np.float32)

# point-goal in the standard frame (x = forward, y = left, meters)
traj, scores = nav.inference_trajectory(obs, goal_xy=np.array([5.0, 0.0]))  # (1, 1, 10, 2) m
vw, best     = nav.inference_vw(obs,        goal_xy=np.array([5.0, 0.0]))   # (1, 2) = [v, ω]
nav.reset()                                   # clear PD smoothing between episodes

3. Drive a robot on NavBench-GS. S2ENavigator outputs local waypoints / (v, ω) in the same standard frame that NavBench-GS's waypoint→action layer consumes (pure-pursuit → Ackermann for COCO, or the Go2 locomotion policy), so the pretrained model plugs straight into the benchmark's control stack. Closed-loop, model-driven NavBench-GS rollouts are being released — see the Release Timeline below.

• NavBench-GS
  • 3D-GS Scenarios
  • Benchmark engine
  • Benchmark detail settings
• Pretraining framework and model zoo
• Finetuning framework and trained weights

The project can not be built without the following open-sourced projects:

• Omniverse: https://www.nvidia.com/en-us/omniverse/

• IsaacSim: https://developer.nvidia.com/isaac/sim

• IsaacLab: https://github.com/isaac-sim/IsaacLab

• gsplat: https://github.com/nerfstudio-project/gsplat.git

• URBAN-SIM: https://github.com/metadriverse/urban-sim

• Vid2Sim: https://github.com/Vid2Sim/Vid2Sim

• CityWalker: https://github.com/ai4ce/CityWalker

• VisualNav-Transformer: https://github.com/robodhruv/visualnav-transformer

If you find S2E helpful for your research, please cite the following BibTeX entry.

@inproceedings{he2025seeing,
  title={From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning},
  author={Honglin He and Yukai Ma and Brad Squicciarini and Wayne Wu and Bolei Zhou},
  booktitle={International Conference on Learning Representations},
  year={2026}
}