Robust Instant Policy (RIP)

Official implementation of Robust Instant Policy: Leveraging Student's t-Regression Model for Robust In-context Imitation Learning of Robot Manipulation, presented at IROS 2025.

[Paper] [Project page]

TL;DR — An LLM-based instant policy can learn a new robot task from a few-shot set of demonstrations (ten in this example), avoiding much of the extra data collection and task-specific fine-tuning typically associated with VLA-style models. The trade-off is that an LLM can hallucinate an implausible action trajectory. RIP addresses this by requesting several candidate trajectories and using robust Student's-t regression to trust their shared motion while downweighting outliers, yielding one safer trajectory to pass to the robot stack.

RIP has a simple idea: ask an LLM for several ways to perform the same robot task, then trust the candidates that agree with one another more than the ones that do not. The result is one robust trajectory instead of one brittle LLM answer. Student's-t regression is the mechanism that gives outlying candidates less weight.

This repository is a compact, runnable guide to the method. It includes a ten-demonstration banana-picking example, DINO keypoint extraction, and an optional rigid-gripper correction step. It does not aim to reproduce every simulation or real-robot result from the paper.

Start with Trajectory aggregation to turn ten saved banana candidates into one RIP trajectory. The DINO and Triangle matching sections are optional building blocks for image-based keypoints and robot-ready gripper geometry.

Python 3.10 or newer is required. The command below installs OpenAI support and plotting support used by the banana example:

git clone --recurse-submodules https://github.com/aistairc/rip.git
cd rip
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[openai,plot]"

The DINO example additionally needs the vision extra. If the repository was cloned without submodules, initialize its optional upstream dependency first:

git submodule update --init --recursive
python -m pip install -e ".[vision]"

Use the Python API when RIP is one component of a larger application. Give it demonstrations and a query observation; it returns one [steps, 10] trajectory:

from rip import RegressionConfig, generate_rip_trajectory
from rip.schema import RIPContext

context = RIPContext.model_validate_json(
    open("examples/data/pick_banana_10_demos.json").read()
)
result = generate_rip_trajectory(
    context,
    num_queries=10,
    regression=RegressionConfig(nu=1.5, learning_steps=40_000, seed=10),
)
trajectory = result.trajectory  # [steps, 10]

Each KAT action is a 10-D vector: three world-frame 3-D gripper keypoints plus one grasp value. Under the hood, RIP follows Algorithm 1:

1. Ask the instant policy for Q candidate trajectories (Q=10 here; use 5 for the paper setting).
2. Stretch or compress them to the same normalized time axis.
3. Fit the Student's-t model and reduce the weight of disagreeing candidates.
4. Read the fitted mean trajectory as the RIP output.

This optional example finds the same visually meaningful image regions across five demonstrations and one new test image. These 2-D correspondences are the image-side input to a KAT-style robot action representation.

examples/data/dino_banana/ contains five banana-picking demonstrations (demo_01.jpg through demo_05.jpg) and one held-out test image (test.jpg). Run:

python examples/extract_dino_banana_keypoints.py \
  --device cpu \
  --output dino_banana_keypoints.json \
  --plot-output dino_banana_keypoints_1x6.png

The command chooses five spatially separated keypoints in every image and writes their pixel coordinates, saliency, patch-grid locations, and descriptor shapes to JSON.

The five grey-bordered panels are demonstrations; the sky-blue-bordered panel is the held-out test image. A point keeps the same colour in each panel, making the shared image regions easy to follow.

The unmodified upstream dino-vit-features repository is pinned at external/dino-vit-features. All RIP-specific integration is in rip/vision/dino_adapter.py; the submodule itself is never modified.

from rip.vision import DinoVITAdapter

adapter = DinoVITAdapter(model_type="dino_vits8", stride=4)
features = adapter.extract("observation.jpg", load_size=224, layer=9)

The first extraction downloads DINO weights through PyTorch Hub. Later feature selection code operates on the adapter output, so this repository's code stays separate from the upstream dependency.

The first run extracts all six images and writes the tensor-only cache dino_banana_features.pt. A later run validates the DINO settings and each image's path, size, and modification time, then reuses descriptors and saliency maps without constructing a DINO model. Use --refresh-features after changing an image or an extraction setting. This retains the original implementation's demo-feature reuse while also caching the held-out test image.

The selection procedure follows the original implementation: retain patches that agree across images, keep salient foreground patches, cluster their DINO descriptors, then select one salient representative from each cluster. This gives every displayed point a stable identity and colour. The defaults are dino_vits8, stride 4, layer 9, load size 224, and a saliency threshold of 0.05.

The JSON output records whether features were extracted or read from cache, and the elapsed feature stage time. The bundled data was verified with five keypoints per image and a warm PyTorch Hub model cache on an NVIDIA RTX PRO 6000 Blackwell Max-Q:

This is a 12.239 s (98.1%) reduction, or 51.4× faster, for the feature stage. Timing varies with device, DINO-weight download state, and input resolution. Changing only --num-keypoints, --saliency-threshold, or --seed still uses the cache.

This example stops at 2-D image correspondences. Converting them to world-frame 3-D robot keypoints needs task-specific RGB-D calibration and is outside this repository.

This is the main runnable example. It turns ten banana-picking demonstrations into one trajectory by combining ten candidate KAT trajectories. Candidates that disagree with the common motion receive less influence in the final RIP trajectory.

The bundled context contains ten demonstrations. Each action step is a 10-D vector containing three 3-D gripper points and one grasp value:

[p1_x, p1_y, p1_z, p2_x, p2_y, p2_z, p3_x, p3_y, p3_z, grasp]

For the fastest first run, aggregate the ten saved GPT-4o candidates. No API key is required:

python examples/generate_rip_trajectory.py \
  --candidate-trajectories examples/data/pick_banana_candidates.json \
  --learning-steps 40000 \
  --seed 10 \
  --plot-output banana_trajectory_vs_step.png

To generate ten new candidates from the demonstrations before aggregating them:

export OPENAI_API_KEY="your-api-key"
python examples/generate_rip_trajectory.py

The command writes rip_trajectory.json. --plot-output additionally writes a simple diagnostic: gripper-center x, y, and z over time. The ten candidate paths are scatter points and the final RIP path is a sky-blue line. Candidate point colour shows how much RIP trusted that point: red is low, sky blue is 1.2 or greater.

The default 2,000 optimizer updates are for a quick run. Use --learning-steps 40000 for the paper's simulation setting. The default LLM is gpt-4o-2024-08-06; use --model to select another supported model. The checked-in examples/data/pick_banana_rip_trajectory.json is the successful 55-step reference output of the saved-candidate command above.

Successful banana lift from ten saved candidates, 40,000 optimizer updates, and regression seed 10. The three panels show the candidate KAT paths and the final RIP line. Candidate-point colour is the learned per-step Student's-t weight: red is low and sky blue is 1.2 or above.

The output JSON also includes candidate_influence. It summarises how strongly each whole candidate was trusted: its mean Student's-t weight, its normalized share, and its standardized residual. This reports fitted downweighting, not a causal “what if this candidate were removed?” analysis; the latter would require ten additional 40,000-step fits.

RIP combines the three gripper keypoints independently. That makes it robust to bad candidates, but the three resulting points may no longer form a gripper of the right size and shape. Triangle matching is an optional final cleanup step: it turns each end-effector, left-finger, and right-finger triplet back into a physically consistent gripper triangle before control.

For every action [ee_xyz, left_finger_xyz, right_finger_xyz, grasp], the example keeps grasp unchanged and finds the closest translated and rotated version of the fixed KAT gripper triangle. The saved result uses batched Kabsch alignment, a deterministic best-fit rigid transform. Its per-action RMSE tells you how much the original RIP keypoints had to move; an application can set its own acceptable correction threshold.

Run it on the bundled successful banana RIP trajectory:

python examples/match_triangle_trajectory.py \
  --input examples/data/pick_banana_rip_trajectory.json \
  --output triangle_matched_trajectory.json \
  --process-output triangle_matching_worst_step_process.png

On the bundled 55-step successful banana trajectory, the mean correction is 0.467 mm and the largest is 1.801 mm. RMSE measures the geometric change made by triangle matching. It does not measure task success or collision safety.

--process-output adds a diagnostic figure for the action that needed the largest correction. It runs the original KAT fitter's 100 Adam updates only to show how the fit improves; this is separate from RIP regression's 40,000 optimizer updates. The JSON always contains the verified Kabsch result, whether or not the figure is requested.

The panels show the action that needed the largest correction. All 100 Adam updates run, but only the best-so-far improvements are displayed. The title shows the update number and RMSE. Magenta is the original RIP triangle; teal is the rigid match. The last Kabsch panel is what is saved to JSON. Triangle matching restores gripper shape only—it does not replace collision checking, inverse kinematics, or execution-time safety checks.

@inproceedings{oh2025rip,
  author    = {Oh, Hanbit and Salcedo-Vázquez, Andrea M. and
               Ramirez-Alpizar, Ixchel G. and Domae, Yukiyasu},
  booktitle = {2025 IEEE/RSJ International Conference on Intelligent Robots
               and Systems (IROS)},
  title     = {Robust Instant Policy: Leveraging Student’s t-Regression Model
               for Robust In-context Imitation Learning of Robot Manipulation},
  year      = {2025},
  pages     = {7973--7980},
  doi       = {10.1109/IROS60139.2025.11247005},
}

This repository is released under the BSD 2-Clause License. Third-party code under external/ remains subject to its own license.