RoboReel

Learning from Observation (LfO) is a fundamental robotic capability that replicates how humans and animals socially learn from each other. Beyond its biological parallels, this modality offers a practical solution for data scaling in sample‑inefficient, data‑starved domains like robotics. Recent work has demonstrated promising results in learning manipulation skills from human videos, yet progress in this area remains difficult to assess: existing methods vary widely in assumptions, hardware choices, and environment setups, making meaningful comparisons — and identifying real advances — hard to come by.

To address this, we introduce RoboReel: a unified benchmark for evaluating models that learn policies from human videos. RoboReel bundles real‑world human demonstration videos, simulated robot trajectories, and evaluation environments across ten manipulation tasks, with four test suites that probe robustness to visual distractors and the ability to complete long‑horizon tasks. We evaluate learning‑from‑observation models from three different representational categories — over seven state‑of‑the‑art algorithms, including our own VLA‑based variants — and analyze where each family succeeds and fails. Our results show that long‑horizon tasks and tasks with low manipulation tolerances remain challenging for every current method.

Select a task to watch its human demonstration and paired robot rollout

Pick Cube — gripper + object keypoints

Close Drawer — 4 corner keypoints (overhead)

Open Lid — lid + basket keypoints

Bowl in Plate — bowl + plate keypoints

A ♠ ♥ ♣

Explicit representation

Video → language

Human demo video

⋯

⋯

Pre‑trained VLM e.g. Qwen3‑VL‑4B

Language instruction

“Put the panda toy in the basket, then put the elephant in the basket.”

+

Sim. observation

Down‑stream VLA or planner acts on the sentence alone

A pre‑trained VLM watches the demo and writes down what it saw. Everything downstream sees only that sentence — the video itself is discarded, so any detail the words miss is gone.

Baselines SeeDo VLM + π₀.₅ π₀.₅ (expert language)

B ♠ ♥ ♣

Implicit representation

Video → latent

Human demo video

⋯

⋯

Pre‑trained visual encoder uniform · keyframe sampling

Latent video encoding

Visual tokens — a skill embedding, never words

+

Sim. observation

Down‑stream end‑to‑end policy conditioned on the embedding

An encoder compresses the demo into latent skill tokens that condition the policy directly. Nothing has to survive a round trip through language — and this family wins on every suite.

Baselines Vid.‑cond. π₀.₅ (Uniform) Vid.‑cond. π₀.₅ (Keyframes) Vid2Robot UniSkill

C ♦

Keypoint‑based learning

Video → 3D keypoints

Pose‑defined demo

⋯

⋯

Keypoint tracking + retargeting DIFT · CoTracker

3D keypoint trajectory

Hand + object keypoints, in the robot’s frame

+

Calibrated env.

Imitation‑learning policy predicts the next end‑effector keypoints

Hand and object keypoints are tracked through the demo and retargeted into the robot’s end‑effector frame, so human and robot share one action space — but grasp pose is hard to read off points alone.

Baselines PointPolicy

Introduced and trained in this work Language oracle — expert‑written instructions, to upper‑bound the VLM’s output Suits mark the test suites each family is scored on.

N/A — SeeDo's action vocabulary is restricted to pick‑and‑place tasks.