Three tasks, one video–action model.
We study egocentric manipulation using three aligned signals: RGB video, articulated 3D hand motion, and camera motion. The hand representation specifies finger articulation and hand position; the camera trajectory records the changing viewpoint.
The same trained model supports three input–output settings. Perceive receives a video and reconstructs motion. Predict receives motion and generates a video. Plan receives an initial image, an instruction, and a goal image, then generates both video and motion.
Train with partially observed video and actions.
For each training clip, we sample which video frames and action states are observed. The first image is always provided. The model uses the observed data to predict the missing frames and states. At inference, we choose the observation mask for Perceive, Predict, or Plan.
Video targets are noised and trained with a flow-matching loss. Missing action states are replaced by learned query embeddings and predicted directly. Observed video stays clean, and observed hand and camera states remain available as inputs.
Inputs and prediction targets
Select a mode to see what is supplied and what is generated.
Example training mask. Some frames and action states are provided; the rest must be predicted. A new mask is sampled during training, and the first image always remains observed.
Train on the missing video frames and action states.
How video and actions interact inside the model
Video and action tokens share attention and base weights. Observed hand skeletons and camera-ray maps also guide the video branch.
Predicting hand parameters and camera poses
The action output contains MANO hand rotations, sequence-level shape, root position, and hand presence. A fixed MANO layer converts these parameters to 3D joints. Camera poses specify rotation and translation relative to the first frame.
Alongside video generation, training supervises hand parameters, 3D joint positions, temporal hand motion, hand presence, and camera states. The predicted hand trajectory can be visualized as a mesh or retargeted to a robot hand.
Compare unified training with task-specific models.
We compare unified training with single-mode training, then evaluate each mode against specialized baselines. Perceive and Predict are evaluated on EgoDex, ARCTIC, HOT3D, and EgoLive. Planning evaluation measures video quality, instruction following, and the progression of the manipulation.
With the same backbone and 5,000 total training steps, unified training improves hand reconstruction, video prediction, and goal-conditioned planning. The comparisons below show hand error, video PSNR, and all five Plan scores; the paper and appendix report the full metrics, training exposure, and evaluation protocols.
What changes when the tasks train together?
Same backbone · 5,000 total training steps per model
Perceive
Camera-space MPJPE (mm)
↓ Lower is better
Predict
PSNR (dB)
↑ Higher is better
Plan
Video quality · five VLM scores
↑ Higher is better · scores 1–5
Perceive / Predict use the equal-weight average of EgoDex-MANO, ARCTIC, HOT3D and EgoLive. Plan uses matched initial images, instructions and goal images. Full metrics and protocols: paper and appendix.
Qualitative results
Selected scenes compare the generated video and hand motion. Each gallery opens on a featured example; other scenes remain available below.
Perceive · Video → motion
Reconstructing 3D hand motion from video
Perceive takes the complete video as input and predicts hand and camera trajectories in one forward pass. We compare the reconstructed hands with HaMeR and Dyn-HaMR, checking both 3D hand position and finger articulation.
Inspect the wrist position, finger pose, and missed hands. Use the reference video to check whether the reconstructed mesh follows each visible hand. The captions mark failed baseline fits where applicable.
Opening a wooden box
ARCTICFilm example, with per-hand Procrustes alignment. Compare the missing left hand near the start and the deformed right-hand mesh later in Dyn-HaMR. Alignment evaluates articulation visually, not absolute hand position.
The next comparison supplies the hand and camera motion and evaluates the generated video.
Predict · Motion → video
Generating video from prescribed motion
Predict receives an initial image, an instruction, and prescribed hand and camera trajectories. It generates the future video while keeping the initial image fixed.
Inspect motion following and object consistency. Compare the generated hand positions with the reference at the same time, and check for changes in object shape or background as the camera moves. Each baseline uses its native conditioning protocol, as detailed in the paper.
Using scissors
ARCTICFilm example. Compare hand shape and the scissors throughout the motion. All six panels share the original 16 FPS timeline.
Plan uses the same model without a prescribed hand or camera trajectory; both are generated together with the video.
Plan · Visual goal → video and motion
Generating video and motion from a visual goal
Plan receives the initial image, instruction, and goal image. It generates the video and 3D trajectories between the supplied images. The examples below include tasks with several visual subgoals.
Inspect each transition, including hand–object contact. The storyboard view shows generated RGB and hand meshes. “Compare methods” shows video-generation baselines under matched starting images, instructions, and goals; captions identify comparisons limited to the first stage.
Building a LEGO structure
EgoDex4 matched stages. Each method receives the same stage-specific initial image, instruction, and goal image. Playback is normalized to stage progress; Cosmos is slowed from 24 to 16 FPS. Terminal goal-anchor frames are omitted for all methods. All panels are stretched to the same display size.
Specify the next object arrangement with an image.
An instruction such as “place the blocks apart” leaves their final positions unspecified. A goal image shows the desired arrangement. Plan combines it with the initial image and instruction to generate the intervening video and motion.
For longer tasks, reference images or an image generator provide a sequence of visual subgoals. Each planning window receives a starting image, an instruction, and the next goal. The sandwich example adds one ingredient at a time.
With and without goal images
In this LEGO comparison, both conditions use the same starting image and instruction for each window. One also receives the goal image. Compare the final block arrangement at each stage with its target.
Retarget the generated hand motion to a robot.
The robot does not execute the generated video. We retarget the predicted human hand trajectory to robot-hand commands, then execute those commands. The two clips show the generated manipulation followed by fruit transfer and bimanual block manipulation on the robot.
- 01Initial image, instruction, and visual goal
- 02Generate video and 3D hand motion
- 03Retarget the trajectory and execute
These demonstrations use open-loop execution: the robot follows the retargeted trajectory without using new observations to correct its motion during execution.
Current capabilities and limitations
P³-WAM supports video-based motion reconstruction, motion-conditioned video generation, and goal-conditioned joint video–motion generation with one trained model. The matched-step experiments show improvements over training each mode separately.
Absolute 3D hand position remains ambiguous from monocular video. The robot demonstrations also lack feedback during execution. Additional sensing could constrain depth, and closed-loop control could use new observations to correct the trajectory.
The paper with appendix contains the formulation, metrics, and evaluation protocols. The supplementary video below presents the method and demonstrations with narration.