MINT: A Unified Model for World-Space Camera and Hand Motion
Estimation from Scalable Egocentric Pipeline Supervision

Zijie Zhu^1,3,4  ·  Weiren Cai³  ·  Yizhou Wang^1,3  ·  Zhenjie Yang⁴  ·  Yide Liu^3,5  ·  Jiahao Chen^3,*  ·  Guanqi He^2,3,*

_{¹ShanghaiTech University   ²Tsinghua University   ³Wuji Technology   ⁴The University of Hong Kong   ⁵Zhejiang University   ^*Corresponding authors}

中文说明

Egocentric motion reconstruction estimates how a camera moves and how both hands are positioned, oriented, and moving in the surrounding space. Producing this data often requires capture hardware with built-in tracking or a pipeline that combines calibration, monocular depth, SLAM, hand reconstruction, and trajectory cleanup. Deploying multiple models, managing errors between stages, and working with different licenses add to the cost of producing data at scale.

MINT takes ordinary egocentric RGB video and uses a unified model to predict camera and two-hand motion, then composes them into world-space outputs. The Web Viewer supports inference and inspection on an NVIDIA GPU with at least 24 GB of VRAM. Long videos are processed in windows, with camera trajectories joined across windows.

• Unified camera and hand modeling. A shared spatiotemporal representation feeds four prediction heads: camera extrinsics, field of view, camera-frame MANO, and per-frame hand presence. Explicit rigid composition produces world-space hand motion without depth maps or point clouds at inference.
• Structured pipeline amortization. EgoPipeline generates structured camera and hand supervision offline. MINT learns from these labels, reducing the need to deploy multiple independent models when processing new videos.
• Open models and research resources. The release provides model weights, training and inference code, structured non-video annotations covering 1,021 hours of egocentric footage, and reference code for EgoPipeline orchestration, cleaning, and export. Source videos and separately licensed assets must be obtained under their own access terms.

The Web Viewer is the primary MINT entry point. It lets you select a checkpoint, load the MINT model, run inference on a LeRobot episode or an egocentric MP4 video, and inspect GT/prediction overlays, camera trajectories, hand motion, frame metrics, and benchmark results in one interface.

Hardware requirement: MINT model inference requires an NVIDIA GPU with at least 24 GB of VRAM.

Install the inference environment and download the public MINT model. The Viewer also requires the separately licensed MANO hand models. Create an account on the official MANO website, accept the MANO license, and download the MANO release before running the asset check below.

git clone https://github.com/wuji-technology/wuji-ego-mint.git
cd wuji-ego-mint
bash scripts/create_env.sh inference
conda activate mint-inference
bash scripts/download_assets.sh

# Copy the MANO files from the downloaded release into the project.
mkdir -p assets/mano/mano_right assets/mano/mano_left
cp /path/to/MANO_RIGHT.pkl assets/mano/mano_right/MANO_RIGHT.pkl
cp /path/to/MANO_LEFT.pkl assets/mano/mano_left/MANO_LEFT.pkl

python -m mint doctor --profile inference
python -m mint viewer

Place the models and assets used by Quick Start at the paths below. scripts/download_assets.sh downloads the public MINT checkpoint automatically; MANO must be downloaded from its official website and copied manually.

The Viewer automatically opens http://127.0.0.1:8011 in the default browser. Then:

1. In Model and Sample, keep checkpoints/model.safetensors or select another compatible checkpoint.
2. Click Load Model and wait until the model status is ready.
3. Select a LeRobot episode or an egocentric MP4 video, then choose the camera-inference, hand-window, and geometry settings.
4. Click Start Inference.
5. Inspect the synchronized GT/Pred 2D view, fixed-world and camera-frame 3D panels, per-frame values, losses, exports, and optional benchmark tools. The export menu can either compose the selected views into one grid MP4 or download 2D, fixed-world 3D, MuJoCo, and Wuji Hand GT/Pred renders as eight individual MP4 files in a ZIP.

Testing your own video needs no configuration change and no command line: browse to the video in the input-directory picker at the bottom right of the Viewer and click it. Supported formats are .mp4, .mov, .avi, .mkv and .webm. A plain video carries no ground truth, so the Viewer runs in prediction-only mode — GT panels, the GT/Pred side-by-side layout and the loss readout are hidden, and everything shown is prediction.

All visualization operations live in the Viewer panel — there is no separate command-line visualization step. See Installation for installation, CUDA, MANO, and offline deployment details.

Within each window, four prediction heads share a spatiotemporal representation. Explicit rigid transforms compose the predicted camera and hand states into world-space motion.

EgoPipeline is our open-source implementation of the conventional multi-stage route, and it stays in the project as the supervision generator, not as the deployment path:

Its output is pseudo-label, not ground truth — that distinction is load-bearing everywhere in this repository.

A multi-stage pipeline typically combines several visual models and post-processing components. Different models may extract features from the same video independently, intermediate estimates can propagate errors across stages, and deployment requires coordinating multiple dependencies and interfaces. MINT shares a spatiotemporal representation across camera and hand prediction, reducing repeated feature extraction and integration work when processing new videos.

Inference throughput measured on RTX 4090D, at 512 × 384 and 30 fps under identical conditions; time is the marginal cost per frame in steady state. Speedups are against VITRA, which EgoPipeline, this project's data pipeline, was built by optimising. The three methods are points on one line of work: VITRA to start, EgoPipeline as the pipeline-level optimisation, MINT replacing the chain with a single model.

Against the conventional pipeline, MINT is 17.4× faster on one GPU (VITRA 1260.0 → 72.4 ms/frame) and 12.5× on four (283.3 → 22.7 ms/frame). The four-GPU columns show where that comes from: optimising the conventional pipeline itself (EgoPipeline) accounts for 3.4× (283.3 → 83.4), and the unified model takes it to 22.7.

EgoPipeline is itself distributed-optimised (Ray multi-GPU operators, persistent workers, asynchronous CPU stages): rewriting its serial single-GPU execution as a scheduled 4-GPU worker pool took 179.0 s down to 63.9 s per 270 frames, 2.8× internal to the pipeline. That is whole-clip wall-clock including decode and write-out, a different accounting from the steady-state cost above; the two do not multiply.

Result sources: MINT and MINT + UKF use the latest coverage-aware evaluation; all other results are taken from the ViDiHand evaluation.

Following the coverage-aware protocol in Sec. V-A of the paper, missed hands receive the error of a canonical MANO placeholder instead of being excluded from pose evaluation. FAcc, recall, and F1 measure detection; MPJPE-p and PA-MPJPE-p measure articulated pose; GO-p and CT-p measure wrist orientation and hand placement; Jitter measures temporal smoothness.

MINT is evaluated on HOT3D and ARCTIC in a zero-shot setting; neither dataset is used in either of its two training stages. MINT + UKF uses the same model weights as MINT and applies an unscented Kalman filter (UKF) at inference time, with all other evaluation settings held constant. Bold marks the best displayed value in each column; ties are all bolded.

In the latest local evaluation, applying UKF reduces Jitter from 12.057 to 2.373 mm/frame² on HOT3D and from 12.77 to 2.67 mm/frame² on ARCTIC, while leaving detection metrics unchanged. The hand prediction heads have not yet been fine-tuned on high-precision data; see Limitations and future work.

HaWoR supplies hand pseudo-labels to EgoPipeline. It is used at stage 04 to estimate camera-frame MANO parameters. These labels provide supervision for MINT's hand branch, but they retain reconstruction errors and do not constitute high-precision ground truth.

The error metrics are transcribed from Table 2 of the paper: HOT3D (27 sequences, 94,978 frames) and the ARCTIC P2 validation split (34 sequences, 25,883 frames). Sequences are evaluated at full length, with SE(3)-only alignment and no fitted scale — so scale error is charged rather than absorbed. The arc-length ratio is GT path length / predicted path length, averaged equally across sequences: above 1 means the predicted path is too short; below 1 means it is too long. MegaSaM† runs without depth refinement. MINT w/o stage 2 never sees metric ground truth. Bold marks follow the paper; for the arc-length ratio, the target is 1.

Following Sec. V-A of the paper, ATE and ATE% measure global trajectory accuracy, while RPE-T and RPE-R measure relative motion accuracy. RPE reflects local motion consistency; full-sequence ATE captures errors that accumulate over longer trajectories. Both are relevant to the quality of the reconstructed motion. ATE and RPE are computed as an RMSE for each sequence and summarized by a mean and a median.

The paper reports mean RPE-T of 3.39 mm on ARCTIC and 4.69 mm on HOT3D for MINT. Dedicated trajectory systems retain an advantage in absolute accuracy: MINT uses 32-frame windows without loop closure or global optimization, so long sequences can accumulate drift, as discussed in Sec. V-C.

ATE is not where this checkpoint wins: 181.7 mm on HOT3D against DROID-SLAM's 49.1 mm. The arc-length ratios of 1.094 on HOT3D and 1.412 on ARCTIC indicate predicted paths that are too short under the GT / Pred convention. These benchmark predictions should be distinguished from the scale-enlarged pipeline trajectories in the released pretraining dataset. On HOT3D, Stage 2 moves the ratio from 0.466 to 1.094, correcting an overlong path towards the target length, and lowers ATE from 524.7 mm to 181.7 mm. On ARCTIC, the ratio moves from 0.755 to 1.412 and ATE increases from 63.7 mm to 81.9 mm.

The hand-reconstruction and camera-trajectory tables follow Tables 1 and 2 of the paper. Non-MINT hand results retain their source in the ViDiHand evaluation; camera-trajectory results follow the full-sequence protocol stated in Table 2. Shared metric definitions alone do not establish that different evaluation splits or manifests are directly comparable.

• Hand evaluation. The coverage-aware protocol includes missed ground-truth hands through a canonical MANO placeholder penalty. Pose metrics therefore include true positives and false negatives.
• Camera evaluation. Each complete sequence is aligned with SE(3), without fitting scale. ATE and RPE are computed per sequence; the reported mean and median aggregate the successfully evaluated sequences.
• Reproduction. Compare local results with a reference only when the checkpoint, evaluation split and manifest, preprocessing, alignment, and aggregation settings match the intended experiment. MINT's metric definitions and reporting code are available in eval/model_effect/benchmark/.

Integrity statement. Our EgoPipeline and MINT training code support reproduction. We commit that every metric reported by this project is an authentic result produced under the stated evaluation protocol; we do not alter the original numeric results. To reproduce another method's or baseline's exact values, use that method's official repository and environment. MINT's metric definitions, alignment rules, aggregation logic, and reporting code are available in eval/model_effect/benchmark/ for inspection.

The complete implementation and tests live in eval/model_effect/benchmark/ and are integrated into the Viewer's Benchmark panel. Before use, download HOT3D and ARCTIC, organize the benchmark data as required by each adapter, and install the optional runtime. The CLI remains available:

python eval/model_effect/benchmark/run.py \
  --ckpt /path/to/checkpoint \
  --config configs/training/mint_step2.yaml \
  --data-root /path/to/benchmark-data

Set CAMERA_TRAJECTORY_ROOT for camera-trajectory exports when needed. Aliyun defaults are placeholders; users must configure the workspace, resource, image, CPFS, credentials, and environment. This project does not provision or maintain benchmark environments.

Training or local pipeline-development work requires the full environment:

bash scripts/create_env.sh full
conda activate mint
python -m mint doctor --profile full

The public release uses the Viewer as the unified entry point for MINT inference, visualization, and artifact export. This project publishes only code that third-party licenses permit us to distribute; license-restricted adaptations and internal integrations are excluded, so this repository does not contain the complete production data pipeline.

GeoCalib, MoGe, and Mega-SAM source snapshots are distributed under third_party/, but some production adaptations cannot be published under their upstream license terms. In particular, the locally adapted HaWoR source is Git-ignored because CC BY-NC-ND prohibits redistribution of modifications. Weights, MANO files, and other separately licensed assets are also excluded.

If you need to reconstruct the data-generation pipeline, first read THIRD_PARTY_NOTICES.md, obtain and install every required upstream library and asset under its own terms, and implement the necessary compatibility adapters locally. The open code in ego_pipeline/ documents the orchestration, interfaces, data flow, cleaning, manifests, and LeRobot export contracts. An AI coding assistant may help reconcile upstream API differences, but the resulting integration remains the user's responsibility and must comply with all licenses. Only after completing that local integration should the data-profile doctor and python -m mint pipeline be treated as usable entry points.

The public release provides an implementation reference, not a one-command reproduction of the production data generator. Use MINT directly for inference; for pipeline reconstruction, supply the licensed upstream code and assets and complete the local integration described in Data pipeline.

If you do go down this road, the least painful order is: install each upstream project separately by its own instructions and get it running on its own first, then wire them together against the source in ego_pipeline/. That wiring step suits an AI coding assistant well — point it at the upstream APIs and at this repository's calling conventions and have it write the compatibility layer. The fixed in-repository paths for each asset are:

One configuration per stage is kept. Stage 1 pretrains on the pipeline supervision; Stage 2 initialises from the Stage 1 weights and trains only the camera head on high-precision camera-trajectory data, and its output is the checkpoint released here.

Run from the repository root. Before Stage 1, prepare the backbone weights at assets/models/lingbot-map.pt and the three LeRobot datasets at data/lerobot/{ego4d,egodex,epickitchen}/lerobot_v3, or update model.pretrained and data.root to your own paths. Both configurations use offline W&B logging by default; see Training for logging and path settings.

python -m mint train --config configs/training/mint_step1.yaml

python -m mint train --config configs/training/mint_step2.yaml

Set the Stage 2 data.root to your prepared camera-trajectory LeRobot dataset; data/lerobot/stage2/lerobot_v3 is a placeholder. Set train.init_from to the model.safetensors produced by your own Stage 1 run, or copy it to the default location checkpoints/stage1/model.safetensors.

mint train consumes a compatible, separately prepared LeRobot dataset and writes training checkpoints. After training, select the new checkpoint directly in the Viewer panel for interactive inspection.

We provide the non-video portions of the ego4d, egodex, and epickitchen data processed by this repository's Ego data-production pipeline:

• Hugging Face: ZZJAsher/wuji_ego_mint
• ModelScope: AsherZhu/wuji_ego_mint

That figure is the whole chain accumulated, not any single rule: normalisation, cutting stretches with no hands or more than two, and dropping segments that end up too short, per-stage failures and timeouts, and outlier rejection during trajectory cleaning all reduce what survives. The final step — packing the cleaned results into the training LeRobot dataset — runs in an internal repository and is not part of this release, so only the overall figure is given here; it cannot be broken down per rule.

Important: the camera trajectories in this data were generated by this repository's Ego data pipeline and currently exhibit obvious scale enlargement. We recommend using the data only to pretrain this project's Ego hand reconstruction model. Do not use it for real-scale evaluation, precise camera trajectory evaluation, or as real-scale ground truth.

The public dataset does not include .mp4 videos. The source videos belong to Ego4D, EgoDex, and EPIC-KITCHENS and are governed by their respective dataset licenses and access terms, so this project cannot redistribute them. Users who need the videos must apply for and download them through the corresponding official dataset channels and verify their own usage and redistribution rights.

Camera-frame hand estimation remains constrained by pseudo-label quality, and the hand prediction heads have not yet been fine-tuned on high-precision data.

Although the stage-two supervision used to refine camera trajectories is more accurate than data generated by traditional pipelines, residual errors remain relative to ground truth.

The 32-frame training window may lead to accumulated drift when processing long sequences.

Future work will prioritize more accurate, diverse, metric-scale supervision to improve generalization and data-generation quality, bring predictions closer to ground truth, and address trajectory drift in long videos.

For limitations of the released data and pipeline code, see Public Ego pretraining data and Training and optional pipeline reconstruction.

mint/
|-- configs/          Two-stage training recipes and inference settings
|-- data/samples/     Approved Hot3D LeRobot v3 sample
|-- eval/model_effect Original visualization, inference adapters, and benchmarks
|-- docs/             Architecture and operational guides
|   `-- asset/        README and documentation images, animations, and videos
|-- environments/     Full and inference-only dependency specifications
|-- mint/             CLI, inference engine, renderer, and web viewer
|-- model_train/      Training engine, model, losses, and LeRobot loader
|-- ego_pipeline/     Ray scheduling, actors, model backends, trajectory cleanup, manifests, and export
|   `-- preprocessing/  video normalisation and hand filtering, ahead of the pipeline stages
|-- scripts/          Reproducible setup, asset, privacy, and sample tools
`-- third_party/      Redistributable source snapshot; adapted HaWoR is local-only, assets excluded

MINT is made possible by the following research projects, models, and datasets.

• VITRA — MINT's data-processing architecture, egocentric reconstruction workflow, world-space camera/hand annotations, and LeRobot conversion conventions evolved from the VITRA and VITRA-1M data engine.
• LingBot-Map — provides the core model architecture and the upstream source adapted for MINT camera-and-hand training and inference.
• HaWoR — provides monocular hand motion reconstruction, MANO estimation, tracking, and world-space hand-processing components used by the optional data pipeline. Its use remains subject to the upstream non-commercial, no-derivatives license.
• Camera, depth, and tracking research — GeoCalib, MoGe, Mega-SAM, DROID-SLAM, UniDepth, Metric3D, DeepCalib, DINOv2, VGGT, InfiniteVGGT, and PyTorch3D.
• Hand models, simulation, and retargeting — MANO, SMPL-X, MuJoCo, and the Wuji hand description and retargeting components used by the optional Viewer panels.
• Datasets and benchmarks — HOT3D, ARCTIC, Ego4D, EPIC-KITCHENS, and EgoDex. Dataset access and redistribution remain governed by each dataset's own terms.

We thank all upstream authors and maintainers. This acknowledgement does not replace their citation or license requirements; see Third-party notices before use or distribution.

wuji-ego-mint's original code is released under the MIT License. Upstream models, datasets, MANO assets, vendored LingBot-Map files, and optional research backends retain their own licenses. Review Third-party notices before distribution.

The paper is under review; this entry will be replaced with the published reference.

@misc{zhu2026mint,
  title  = {MINT: A Unified Model for World-Space Camera and Hand Motion
            Estimation from Scalable Egocentric Pipeline Supervision},
  author = {Zhu, Zijie and Cai, Weiren and Wang, Yizhou and Yang, Zhenjie and
            Liu, Yide and Chen, Jiahao and He, Guanqi},
  year   = {2026},
  note   = {Manuscript under review}
}

We are glad to share what we have learned and to help move egocentric data forward. If you run into anything — installation, data generation, model training, inference, or the Viewer — please contact Zijie Zhu. WeChat: z3132544408; email: 3132544408@qq.com.