Preprocessing turns a raw Project Aria recording (sample.vrs + MPS SLAM & hand tracking) into a structured, per-frame dataset with 3D hand/object tracking, ready to train HumanEgo policies. This doc covers (1) how to download a sample and run the pipeline, (2) the pipeline architecture, and (3) every file it produces and what it means.

The data lives on the public HuggingFace dataset Leo-TX/HumanEgo — no token required.

pip install huggingface_hub        # for downloading
# and follow the repo README to install the full preprocessing environment

Download data (scripts/download_data.py) — pick a task and how many recordings:

# 1 serve_bread recording (default), WITH precomputed output (~2 GB); all_data.tar auto-extracted
python scripts/download_data.py

# the first 20 serve_bread recordings, with precomputed output
python scripts/download_data.py --task serve_bread --num 20

# INPUT ONLY (~0.6 GB/recording) — to run the pipeline yourself
python scripts/download_data.py --task serve_bread --num 1 --input-only

Run the pipeline:

python -m preprocess.Preprocess \
    --mps_path ./data/serve_bread/aria/mps_serve_bread_000_vrs --task serve_bread

Everything is written to …/mps_serve_bread_000_vrs/preprocess/. On an RTX 4090 a ~33 s / 1000-frame recording takes ~20 min end-to-end.

from huggingface_hub import snapshot_download
# input only
snapshot_download(
    repo_id="Leo-TX/HumanEgo", repo_type="dataset", local_dir="./data",
    allow_patterns=["serve_bread/aria/mps_serve_bread_000_vrs/*"],
    ignore_patterns=["serve_bread/aria/mps_serve_bread_000_vrs/preprocess/*"],
)

mps_serve_bread_000_vrs/
├── sample.vrs              # raw Aria recording                     ┐
├── slam/                   # MPS SLAM (trajectory + semidense points)│ INPUT
├── hand_tracking/          # MPS hand tracking (hand_tracking_results.csv)
├── else/                   # VRS metadata / health checks           ┘
└── preprocess/             # ← everything below is GENERATED by the pipeline
    ├── <top-level files>   # per-stage results + diagnostic plots
    ├── vis/                # visualization clips (mp4 + gif)
    ├── cfg/                # snapshot of the configs used (auto-regenerated)
    └── all_data/<idx>/     # per-frame data (rgb, masks, poses, training_data.json)

preprocess.Preprocess runs these stages in order (entry point: preprocess/Preprocess.py):

The released default runs hand tracking via aria_mps only — that is all the Quick Start needs. The mediapipe / wilor / hamer methods are an ablation, off by default. Reproducing it takes three steps: install the extra packages (SKIP_HAND=0 bash setup.sh), list the methods in cfg/preprocess/base/Preprocess.yaml → hand_tracking_methods, and un-comment self.preprocess_hands() in preprocess/Preprocess.py (it is commented out by default).

<idx> is the zero-padded frame index. Image-name convention: W=with, Wo=without, Arm/Obj=arm/object, Kpts=keypoints overlaid.

The final per-frame dataloader target produced by DatasetGen:

{
  "metadata": {
    "idx", "ts",                 // frame index, timestamp
    "w", "h", "fps",             // image size, frame rate
    "k":   [3×3 intrinsics],
    "c2w": [4×4 camera-to-world],
    "anchor_key": "obj1",        // object designated as the world origin (object-centric frame)
    "is_finished": <float>,      // 1.0 on the final frames of the demonstration
    "world_transforms": { ... }  // reference-frame transforms (cam0, virtual_static_anchor)
  },
  "obs": {                       // image / mask paths for this frame
    "rgb_path", "mask_arm_path",
    "mask_obj_path",             // the COMBINED arm+object mask; per-object masks are mask_obj1.png / mask_obj2.png
    "rgb_WoArm_path", "rgb_WArmObjKpts_path", "rgb_WoArm_WArmObjKpts_path", ...
  },
  "entities": {
    "hands":   { "right": { "T_hand_to_world": [4×4 SE(3)], "grasp": <float> } },
    "objects": { "obj1": { "T_obj_to_world": [4×4], "is_dynamic": <bool> },
                 "obj2": { ... } },
    // ablation only (empty unless mediapipe/wilor/hamer were enabled):
    "hands_mediapipe": {...}, "hands_wilor": {...}, "hands_hamer": {...}
  }
}

A snapshot of the merged YAML configs used for this run, one file per module (AriaCam.yaml, DINOSAM.yaml, …). Auto-generated every run and fully regenerable — it documents exactly which settings produced the outputs.

Configs live under cfg/preprocess/ and are compiled per run (load_cfg_dynamic_task):

• cfg/preprocess/base/Preprocess.yaml — the master config. It points to each module's YAML (AriaCam_path, DINOSAM_path, …) and sets hand_tracking_methods.
• cfg/preprocess/base/<Module>.yaml — per-module defaults, one file per pipeline stage (AriaCam, AriaHands, AriaSlam, AriaPhases, DINOSAM, KptsSelector, CoTracker, CamTriangulator, Lama, VisualKpts, DatasetGen). Every field is documented inline in these files — read them for the full set of knobs and their defaults.
• cfg/preprocess/tasks/<task>.yaml — a small per-task override selected by --task <task>, deep-merged over the base so you only specify what differs for your task.

python -m preprocess.Preprocess --mps_path <recording_dir> --task <task> \
    [--cfg_path cfg/preprocess/base/Preprocess.yaml] [--no-video] [--no-gif]

If cfg/preprocess/tasks/<task>.yaml is missing, preprocessing still runs but uses the base defaults — whose dinosam_prompt is a placeholder ("a pretty cat ."), so it won't detect your objects. A task file is therefore effectively required.

Each top-level key targets the module of the same name; the values are deep-merged onto that module's base YAML. The fields that typically vary per task:

For every other tunable field and its default, see the matching cfg/preprocess/base/<Module>.yaml.

1. Create cfg/preprocess/tasks/<your_task>.yaml.
2. Set DINOSAM.dinosam_prompt for each object you manipulate (obj1, obj2, …) plus the arm prompt.
3. Set AriaHands.hand_selection (and grasp_threshold if needed), and pick a CamTriangulator.pose_method per object.
4. Run python -m preprocess.Preprocess --mps_path <recording_dir> --task <your_task>.
5. Verify: check …/preprocess/dinosam_mask_obj*.png to confirm the prompts segment the right objects, and the …/preprocess/cfg/ snapshot to confirm your overrides were applied; iterate on the prompts/thresholds as needed.

Example — cfg/preprocess/tasks/serve_bread.yaml:

AriaHands:
  hand_selection: "right"            # serve_bread is one-handed (right)

DINOSAM:
  dinosam_prompt:
    obj1: "piece of bread ."
    obj2: "a plate ."
    arm:  "human arms . human hands ."

CamTriangulator:
  pose_method:
    obj1: "pca1"
    obj2: "pca1"
    default: "pca1"

# an entire task (all serve_bread recordings), with precomputed output
python scripts/download_data.py --task serve_bread --num all

# the whole dataset (serve_bread + water_flowers) — large
python scripts/download_data.py --task all --num all

# inputs only, to reprocess everything yourself
python scripts/download_data.py --task all --num all --input-only

all_data.tar is just the per-frame files packed into one archive (so the dataset stays at a few dozen files per recording instead of hundreds of thousands). It is auto-extracted on download and is fully regenerable by running preprocessing, so you only need it to get the precomputed per-frame results without running the pipeline.