Learning Intent-aware Camera Motion from Egocentric Video

Boyang Sun^1,* · Jiajie Li^1,* · Yung-Hsu Yang¹ · Chenyangguang Zhang¹ · Tim Engelbracht¹
Sunghwan Hong¹ · Cesar Cadena¹ · Marc Pollefeys^1,2 · Hermann Blum³

¹ETH Zürich   ²Microsoft   ³University of Bonn
^*Equal contribution

Given one RGB image and a free-form intent, LIME predicts where the camera should move next: a relative target pose, and a description of what the new view should reveal.

• 🔧 Setup — Create the environment and download the weights.
• 🚀 Prediction — Predict camera motion from an image and an intent.
• 🖥️ Interactive Demo — Try intents in the browser and see the predicted cameras in 3D.

Clone the repository, then create the environment and download the weights:

git clone https://github.com/cvg/LIME.git && cd LIME

bash scripts/setup_env.sh
conda activate lime

setup_env.sh creates the conda environment lime from environment.yml (Python 3.11, PyTorch 2.8 with CUDA 12.8), checks that PyTorch sees your GPU, and downloads the weights (8.9 GB) from Hugging Face to weights/lime_v1. Run it again to resume an interrupted download. You need conda and an NVIDIA GPU with about 9 GB of memory, with a driver that supports CUDA 12.

conda create -n lime python=3.11 -y && conda activate lime
pip install torch==2.8.0+cu128 torchvision==0.23.0+cu128 \
    --extra-index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt
pip install -e . --no-deps
bash scripts/download_weights.sh

python scripts/predict.py --image examples/living_room.jpg --intent "Find my bike." --seed 0

This prints JSON with the generated observation_gain text, the mean target pose T_mean and the K sampled poses T_samples. Repeat --intent to run several on one image, and write to a file with --out. --samples sets K (default 5) and --steps the Euler steps (default 10).

from lime_camera import LIME

model = LIME.from_pretrained("weights/lime_v1")
prediction = model.predict("examples/living_room.jpg", "Find my bike.", n_samples=5, seed=0)

prediction.observation_gain     # what the new view should reveal
prediction.T_mean               # 4x4 target pose, the mean of the K samples
prediction.T_samples            # (K, 4, 4) sampled target poses

The image is a file path or a PIL image. The text is deterministic, and seed fixes the pose samples. The paper executes the mean of K=5 samples.

Poses are the end camera in the start camera's frame, in metres, with OpenCV axes (x right, y down, z forward), so tz > 0 moves forward. With a known start pose, the target is T_world_end = T_world_start @ T_mean. A prediction takes about 1 s on an RTX 4090, and about 18 s on CPU.

bash scripts/run_demo.sh

Once the model has loaded, it prints where to open the demo:

• On the same machine: open http://localhost:8080.
• From another computer: run ssh -N -L 8080:localhost:8080 user@gpu-machine there, then open http://localhost:8080. VS Code Remote-SSH forwards the port for you.

Type an intent or press a preset, and the predicted cameras appear in the room with the generated text. The example image is a frame from ScanNet++. Its point cloud was estimated from the image alone with MoGe-2, ahead of time and for display only: the model sees nothing but the RGB image and the intent.

To use your own image:

bash scripts/run_demo.sh --image path/to/room.jpg

Without a point cloud you see the cameras only. To add one and preset intents, put room_points.npz and room_command.json (one intent per line) next to the image, as in examples/. If port 8080 is taken, the demo uses the next free one. --host 0.0.0.0 serves your whole network, without a password.

bash scripts/run_demo.sh --scene-root /path/to/scannetpp/data/09c1414f1b --frame frame_003943

• Inference code and the lime_v1 weights.
• Interactive demo.
• Training code.
• Data generation pipeline.

The code is released under the Apache-2.0 license.

The weights are released under CC BY-NC 4.0, for non-commercial use. They are a modified version of Qwen3-VL-4B-Instruct (Apache-2.0); see the model card for both licenses.

If you use any ideas from the paper or code from this repo, please consider citing:

@inproceedings{sun2026lime,
  title={LIME: Learning Intent-aware Camera Motion from Egocentric Video},
  author={Sun, Boyang and Li, Jiajie and Yang, Yung-Hsu and Zhang, Chenyangguang
          and Engelbracht, Tim and Hong, Sunghwan and Cadena, Cesar
          and Pollefeys, Marc and Blum, Hermann},
  booktitle={Conference on Robot Learning (CoRL)},
  year={2026}
}

For questions, feedback, or collaboration, feel free to reach out to Boyang Sun:

📧 boysun@ethz.ch 🌐 boysun045.github.io