🤗 If you find TrajectoryCrafter useful, please help ⭐ this repo, which is important to Open-Source projects. Thanks!

• [2024-03-10]: 🔥🔥 Update the arXiv preprint.
• [2024-02-23]: Launch the project page.

TrajectoryCrafter can generate high-fidelity novel views from casually captured monocular video, while also supporting highly precise pose control. Below shows some examples:

Input Video                   New Camera Trajectory

git clone --recursive https://github.com/TrajectoryCrafter/TrajectoryCrafter.git
cd TrajectoryCrafter

conda create -n trajcrafter python=3.10
conda activate trajcrafter
pip install -r requirements.txt

Ideally, you can load pretrained models directly from HuggingFace. If you encounter issues connecting to HuggingFace, you can download the pretrained models locally instead. To do so, you can:

1. Download the pretrained models using HuggingFace or using git-lfs

# HuggingFace (recommand)
sh download/download_hf.sh 

# git-lfs (much slower but more stable)
sh download/download_lfs.sh 

2. Change default path of the pretrained models to your local path in inference.py.

Run inference.py using the following script. Please refer to the configuration document to set up inference parameters and camera trajectory.

  sh run.sh

  python gradio_app.py

Our model excels at handling videos with well-defined objects and clear motion, as demonstrated in the demo videos. However, since it is built upon a pretrained video diffusion model, it may struggle with complex cases that go beyond the generation capabilities of the base model.

Many thanks to these open-source projects: CogVideo-Fun, ViewCrafter and DepthCrafter.