Maximilian Fehrentz*1,2,4 · Nicolas Stellwag*2 · Robert Wiebe2 · Nicole Thorisch2 · Fabian Grob2 · Patrick Remerscheid2 · Ken-Joel Simmoteit2 · Benjamin D. Killeen1,4 · Christian Heiliger3 · Nassir Navab1,4
1Computer Aided Medical Procedures, TU Munich
2TUM.ai
3University Hospital of Ludwig Maximilian University (LMU) Munich
4Munich Center for Machine Learning
Project Page | Paper | Dataset
Official implementation of "A 4D Representation for Training-Free Agentic Reasoning from Monocular Laparascopic Video".
Install the pixi package manger using the official instructions.
Note: This installs pixi into the user's home directory and python environments will later be placed in this project directory. So there should be no issues on compute clusters.
git clone --recurse-submodules git@github.com:tum-ai/surg4d.git# install conda and pypi packages
pixi install
# install custom packages and download checkpoints
pixi run setup
# optional: test importing key packages
pixi run test-installDownload the CholecSeg8k dataset and our annotations:
pixi run download-cholecseg8k
pixi run download-benchmark-annotationsNote: If you want to annotate your own queries, check out our annotation tool repository.
The pipeline is based on the configuration system hydra. Config files can be found in conf/. Check out the hydra getting started guide.
Run the pipeline using the following scripts:
# train segmentation model and create masks
pixi run python segment.py
# preprocess frames, masks, and annotations
pixi run python preprocess.py
# predict depth and pose
pixi run python extract_geometry.py
# create temporally consistent instances
pixi run python track_objects.py
# build 4d scene graphs
pixi run python extract_graphs.py
# predict benchmark queries
pixi run python evaluate_benchmark.py
# compute benchmark metrics
pixi run python compute_metrics.pyOr the whole pipeline, including all ablations:
bash ablate_all.shFor questions, please open an issue or contact maximilian.fehrentz@tum.de.