Towards Unified Molecule-Enhanced Pathology Image Representation Learning via Integrating Spatial Transcriptomics

TL;DR: UMPIRE introduces the first large-scale multimodal pre-training framework for pathology images and spatial transcriptomics.

First clone the repo and cd into the directory:

git clone https://github.com/Hanminghao/UMPIRE.git
cd UMPIRE

Then create a conda env and install the dependencies:

conda create -n umpire python=3.9 -y
conda activate umpire
pip install --upgrade pip
pip install -e .

• 03/04/2025: Updated the tokenize step and inference step in the downstream task.

• 03/04/2025: Updated the pre-trained weights with CONCH, Phikon, and UNI.

• Upload the tokenize code.
• Upload the inference code.
• Upload the pretrained model checkpoint.
• Upload the full pre-trained code after the receipt of the article.
• Open source VisTomic-4M after the receipt of the article.

1. Request access to the model weights and example data from Google Drive.

2. Run tokenize_downstream.py to tokenize the downstream data.

3. Run tutorial.ipynb to learn downstream encoding and t-SNE visualization.

If you find our work useful in your research or if you use parts of this code please consider citing our paper:

@article{han2024towards,
  title={Towards Unified Molecule-Enhanced Pathology Image Representation Learning via Integrating Spatial Transcriptomics},
  author={Han, Minghao and Yang, Dingkang and Cheng, Jiabei and Zhang, Xukun and Qu, Linhao and Chen, Zizhi and Zhang, Lihua},
  journal={arXiv preprint arXiv:2412.00651},
  year={2024}
}