Localization-Grounded Supervision: Revisiting Vanilla SFT of Large Vision-Language Models for Medical Image Analysis

Localization-Grounded Supervision (LGS) is a simple and architecture-agnostic supervision framework for LVLMs finetuning. LGS introduces explicit localization signals into standard SFT targets, enabling LVLMs to better leverage pretrained spatial alignment and establish fine-grained vision-language semantic alignment.

LGS supports LVLM adaptation across model architectures and parameter scales. Beyond improving model performance, LGS also provides more interpretable spatial grounding for LVLMs.

# 1. clone and navigate
git clone https://github.com/MSIIP/LGS.git
cd LGS

# 2. create a conda environment, activate it and install packages
conda create -n lgs python=3.11
conda activate lgs
pip install 'ms-swift[all]' -U

Take LUNA16 as an example:

# generate 2D samples from 3D segmentation masks
python src/dataset/luna16_lgs.py

# train
bash scripts/train_and_eval/sft_luna16_lgs.sh

# eval
bash scripts/train_and_eval/eval_luna16_lgs.sh

python src/others/draw_depass.py

TBD

We would like to express our gratitude to the following resources:

• LUNA16 - Chest CT dataset with 3D lung nodule segmentation masks.
• CrossMoDA2021 - MRI dataset with 3D vestibular schwannoma segmentation masks.
• CT-RATE - Chest CT dataset with radiology reports.
• RadGenome-Chest CT - Chest CT dataset with grounded radiology reports.
• DePass - Attribution framework for analyzing transformer-based models.