LDSA: Adapting Dense Vision-Language Relationships for Multi-label Classification with Partial Label

Official implementation of Adapting Dense Vision-Language Relationships for Multi-label Classification with Partial Label.

LDSA adapts pretrained CLIP knowledge for partial-label multi-label classification. The method freezes the CLIP backbone, learns dense visual adaptation, tunes class-specific language prompts, and uses a language-driven interactive decoder for semantic-aware prediction.

Python 3.10+ and a CUDA-enabled PyTorch installation are recommended.

Core dependencies:

pip install torch torchvision pillow numpy ftfy regex tqdm setuptools randaugment optuna requests colorlog prettytable GitPython typing_extensions tensorboard

Notes:

• Install the PyTorch build that matches your CUDA version from the official PyTorch instructions.
• chinopie is also required and provides the training recipe, dataset slots, distributed samplers, EMA helper, and mAP meter used by this codebase. The command above includes the Python packages imported by the chinopie modules used here.
• wandb and swanlab are only needed if you enable those optional logging backends in chinopie.

The training recipe supports:

• coco2014
• voc2007
• vg

When training starts, coco2014 and voc2007 are downloaded automatically. For vg200, please download the dataset manually in advance.

Partial labels are generated by randomly retaining a proportion of labels specified by label_percent.

Run the LDSA recipe with the dataset and partial-label setting:

dataset_type=coco2014 \
backbone_type=RN101@448 \
label_percent=0.1 \
resolution=14 \
python recipe_mixman10.py

Arguments are passed through environment variables:

• dataset_type: one of coco2014, voc2007, or vg.
• backbone_type: currently RN101@448.
• label_percent: known-label proportion, for example 0.1 for 10%.
• resolution: dense feature resolution, usually 14.

The default recipe trains the model with frozen CLIP visual/text encoders and EMA evaluation.

Checkpoints:

• coco0p1: https://drive.google.com/file/d/11qhD4HZBwn6cv35tH4qb8w85x4dLuXXU/view?usp=sharing
• More ckpts are coming soon.

If this project is useful for your research, please cite: