Authors
Zhixiang Wei¹², Guangting Wang² et al.
¹ University of Science and Technology of China
² WeChat Vision, Tencent Inc.

• 🏭 Efficient Data Generation Pipeline
  Multi-grained annotation pipeline using Large Vision-Language Models (LVLMs)
• 🗂️ High-Quality Image-Text Datasets
  Generated by state-of-the-art LVLMs with positive/negative examples and rich text descriptions:
  • VLM-1B: Billion-scale dataset
  • VLM-150M: Curated 150M samples
• 🧠 HQ-CLIP Training Framework
  Novel CLIP training paradigm extending contrastive learning with:
  • Negative description supervision
  • Short tag augmentation

Recaption Model: Qwen2VL

1. (Optional) Download CommonPool Foundation Datasets
   Access CommonPool Large and XLarge versions via:
   CommonPool GitHub Repository

2. Acquire DFN Base Datasets
   Download DFN Large and XLarge from:
   DFN Hugging Face Datasets

3. Download HQ-CLIP Datasets
   Obtain our enhanced datasets:

   • VLM-150M
   • VLM-1B

Each JSON entry in VLM-150M and VLM-1B corresponds directly to a DFN dataset UID through matching filenames. To utilize our enhanced annotations:

• Option 1: Direct Caption Replacement
  Substitute the original DFN captions with our generated annotations

• Option 2: Dynamic Data Loading
  Modify the Open CLIP dataloader to load our annotations during training runtime

🔜 Detailed implementation guidance will be published in future releases.

Our uploaded weights are compatible with both open_clip and huggingface transformers.

import open_clip

Initialize model with transforms

model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms(
    'hf-hub:zhixiangwei/vlm150m-hqclip-large-vitb16'
)
tokenizer = open_clip.get_tokenizer(
    'hf-hub:zhixiangwei/vlm150m-hqclip-large-vitb16'
)

from transformers import AutoModel

Load model directly from hub

model = AutoModel.from_pretrained(
    'zhixiangwei/vlm150m-hqclip-large-vitb16'
)

@misc{hqclip,
      title={HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models}, 
      author={Zhixiang Wei and Guangting Wang and Xiaoxiao Ma and Ke Mei and Huaian Chen and Yi Jin and Fengyun Rao},
      year={2025},
      eprint={2507.22431},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2507.22431}, 
}

These works have greatly inspired us, providing us with codebases, data, and support. We thank their authors!

• open_clip
• clip_benchmark
• datacomp
• DFN
• What If