Code for the paper accepted by AAAI 2025: "Multi-Grained Query-Guided Set Prediction Network for Grounded Multimodal Named Entity Recognition." (Paper Link).

• python 3.8

• pytorch 1.9.1

• transformers 4.11.3

We suggest utilizing the conda environment. Please make sure you have enough GPU memory no less than 24GB.

conda create -n mqspn python=3.8
conda activate mqspn
pip install -r requirements.txt

We use publicly released datasets Twitter-GMNER and Twitter-FMNERG to train and evaluate our proposed model. You can find datasets' detailed information from their original papers:

• The preprocessed CoNLL format files are provided in this repo data file. For each tweet, the first line is its image id, and the following lines are its textual contents.
• Download each tweet's associated images via this link.
• Following H-Index , we utilize open-sourced VinVL as class-agnostic RPN to identify all the candidate regions. To make MQSPN training efficient, we execute it in an individual process and store pre-detection results under the folder named "Vinvl_detection_path". The features and regions extracted by VinVL can be found at this.
• Please place the corresponding files as the following structure in the data file:

|----datasets\
    |----images_annotation # annotation xml file here
    |----Vinvl_detection_path # candidate regions for twitter15 and twitter17 here
    |----twitter # original raw text with BIO annotations
    |    |----twitter10k_types.json
    |    |----dev.txt
    |    |----samples.txt
    |    |----train.txt
    |    |----test.txt
    |----raw_images # all twitter15 and twitter17 raw images here

python run.py train --config configs/twitter.conf

python run.py eval --config configs/twitter_eval.conf

The datasets we used are from Yu et al. and Wang et al.. Some codes are based on the open-sourced codes PIQN, DQSetGen and MKGFormer. Thanks for their great works!

@inproceedings{tang2025multi,
  title={Multi-grained query-guided set prediction network for grounded multimodal named entity recognition},
  author={Tang, Jielong and Wang, Zhenxing and Gong, Ziyang and Yu, Jianxing and Zhu, Xiangwei and Yin, Jian},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={39},
  number={24},
  pages={25246--25254},
  year={2025}
}