Hierarchical Alleviation of Heterogeneity in Multimodal Sentiment Analysis

Inspired by the fusion of objects in the natural world, we conceptualize multimodal fusion as a continuous process, dividing the entire procedure into three steps. We design MCL-MCF. MCF simulates the continuous fusion process, while MCL, through multilevel alleviation of heterogeneity, assists MCF in achieving multilevel fusion. Their collaborative operation yields optimal fusion results.

1.Download the CMU-MOSI and CMU-MOSEI dataset from Google Drive or Baidu Disk (extraction code: g3m2)
Place them under the folder MCL-MCF/datasets

2. environment

conda env create -f environment.yml
conda activate MCL

3. starting

cd ~/autodl-tmp/MCL-MCF/MCL-MCF

python -m src.main

python -m src.main --dataset mosi

python -m src.main --dataset mosei

cd ~/autodl-tmp/MCL-MCF/MCL-MCF python auto_search.py

We are grateful for the open source baseline of MMIM. We built MCL-MCF on it. For configuration and dataset related open source, please refer to MMIM. Please cite this paper if you find our work useful for your research:

@inproceedings{han2021improving,
  title={Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis},
  author={Han, Wei and Chen, Hui and Poria, Soujanya},
  booktitle={Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  pages={9180--9192},
  year={2021}
}