Although Mixup-based augmentation improves generalization in unimodal tasks, its direct application to MSA introduces critical challenges: random mixing often amplifies label ambiguity and semantic inconsistency due to the lack of emotion-aware mixing mechanisms. To overcome these issues, we propose MS-Mix, an adaptive, emotion-sensitive augmentation framework that automatically optimizes sample mixing in multimodal settings.
We use the M-SENA framework ("https://github.com/thuiar/MMSA") to evaluate MSA models. It is a unified framework for multimodal sentiment analysis. Please install it before proceeding.
Run pip install MMSA in your python virtual environment.
We conducted extensive experiments on the CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets. You can download raw videos, feature files, and label files via the following links: Dataset
After install MMSA and download dataset, replace the data link in code (/config_regression.json).
Choose the mixup variants or not.
Create a new .py file in the project and run it:
import os
os.environ['TF_ENABLE_ONEDNN_OPTS'] = '0'
from MMSA import MMSA_run
def main():
MMSA_run('almt', 'mosei', seeds=[1111], gpu_ids=[0])
if __name__ == '__main__':
main()