Multi-microphone and multi-modal emotion recognition in reverberant environment
O Cohen, G Hazan, S Gannot - arXiv preprint arXiv:2409.09545, 2024 - arxiv.org
O Cohen, G Hazan, S Gannot
arXiv preprint arXiv:2409.09545, 2024•arxiv.orgThis paper presents a Multi-modal Emotion Recognition (MER) system designed to enhance
emotion recognition accuracy in challenging acoustic conditions. Our approach combines a
modified and extended Hierarchical Token-semantic Audio Transformer (HTS-AT) for multi-
channel audio processing with an R (2+ 1) D Convolutional Neural Networks (CNN) model
for video analysis. We evaluate our proposed method on a reverberated version of the
Ryerson audio-visual database of emotional speech and song (RAVDESS) dataset using …
emotion recognition accuracy in challenging acoustic conditions. Our approach combines a
modified and extended Hierarchical Token-semantic Audio Transformer (HTS-AT) for multi-
channel audio processing with an R (2+ 1) D Convolutional Neural Networks (CNN) model
for video analysis. We evaluate our proposed method on a reverberated version of the
Ryerson audio-visual database of emotional speech and song (RAVDESS) dataset using …
This paper presents a Multi-modal Emotion Recognition (MER) system designed to enhance emotion recognition accuracy in challenging acoustic conditions. Our approach combines a modified and extended Hierarchical Token-semantic Audio Transformer (HTS-AT) for multi-channel audio processing with an R(2+1)D Convolutional Neural Networks (CNN) model for video analysis. We evaluate our proposed method on a reverberated version of the Ryerson audio-visual database of emotional speech and song (RAVDESS) dataset using synthetic and real-world Room Impulse Responsess (RIRs). Our results demonstrate that integrating audio and video modalities yields superior performance compared to uni-modal approaches, especially in challenging acoustic conditions. Moreover, we show that the multimodal (audiovisual) approach that utilizes multiple microphones outperforms its single-microphone counterpart.
arxiv.org