Hierarchical modality reconstruction and transformer interaction with adaptive fusion for multimodal sentiment analysis with missing modalities

Multimodal sentiment analysis (MSA) aims to recognize human emotions by integrating information from multiple sources, such as text, audio, and visual modalities. In recent years, this task has faced two major challenges in real-world applications. First, uncertain modality missingness leads to incomplete input information.

Second, insufficient cross-modal interaction limits the effectiveness of multimodal fusion. Existing methods often treat reconstructed modalities and available modalities equally and lack adequate mechanisms for cross-modal interaction, which makes them prone to reconstruction errors and weakens the quality of multimodal fusion. To address these issues, we propose a novel framework for multimodal sentiment analysis with missing modalities, named AMF-MMSA (Attention Mechanisms and Missingness-Aware Fusion for Multimodal Sentiment Analysis with Missing Modalities).

Under the complete-modality setting, AMF-MMSA first employs a self-attention mechanism to enhance modality-specific feature representations. When modality missing occurs, based on the enhanced features, we further design a modality reconstruction module (MR) guided by inter-modal affective consistency, while considering both random single-modality missing and multi-modality missing scenarios. Subsequently, all modality features, including reconstructed ones, are fed into a Transformer interaction module (TIM) to enable sufficient cross-modal interaction.

Finally, a missingness-aware fusion module (MAF) explicitly distinguishes available modalities from reconstructed modalities and adaptively down-weights potentially low-quality reconstructed modalities, thereby achieving more robust multimodal fusion. Experimental results on three public datasets, CMU-MOSI, CMU-MOSEI, and CH-SIMS, demonstrate that under missing-rate settings of 0.0-0.5, AMF-MMSA significantly outperforms existing baseline methods across all evaluation metrics. This work is partially supported by the National Natural Science Foundation of China (62166039); the Natural Science Foundation of the Xinjiang Uygur Autonomous Region (2026D01C86); and Sub-project 2 of the Key Research and Development Program of the Xinjiang Uygur Autonomous Region (2025B01015-2).

College of Information Management, Xinjiang University of Finance and Economics, Urumqi, Xinjiang, 830012, China School of intelligent systems engineering, Sun Yat-sen University, Shenzhen, 528406, Guangdong, China The authors declare no competing interests. Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.