Abstract:
Multimodal sentiment analysis aims to integrate multiple modal signals to understand human sentiment expression. Existing methods usually treat unimodal or low-order inverse interaction features as modality-specific features, ignoring the potential conflict of modality-invariant features therein, leading to bias in the co-representation of the above two features. Besides, intra- and inter-sample sentiment intensity differences are often ignored, making contrastive learning pay insufficient focus on pairs with significant sentiment differences, and further exacerbating the above biases. To this end, this paper proposes a multi-level fusion multimodal sentiment analysis network based on difference guidance. Firstly, the multi-level interaction transformer-encoder is proposed to capture inter-modal consistency and differences more accurately, and is combined with the global-local fusion network to integrate the above features, so as to effectively reduce the conflicts between modality-invariant and modality-specific features. Secondly, the contrastive learning guided by sentiment differences is proposed to further guide the learning of consistency and differences. The method can automatically select intra- and inter-modal positive/negative pairs according to the differences in the intensity of multimodal and unimodal sentiments, and adaptively and dynamically adjust the learning weights for more refined cross-modal interactions. Experimental results on the CMU-MOSEI, CMU-MOSI, and CH-SIMS datasets show that the proposed method more effectively captures the inter-modal consistent and differential information, and improves over the baseline methods on most classification metrics by 0.17%-7.58%.