高级检索

    基于差异引导的多级融合多模态情感分析网络

    A Multi-Level Fusion Multimodal Sentiment Analysis Network Based on Difference Guidance

    • 摘要: 多模态情感分析旨在整合多种模态信号来理解人类情感表达。现有方法通常将单模态或低阶逆交互特征作为模态特定特征,忽略其中潜在的模态不变特征冲突,导致对于上述2种特征的共同表示出现偏差。此外,样本内和样本间情感强度的差异性往往被忽略,使得对比学习对具有明显情感差异配对的关注不足,进一步加剧了上述偏差。为此,提出一种基于差异引导的多级融合多模态情感分析网络。首先,提出多级交互转换器-编码器来精准捕捉模态间一致性和差异性特征,并结合全局-局部融合网络整合上述特征,从而有效减少模态不变和模态特定特征间的冲突。其次,提出情感差异引导的对比学习方法进一步指导一致性和差异性的学习。该方法能够根据多模态和单模态情感强度差异自动选择模态内和模态间正/负对并自适应动态调整学习权重,从而实现更精细的跨模态交互。在CMU-MOSEI,CMU-MOSI,CH-SIMS数据集上的实验结果表明,所提方法在多数分类指标上比基线方法提升了0.17%~7.58%。

       

      Abstract: Multimodal sentiment analysis aims to integrate multiple modal signals to understand human sentiment expression. Existing methods usually treat unimodal or low-order inverse interaction features as modality-specific features, ignoring the potential conflict of modality-invariant features therein, leading to bias in the co-representation of the above two features. Besides, intra- and inter-sample sentiment intensity differences are often ignored, making contrastive learning pay insufficient focus on pairs with significant sentiment differences, and further exacerbating the above biases. To this end, this paper proposes a multi-level fusion multimodal sentiment analysis network based on difference guidance. Firstly, the multi-level interaction transformer-encoder is proposed to capture inter-modal consistency and differences more accurately, and is combined with the global-local fusion network to integrate the above features, so as to effectively reduce the conflicts between modality-invariant and modality-specific features. Secondly, the contrastive learning guided by sentiment differences is proposed to further guide the learning of consistency and differences. The method can automatically select intra- and inter-modal positive/negative pairs according to the differences in the intensity of multimodal and unimodal sentiments, and adaptively and dynamically adjust the learning weights for more refined cross-modal interactions. Experimental results on the CMU-MOSEI, CMU-MOSI, and CH-SIMS datasets show that the proposed method more effectively captures the inter-modal consistent and differential information, and improves over the baseline methods on most classification metrics by 0.17%-7.58%.

       

    /

    返回文章
    返回