A Multimodal Cross-Domain Sentiment Analysis Algorithm Based on Feature Disentanglement Meta-Optimization

Jia Xibin; Li Chen; Wang Luo; Zhang Muchen; Liu Xiaojian; Zhang Yangyang; Wen Jiakai

doi:10.7544/issn1000-1239.202440624

Journal of Computer Research and Development > 2025 > Uncorrected proof > DOI: 10.7544/issn1000-1239.202440624 CSTR: 32373.14.issn1000-1239.202440624

Jia Xibin, Li Chen, Wang Luo, Zhang Muchen, Liu Xiaojian, Zhang Yangyang, Wen Jiakai. A Multimodal Cross-Domain Sentiment Analysis Algorithm Based on Feature Disentanglement Meta-Optimization[J]. Journal of Computer Research and Development. DOI: 10.7544/issn1000-1239.202440624

Citation:

PDF (1242 KB)

A Multimodal Cross-Domain Sentiment Analysis Algorithm Based on Feature Disentanglement Meta-Optimization

1.
College of Computer, Beijing University of Technology, Beijing 100124
2.
Beijing Institute of Artificial Intelligence(Beijing University of Technology), Beijing 100124
3.
China Electronics Standardization Institute, Beijing 100007
4.
Guangxi Daring Technology Co., Ltd, Nanning 530006

Funds: This work was supported by the General Program of the National Natural Science Foundation of China (62476015) and the Cultivation Program for Excellent Achievements in Graduate Education and Teaching at Beijing University of Technology (GER202316).

More Information

Author Bio:
Jia Xibin: born in 1969. Professor. Member of distinguished CCF(29295D). Her main research interests include visual computing, multi-modality deep learning, affection computing, intelligent medical image, behaviour understanding and computing

Li Chen: born in 1998. Master. Her main research interests include multimodal sentiment and transfer learning

Wang Luo: born in 1990. PhD. Lecturer. Member of CCF(P4015M). His main research interests include computer vision and deep learning

Zhang Muchen: born in 2002. Master candidate. Her main research interests include multi-modality deep learning and computer vision

Liu Xiaojian: born in 1989. PhD. Senior engineer. Member of CCF(P2866M). His main research interests include software engineering and cybersecurity

Zhang Yangyang: born in 1976. Master. Professorial senior engineer. Member of CCF(E9145M). Her main research interests include Software and Systems Engineering, Standardization of IT

Wen Jiakai: born in 1979. Bachelor. Senior engineer. Member of CCF(C2990M). His main research interests include machine translation and big data
Received Date: July 19, 2024
Revised Date: January 14, 2025
Accepted Date: March 02, 2025
Available Online: March 02, 2025

Graphical Abstract

Abstract

Abstract

Multimodal sentiment analysis aims to utilize the multimodal customer comments and other data to identify users' sentimental tendencies. To realize cross-domain application with the domain bias, commonly used solutions are unsupervised domain adaptation methods. Nevertheless, this type of solutions focuses on the extraction of domain-invariant features, and it neglects the significance of domain-specific features at the target domain. Thus, a meta-optimization based domain-invariant and domain-specific feature disentanglement network is proposed. First, by embedding adapters into the pre-trained large model with fine-tuning fitting, the image-text fused sentiment feature encoder is accordingly constructed. Then, a feature disentanglement module is constructed on the basis of the factorization operation, which utilizes domain adversary and domain classification, together with collaborative independence constraints, respectively, to achieve knowledge-transferable domain-invariant feature embedding while extracting the domain-specific features to enhance the performance of sentiment classification at the target domain. To ensure the consistency of the overall optimization tendency for feature disentanglement and sentiment classification, a meta-learning-based meta-optimization training strategy is put forward to synergistically optimize the sentiment analysis network. Comparative experiments on bidirectional sentiment transfer tasks constructed by MVSA and Yelp datasets demonstrate that compared to other advanced image-text sentiment transfer algorithms, the proposed algorithm achieves superior performance on bidirectional sentiment transfer tasks in terms of three consensus metrics: Precision, Recall and F1 score.

FullText(HTML)

References (33)

References

[1]	Das R, Singh T D. Multimodal sentiment analysis: A survey of methods, trends, and challenges[J]. ACM Computing Surveys, 2023, 55(13s): 1−38
[2]	Chan J Y L, Bea K T, Leow S M H, et al. State of the art: A review of sentiment analysis based on sequential transfer learning[J]. Artificial Intelligence Review, 2023, 56(1): 749−780 doi: 10.1007/s10462-022-10183-8
[3]	Singhal P,Walambe R,Ramanna S,et al. Domain adaptation:Challenges,methods,datasets,and applications[J]. IEEE Access,2023,11:6973-7020(没有期
[4]	Azuma C, Ito T, Shimobaba T. Adversarial domain adaptation using contrastive learning[J]. Engineering Applications of Artificial Intelligence, 2023, 123: 106394 doi: 10.1016/j.engappai.2023.106394
[5]	Zhou Qianyu, Gu Qiqi, Pang Jiangmiao, et al. Self-adversarial disentangling for specific domain adaptation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(7): 8954−8968
[6]	Li Jingjing, Chen Erpeng, Ding Zhengming, et al. Maximum density divergence for domain adaptation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 43(11): 3918−3930
[7]	Zhu Yongchun, Zhuang Fuzhen, Wang Jindong, et al. Deep subdomain adaptation network for image classification[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 32(4): 1713−1722
[8]	Zhao Han, Des Combes R T, Zhang Kun, et al. On learning invariant representations for domain adaptation[C]//Proc of the 36th Int Conf on Machine Learning. New York: PMLR, 2019: 7523−7532
[9]	Johansson F D, Sontag D, Ranganath R. Support and invertibility in domain-invariant representations[C]//Proc of the 22nd Int Conf on Artificial Intelligence and Statistics. New York: PMLR, 2019: 527−536
[10]	Zadeh A, Chen Minghai, Poria S, et al. Tensor fusion network for multimodal sentiment analysis[C]//Proc of the 2017 Conf on Empirical Methods in Natural Language Processing. New York: PMLR, 2017: 1103−1114
[11]	Truong Q T, Lauw H W. Vistanet: Visual aspect attention network for multimodal sentiment analysis[C]//Proc of the 33rd Association for the Advancement of Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 305−312
[12]	Gui Tao, Zhu Liang, Zhang Qi, et al. Cooperative multimodal approach to depression detection in twitter[C]//Proc of the 33rd Association for the Advancement of Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 110−117
[13]	Ling Yan, Yu Jianfei, Xia Rui. Vision-language pre-Training for multimodal aspect-based sentiment analysis[C]//Proc of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Stroudsburg, PA: ACL, 2022: 2149−2159
[14]	Ganin Y, Ustinova E, Ajakan H, et al. Domain-adversarial training of neural networks[J]. Journal of Machine Learning Research, 2016, 17(59): 1−35
[15]	Jia Xibin, Li Chen, Zeng Meng, et al. An improved unified domain adversarial category-wise alignment network for unsupervised cross-domain sentiment classification[J]. Engineering Applications of Artificial Intelligence, 2023, 126: 107108 doi: 10.1016/j.engappai.2023.107108
[16]	黄学坚,马廷淮,王根生. 基于样本内外协同表示和自适应融合的多模态学习方法[J]. 计算机研究与发展,2024,61(5):1310−1324 doi: 10.7544/issn1000-1239.202330722 Huang Xuejian, Ma Tinghuai, Wang Gensheng. Multimodal Learning Method Based on Intra- and Inter-Sample Cooperative Representation and Adaptive Fusion[J]. Journal of Computer Research and Development, 2024, 61(5): 1310−1324 (in Chinese) doi: 10.7544/issn1000-1239.202330722
[17]	Qi Fan, Yang Xiaoshan, Xu Changsheng. A unified framework for multimodal domain adaptation[C]//Proc of the 26th ACM Int Conf on Multimedia. New York, NY: ACM, 2018: 429−437
[18]	Ma Xinhong, Zhang Tianzhu, Xu Changsheng. Deep multi-modality adversarial networks for unsupervised domain adaptation[J]. IEEE Transactions on Multimedia, 2019, 21(9): 2419−2431 doi: 10.1109/TMM.2019.2902100
[19]	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proc of the 31st Int Conf on Neural Information Processing Systems. New York: ACM, 2017: 6000−6010
[20]	Li Junnan, Selvaraju R, Gotmare A, et al. Align before fuse: Vision and language representation learning with momentum distillation[J]. Advances in Neural Information Processing Systems, 2021, 34: 9694−9705
[21]	Chen Shoufa, Ge Chongjian, Tong Zhan, et al. Adaptformer: Adapting vision transformers for scalable visual recognition[J]. Advances in Neural Information Processing Systems, 2022, 35: 16664−16678
[22]	Li Ya, Tian Xinmei, Gong Mingming, et al. Deep domain generalization via conditional invariant adversarial networks[C]//Proc of the 2018 European Conf on Computer Vision(ECCV). Berlin: Springer, 2018: 624−639
[23]	Bui M H, Tran T, Tran A, et al. Exploiting domain-specific features to enhance domain generalization[J]. Advances in Neural Information Processing Systems, 2021, 34: 21189−21201
[24]	Wei Guoqiang, Lan Cuiling, Zeng Wenjun, et al. Metaalign: Coordinating domain alignment and classification for unsupervised domain adaptation[C]//Proc of the IEEE/CVF Conf on Computer Vision and Pattern Recognition(CVPR). Piscataway, NJ: IEEE, 2021: 16643−16653
[25]	Niu Teng, Zhu Shiai, Pang Lei, et al. Sentiment analysis on multi-view social data[C]//Proc of the 22nd Int Conf on Multimedia Modeling. Berlin: Springer, 2016: 15−27
[26]	刘琦玮,李俊,顾蓓蓓,等. TSAIE:图像增强文本的多模态情感分析模型[J]. 数据与计算发展前沿,2022,4(3):131−140 Liu Qiwei, Li Jun, Gu Beibei, et al. TSAIE: Text sentiment analysis model based on image enhancement[J]. Frontiers of Data & Computing, 2022, 4(3): 131−140 (in Chinese)
[27]	Zhang Yuhao, Zhang Ying, Guo Wenya, et al. Learning disentangled representation for multimodal cross-domain sentiment analysis[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 34(10): 7956−7966
[28]	Baltrušaitis T, Ahuja C, Morency L P. Multimodal machine learning: A survey and taxonomy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41(2): 423−443
[29]	Dosovitskiy A. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv preprint, arXiv: 2010.11929, 2020
[30]	Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint, arXiv: 1810.04805, 2018
[31]	Li Junnan, Li Dongxu, Xiong Caiming, et al. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation[C]//Proc of the 39th Int Conf on Machine Learning. New York: PMLR, 2022: 12888−12900
[32]	Zhu Tong, Li Leida, Yang Jufeng, et al. Multimodal sentiment analysis with image-text interaction network[J]. IEEE Transactions on Multimedia, 2022, 25: 3375−3385
[33]	Li Jingzhe, Wang Chengji, Luo Zhiming, et al. Modality-dependent sentiments exploring for multi-modal sentiment classification[C]//Proc of the 2024 IEEE Int Conf on Acoustics, Speech and Signal Processing (ICASSP). Piscataway, NJ: IEEE, 2024: 7930−7934

[1]	Gao Yujia, Wang Pengfei, Liu Liang, Ma Huadong. Personalized Federated Learning Method Based on Attention-Enhanced Meta-Learning Network[J]. Journal of Computer Research and Development, 2024, 61(1): 196-208. DOI: 10.7544/issn1000-1239.202220922
[2]	Li Gengsong, Liu Yi, Zheng Qibin, Li Xiang, Liu Kun, Qin Wei, Wang Qiang, Yang Changhong. Algorithm Selection Method Based on Multi-Objective Hybrid Ant Lion Optimizer[J]. Journal of Computer Research and Development, 2023, 60(7): 1533-1550. DOI: 10.7544/issn1000-1239.202220769
[3]	Ren Jiarui, Zhang Haiyan, Zhu Menghan, Ma Bo. Embedding Learning Algorithm for Heterogeneous Network Based on Meta-Graph Convolution[J]. Journal of Computer Research and Development, 2022, 59(8): 1683-1693. DOI: 10.7544/issn1000-1239.20220063
[4]	Jia Xibin, Zeng Meng, Mi Qing, Hu Yongli. Domain Alignment Adversarial Unsupervised Cross-Domain Text Sentiment Analysis Algorithm[J]. Journal of Computer Research and Development, 2022, 59(6): 1255-1270. DOI: 10.7544/issn1000-1239.20210039
[5]	Wang Yuan, Chen Ming, Xing Lining, Wu Yahui, Ma Wubin, Zhao Hong. Deep Intelligent Ant Colony Optimization for Solving Travelling Salesman Problem[J]. Journal of Computer Research and Development, 2021, 58(8): 1586-1598. DOI: 10.7544/issn1000-1239.2021.20210320
[6]	Song Rui, LiTong, Dong Xin, Ding Zhiming. A User Requirements Preference Analysis Method of Mobile Applications Based on Meta-Path Embedding[J]. Journal of Computer Research and Development, 2021, 58(4): 749-762. DOI: 10.7544/issn1000-1239.2021.20200737
[7]	Wu Yao, Shen Derong, Kou Yue, Nie Tiezheng, Yu Ge. Heterogeneous Information Networks Embedding Based on Multiple Meta-Graph Fusion[J]. Journal of Computer Research and Development, 2020, 57(9): 1928-1938. DOI: 10.7544/issn1000-1239.2020.20190553
[8]	Jia Xibin, Jin Ya, Chen Juncheng. Domain Alignment Based on Multi-Viewpoint Domain-Shared Feature for Cross-Domain Sentiment Classification[J]. Journal of Computer Research and Development, 2018, 55(11): 2439-2451. DOI: 10.7544/issn1000-1239.2018.20170496
[9]	Sun Jing, Yu Hongliang, and Zheng Weimin. Index of Meta-Data Set of the Similar Files for Inline De-Duplication in Distributed Storage Systems[J]. Journal of Computer Research and Development, 2013, 50(1): 197-205.
[10]	Lu Han, Cao Cungen, Wang Shi. Implementation of a Meta-Property Based Quantity Attribute-Value Extraction System[J]. Journal of Computer Research and Development, 2010, 47(10): 1741-1748.

Cited By

Cited by

Periodical cited type(18)

1.	徐宁，李静秋，王岚君，刘安安. 时序特性引导下的谣言事件检测方法评测. 南京大学学报(自然科学). 2025(01): 71-82 .
2.	崔蒙蒙，刘井平，阮彤，宋雨秋，杜渂. 基于双重多视角表示的目标级隐性情感分类. 计算机工程. 2024(01): 79-90 .
3.	张乐怡，周怡洁，俞定国，闫燕勤. 媒介变迁下的谣言传播研究. 新媒体研究. 2024(14): 12-16 .
4.	王世雄，吴泽政. 基于异质信息网络表征学习的微博虚假信息甄别研究. 情报杂志. 2024(12): 152-160 .
5.	陈雄逸，许力，张欣欣，尤玮婧. 社交网络基于意见领袖的谣言抑制方案. 信息安全研究. 2023(01): 57-65 .
6.	张欣欣，许力，徐振宇 . 基于网络模体的移动社会网络信息可控传播方法. 电子与信息学报. 2023(02): 635-643 .
7.	杨晓晖，王卫宾. 基于门控图神经网络的谣言检测模型. 燕山大学学报. 2023(01): 73-81 .
8.	孙书魁，范菁，李占稳，曲金帅，路佩东. 人工智能在新型冠状病毒肺炎中的研究综述. 计算机工程与应用. 2023(05): 28-39 .
9.	陈卓敏，王莉，朱小飞，王子康. 基于对抗图增强对比学习的虚假新闻检测. 中文信息学报. 2023(06): 137-146 .
10.	鲁贻锦，吴蕾. 基于大数据驱动技术的媒体风险感知模型研究. 佳木斯大学学报(自然科学版). 2023(06): 52-56 .
11.	许云红，崔乐靖，朱南丽，郑娜娜. 社交媒体用户谣言传播行为的影响因素研究综述. 新媒体研究. 2023(24): 14-17+33 .
12.	龙小农，靳旭鹏. 新冠疫情、信息疫情与政治疫情的互动关系及作用机制. 现代传播(中国传媒大学学报). 2022(02): 66-76 .
13.	杨秀璋，刘建义，任天舒，宋籍文，武帅，姜婧怡，陈登建，周既松，李娜. 基于改进LDA-CNN-BiLSTM模型的社交媒体情感分析研究. 现代计算机. 2022(02): 29-36 .
14.	张放，范琳琅. 公共危机中社交媒体辟谣信息采纳的关键要素探究——基于新冠疫情微博辟谣文本的计算分析. 新闻界. 2022(10): 75-85 .
15.	朱梦蝶，付少雄，郑德俊，李杨. 文献视角下的社交媒体健康谣言研究：特征、传播与治理. 图书情报知识. 2022(05): 131-143 .
16.	肖喜珠，杨闻远，高慧敏，高世奇，郭书恒，路思玲，聂欣政，任书漫，王一民，温馨. “后真相”时代的风险感知与反击：青年社交媒体用户信息行为研究. 新媒体研究. 2022(21): 40-46 .
17.	徐建民，王恺霖，吴树芳. 基于改进D-S证据理论的微博不可信用户识别研究. 数据分析与知识发现. 2022(12): 99-112 .
18.	周晖. 国内外基于社交媒体的社会情绪对比分析. 中华医学图书情报杂志. 2022(12): 65-69 .