A Unified Framework Based on Multimodal Aspect-Term Extraction and Aspect-Level Sentiment Classification

Zhou Ru; Zhu Haoze; Guo Wenya; Yu Shenglong; Zhang Ying

doi:10.7544/issn1000-1239.202220441

Journal of Computer Research and Development > 2023 > 60(12): 2877-2889. > DOI: 10.7544/issn1000-1239.202220441

Zhou Ru, Zhu Haoze, Guo Wenya, Yu Shenglong, Zhang Ying. A Unified Framework Based on Multimodal Aspect-Term Extraction and Aspect-Level Sentiment Classification[J]. Journal of Computer Research and Development, 2023, 60(12): 2877-2889. DOI: 10.7544/issn1000-1239.202220441

Citation:

PDF (1479 KB)

A Unified Framework Based on Multimodal Aspect-Term Extraction and Aspect-Level Sentiment Classification

College of Computer Science, Nankai University, Tianjin 300350

Funds: This work was supported by the National Natural Science Foundation of China-Joint Fund program（U1903128）.

More Information

Author Bio:
Zhou Ru: born in 1999. Master candidate. Her main research interests include sentiment analysis and information retrieval

Zhu Haoze: born in 2001. Undergraduate. His main research interests include sentiment analysis and information retrieval

Guo Wenya: born in 1994. PhD. Her main research interests include sentiment analysis and data mining

Yu Shenglong: born in 1998. Master candidate. His main research interests include data mining and reinforcement learning

Zhang Ying: born in 1986. PhD, professor. Her main research interests include sentiment analysis, data mining, and information retrieval
Received Date: May 27, 2022
Revised Date: February 19, 2023
Available Online: September 19, 2023

Graphical Abstract

Abstract

Abstract

Aspect-term extraction (AE) and aspect-level sentiment classification (ALSC) extract aspect-sentiment pairs in the sentence, which helps social media platforms such as Twitter and Facebook to mine users’ sentiments of different aspects and is of great significance to personalized recommendation. In the field of multimodality, the existing method uses two independent models to complete two subtasks respectively. Aspect-term extraction identifies goods, important people and other entities or entities’ aspects in the sentence, and aspect-level sentiment classification predicts the user’s sentiment orientation according to the given aspect terms. There are two problems in the above method: first, using two independent models loses the continuity of the underlying features between the two tasks, and cannot model the potential semantic association of sentences; second, aspect-level sentiment classification can only predict the sentiment of one aspect at a time, which does not match the throughput of aspect-term extraction model that extracts multiple aspects simultaneously, and the serial execution of the two models makes the efficiency of extracting aspect-sentiment pairs low. To solve the above problems, a unified framework based on multimodal aspect-term extraction and aspect-level sentiment classification, called UMAS, is proposed in this paper. Firstly, the shared feature module is built to realize the latent semantic association modeling between tasks, and to make the two subtasks only need to care about their upper network, which reduces the complexity of the model. Secondly, multiple aspects and their corresponding sentiment categories in the sentence are output at the same time by using sequence tagging, which improves the extraction efficiency of aspect-sentiment pairs. In addition, we introduce part of speech in two subtasks at the same time: using the grammatical information to improve the performance of aspect-term extraction, and the information of opinion words is obtained through part of speech to improve the performance of aspect-level sentiment classification. The experimental results show that the unified model has superior performance compared with multiple baseline models on the two benchmark datasets of Twitter2015 and Restaurant2014.

FullText(HTML)

References (38)

References

[1]	Ju Xincheng, Zhang Dong, Xiao Rong, et al. Joint multi-modal aspect-sentiment analysis with auxiliary cross-modal relation detection[C] // Proc of the 26th Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2021: 4395−4405
[2]	Li Z Y, Cheng Wei, Kshetramade R, et al. Recommend for a reason: Unlocking the power of unsupervised aspect-sentiment co-extraction[C]// Proc of the 26th Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2021: 763−778
[3]	Gong Chenggong, Yu Jianfei, Xia Rui. Unified feature and instance based domain adaptation for aspect-based sentiment analysis[C] // Proc of the 25th Conf on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg, PA: ACL, 2020: 7035−7045
[4]	Cai Guoyong, Xia Binbin. Convolutional Neural Networks for Multimedia Sentiment Analysis[M] //Natural Language Processing and Chinese Computing. Cham: Springer, 2015: 159−167
[5]	Zadeh A, Chen Minghai, Poria S, et al. Tensor fusion network for multimodal sentiment analysis[C] //Proc of the 22nd Conf on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg, PA: ACL, 2017: 1103−1114
[6]	Mai Sijie, Hu Haifeng, Xing Songlong. Divide, conquer and combine: hierarchical feature fusion network with local and global perspectives for multimodal affective computing[C] //Proc of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2019: 481−492
[7]	Zhang Qi, Fu Jinlan, Liu Xiaoyu, et al. Adaptive co-attention network for named entity recognition in tweets[C] //Proc of the 32nd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 5674−5681.
[8]	Yu Jianfei, Jiang Jing, Xia Rui. Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020, 28: 429−439 doi: 10.1109/TASLP.2019.2957872
[9]	Lu Di, Neves L, Carvalho V, et al. Visual attention model for name tagging in multimodal social media[C] //Proc of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2018: 1990−1999
[10]	Yu Jianfei, Jiang Jing, Yang Li, et al. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer[C] //Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 3342−3352
[11]	Honnibal M, Montani I, Van Landeghem S, et al. spaCy: Industrial-strength natural language processing in Python[EB/OL]. [2022-05-17].https://spacy.io
[12]	刘路路,杨燕,王杰. ABAFN:面向多模态的方面级情感分类模型[J]. 计算机工程与应用,2022,58(10):193−199 doi: 10.3778/j.issn.1002-8331.2108-0056 Liu Lulu, Yang Yan, Wang Jie. ABAFN: Aspect-based sentiment analysis model for multimodal[J]. Computer Engineering and Applications, 2022, 58(10): 193−199 (in Chinese) doi: 10.3778/j.issn.1002-8331.2108-0056
[13]	Li Ruifan, Chen Hao, Feng Fangxiang, et al. Dual graph convolutional networks for aspect-based sentiment analysis[C] //Proc of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int Joint Conf on Natural Language Processing. Stroudsburg, PA: ACL, 2021: 6319−6329
[14]	Xiao Zeguan, Wu Jiarun, Chen Qingliang, et al. BERT4GCN: Using BERT intermediate layers to augment GCN for aspect-based sentiment classification[J]. arXiv preprint, arXiv: 2110.00171, 2021
[15]	齐嵩喆,黄贤英,孙海栋,等. 基于渐进增强与图卷积的方面级情感分类模型[J]. 计算机应用研究,2022,39(7):2037−2042 Qi Songzhe, Huang Xianying, Sun Haidong, et al. Aspect based sentiment analysis with progressive enhancement and graph convolution[J]. Application Research of Computers, 2022, 39(7): 2037−2042 (in Chinese)
[16]	韩虎, 郝俊, 张千锟, 等. 知识增强的交互注意力方面级情感分类模型[J]. 计算机科学与探索, 2022 [2021-12-31]. http: //fcst.ceaj.org/CN/10.3778/j.issn.1673−9418.2108082 Han Hu, Hao Jun, Zhang Qiankun, et al. Knowledge-enhanced interactive attention model for aspect-based sentiment analysis[J/OL]. Journal of Frontiers of Computer Science and Technology, 2022 [2021-12-31]. http://fcst.ceaj.org/CN/10.3778/j.issn.1673−9418.2108082 (in Chinese)
[17]	毛腾跃, 郑志鹏, 郑禄. 基于改进自注意力机制的方面级情感分类[J]. 中南民族大学学报: 自然科学版, 2022, 41(1): 94−100 Mao Tengyue, Zheng Zhipeng, Zheng Lu. Aspect-level sentiment analysis based on improved self-attention mechanism[J]. Journal of South Central University for Nationalities: Natural Science Edition, 2022, 41(1): 94−100(in Chinese)
[18]	孙小婉,王英,王鑫,等. 面向双注意力网络的特定方面情感分析模型[J]. 计算机研究与发展,2019,56(11):2384−2395 doi: 10.7544/issn1000-1239.2019.20180823 Sun Xiaowan, Wang Ying, Wang Xin, et al. Aspect-based sentiment analysis model based on dual-attention networks[J]. Journal of Computer Research and Development, 2019, 56(11): 2384−2395 (in Chinese) doi: 10.7544/issn1000-1239.2019.20180823
[19]	Ying Chengcan, Wu Zhen, Dai Xinyu, et al. Opinion transmission network for jointly improving aspect-oriented opinion words extraction and sentiment classification[C] // Proc of the 9th CCF Int Conf on Natural Language Processing and Chinese Computing. Cham: Springer, 2020: 629−640
[20]	Oh S, Lee D, Whang T et al. Deep context- and relation-aware learning for aspect-based sentiment analysis[C] //Proc of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int Joint Conf on Natural Language Processing. Stroudsburg, PA: ACL, 2021: 495−503
[21]	Chen Zhuang, Qian Tieyun. Relation-aware collaborative learning for unified aspect-based sentiment analysis[C] //Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 3685−3694
[22]	Xu Lu, Li Hao, Lu Wei, et al. Position-aware tagging for aspect sentiment triplet extraction[C]//Proc of the 25th Conf on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg, PA: ACL, 2020: 2339−2349
[23]	Phan M H, Ogunbona P O. Modelling context and syntactical features for aspect-based sentiment analysis[C] //Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 3211−3220
[24]	薛芳,过弋,李智强,等. 基于双层词性感知和多头交互注意机制的方面级情感分类[J]. 计算机应用研究,2022,39(3):704−710 Xue Fang, Guo Yi, Li Zhiqiang, et al. Aspect-level sentiment analysis based on double-layer part-of-speech-aware and multi-head interactive attention mechanism[J]. Application Research of Computers, 2022, 39(3): 704−710 (in Chinese)
[25]	He Ruidan, Lee W S, Ng H T, et al. An interactive multi-task learning network for end-to-end aspect-based sentiment analysis[C] //Proc of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2019: 504−515
[26]	Wang Feixiang, Lan Man, Wang Wenting. Towards a one-stop solution to both aspect extraction and sentiment analysis tasks with neural multi-task learning[C/OL]// Proc of the 2018 Int Joint Conf on Neural Networks (IJCNN). Piscataway, NJ : IEEE, 2018 [2022-03-01].https://ieeexplore.ieee.org/abstract/document/8489042
[27]	Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint, arXiv: 1409.1556, 2014
[28]	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C] // Proc of the 31st Conf on Advances in Neural Information Processing Systems. Cambridge, MA: MIT, 2017: 5998−6008
[29]	Jeffrey P, Richard S, Christopher M. GloVe: Global vectors for word representation[C] //Proc of the 19th Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2014: 1532−1543
[30]	Wu Yuanbin, Zhang Qi, Huang Xuanjing, et al. Phrase dependency parsing for opinion mining[C] //Proc of the 14th Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2009: 1533−1541
[31]	Xu Hu, Liu Bing, Shu Lei, et al. Double embeddings and CNN-based sequence labeling for aspect extraction[C] //Proc of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2018: 592−598
[32]	Chen Zhuang, Qian Tieyun. Transfer capsule network for aspect level sentiment classification[C] //Proc of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2019: 547−556
[33]	Li Xin, Bing Lidong, Li Piji, et al. A unified model for opinion target extraction and target sentiment prediction[C] //Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 6714−6721
[34]	Luo Huaishao, Li Tianrui, Liu Bing, et al. DOER: Dual cross-shared RNN for aspect term-polarity co-extraction[C] //Proc of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2019: 591−601
[35]	Hazarika D, Poria S, Zadeh A, et al. Conversational memory network for emotion recognition in dyadic dialogue videos[C] //Proc of the 16th Conf on Association for Computational Linguistics, North American Chapter. Stroudsburg, PA: ACL, 2018: 2122−2132
[36]	Chen Peng, Sun Zhongqian, Bing Lidong, et al. Recurrent attention network on memory for aspect sentiment analysis[C]// Proc of the 22nd Conf on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg, PA: ACL, 2017: 452−461
[37]	Fan Feifan, Feng Yansong, Zhao Dongyan. Multi-grained attention network for aspect-level sentiment classification[C] //Proc of the 23rd Conf on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg, PA: ACL, 2018: 3433−3442
[38]	Xu Nan, Mao Wenji, Chen Guandan. Multi-interactive memory network for aspect based multimodal sentiment analysis[C] //Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 371−378

[1]	Lin Liansheng, Zheng Huanqin, Su Shen, Lei Kai, Chen Xiaofeng, Tian Zhihong. An On-Chain Mechanism Against DeFi Price Manipulation Attacks[J]. Journal of Computer Research and Development, 2025, 62(2): 443-457. DOI: 10.7544/issn1000-1239.202330291
[2]	Song Shuwei, Ni Xiaoze, Chen Ting. Gas Optimization for Smart Contracts: A Survey[J]. Journal of Computer Research and Development, 2023, 60(2): 311-325. DOI: 10.7544/issn1000-1239.202220887
[3]	Ying Chenhao, Xia Fuyuan, Li Jie, Si Xueming, Luo Yuan. Incentive Mechanism Based on Truth Estimation of Private Data for Blockchain-Based Mobile Crowdsensing[J]. Journal of Computer Research and Development, 2022, 59(10): 2212-2232. DOI: 10.7544/issn1000-1239.20220493
[4]	Feng Jingyu, Yang Jinwen, Zhang Ruitong, Zhang Wenbo. A Spectrum Sharing Incentive Scheme Against Location Privacy Leakage in IoT Networks[J]. Journal of Computer Research and Development, 2020, 57(10): 2209-2220. DOI: 10.7544/issn1000-1239.2020.20200453
[5]	Hai Mo, Zhu Jianming. A Propagation Mechanism Combining an Optimal Propagation Path and Incentive in Blockchain Networks[J]. Journal of Computer Research and Development, 2019, 56(6): 1205-1218. DOI: 10.7544/issn1000-1239.2019.20180419
[6]	He Yunhua, Li Mengru, Li Hong, Sun Limin, Xiao Ke, Yang Chao. A Blockchain Based Incentive Mechanism for Crowdsensing Applications[J]. Journal of Computer Research and Development, 2019, 56(3): 544-554. DOI: 10.7544/issn1000-1239.2019.20170670
[7]	He Haiwu, Yan An, Chen Zehua. Survey of Smart Contract Technology and Application Based on Blockchain[J]. Journal of Computer Research and Development, 2018, 55(11): 2452-2466. DOI: 10.7544/issn1000-1239.2018.20170658
[8]	Xiong Jinbo, Ma Rong, Niu Ben, Guo Yunchuan, Lin Li. Privacy Protection Incentive Mechanism Based on User-Union Matching in Mobile Crowdsensing[J]. Journal of Computer Research and Development, 2018, 55(7): 1359-1370. DOI: 10.7544/issn1000-1239.2018.20180080
[9]	Wang Bo, Huang Chuanhe, Yang Wenzhong, Dan Feng, and Xu Liya. An Incentive-Cooperative Forwarding Model Based on Punishment Mechanism in Wireless Ad Hoc Networks[J]. Journal of Computer Research and Development, 2011, 48(3): 398-406.
[10]	Yue Guangxue, Li Renfa, Chen Zhi, Zhou Xu. Analysis of Free-riding Behaviors and Modeling Restrain Mechanisms for Peer-to-Peer Networks[J]. Journal of Computer Research and Development, 2011, 48(3): 382-397.

Cited By

Cited by

Periodical cited type(2)

1.	李硕，王馨爽. 多场景融合的码号数据分发架构及关键技术研究. 数据通信. 2024(06): 1-3+11 .
2.	俞惠芳，李磊. 基于椭圆曲线签密的跨链医疗数据共享方案. 通信学报. 2024(12): 57-66 .