Conversational Emotion Recognition Incorporating Knowledge Graph and Curriculum Learning

Du Jinming; Sun Yuanyuan; Lin Hongfei; Yang Liang

doi:10.7544/issn1000-1239.202220951

Journal of Computer Research and Development > 2024 > 61(5): 1299-1309. > DOI: 10.7544/issn1000-1239.202220951

Du Jinming, Sun Yuanyuan, Lin Hongfei, Yang Liang. Conversational Emotion Recognition Incorporating Knowledge Graph and Curriculum Learning[J]. Journal of Computer Research and Development, 2024, 61(5): 1299-1309. DOI: 10.7544/issn1000-1239.202220951

Citation:

PDF (1262 KB)

Conversational Emotion Recognition Incorporating Knowledge Graph and Curriculum Learning

College of Computer Science and Technology, Dalian University of Technology, Dalian, Liaoning 116024

Funds: This work was supported by the National Key Research and Development Program of China (2022YFC3301801) and the Fundamental Research Funds for the Central Universities (DUT22ZD205)

More Information

Author Bio:
Du Jinming: born in 2000. Master. His main research interests include deep learning, natural language processing, and sentiment analysis

Sun Yuanyuan: born in 1979. PhD, professor, PhD supervisor. Senior member of CCF. Her main research interests include deep learning and natural language processing

Lin Hongfei: born in 1962. PhD, professor, PhD supervisor. Member of CCF. His main research interests include deep learning, natural language processing, and sentiment analysis. (hflin@dlut.edu.cn)

Yang Liang: born in 1986. PhD, lecturer. His main research interests include deep learning, natural language processing, and information retrieval. (liang@dlut.edu.cn)
Received Date: November 17, 2022
Revised Date: July 23, 2023
Available Online: February 21, 2024

Graphical Abstract

Abstract

Abstract

Conversational emotion recognition is the task of classifying emotions based on conversations. The conversation data are characterized by colloquial language and a wide range of topics, with semantic similarities among labels. Colloquial language exhibits issues such as word ambiguity and the omission of semantic information, emphasizing the importance of common sense and grammatical knowledge in conversational emotion recognition tasks, and these factors enable the model to accurately capture semantic information. Moreover, the current challenge lies in the variations in text richness and the frequency of emotion transfer across different dialogue scenarios, which result in suboptimal classification performance. We propose CK-ERC model to address these challenges. In the pre-training phase, CK-ERC model extracts structured data to incorporate common sense knowledge graphs and grammatical knowledge graphs, aiding the model in accurately capturing colloquial information. In the fine-tuning phase, a supervised contrast learning task is introduced to help the model identify similar emotional labels. Furthermore, a dynamic threshold-based curriculum learning strategy is designed for training and optimizing the model based on text richness (from high to low) and emotion transfer frequency (from low to high). CK-ERC model demonstrates superior performance in various conversation modes, including two-person conversation, multi-person conversation, simulated conversation, and daily conversation. Particularly, CK-ERC model achieves the best performance on MELD and EmoryNLP datasets.
- conversational emotion recognition,
- contrast learning,
- knowledge graph,
- curriculum learning,
- transfer learning

FullText(HTML)

References (38)

References

[1]	赵妍妍,秦兵,刘挺. 文本情感分析[J]. 软件学报,2010,21(8):1834−1848 doi: 10.3724/SP.J.1001.2010.03832 Zhao Yanyan, Qin Bing, Liu Ting. Sentiment analysis[J]. Journal of Software, 2010, 21(8): 1834−1848(in Chinese) doi: 10.3724/SP.J.1001.2010.03832
[2]	李然,林政,林海伦,等. 文本情绪分析综述[J]. 计算机研究与发展,2018,55(1):30−52 doi: 10.7544/issn1000-1239.2018.20170055 Li Ran, Lin Zheng, Lin Hailun, et al. Text emotion analysis: A survey[J]. Journal of Computer Research and Development, 2018, 55(1): 30−52(in Chinese) doi: 10.7544/issn1000-1239.2018.20170055
[3]	尹庆宇,张伟男,张宇,等. 省略识别及恢复联合模型研究[J]. 计算机研究与发展,2015,52(11):2460−2467 Yin Qingyu, Zhang Weinan, Zhang Yu, et al. A joint model for ellipsis identification and recovery [J]. Journal of Computer Research and Development, 2015, 52(11): 2460−2467(in Chinese)
[4]	Ghosal D, Majumder N, Gelbukh A, et al. Cosmic: Commonsense knowledge for emotion identification in conversations [J]. arXiv preprint, arXiv: 2010.02795, 2020
[5]	Zhang Duzhen, Chen Xiuyi, Xu Shuang, et al. Knowledge aware emotion recognition in textual conversations via multi-task incremental transformer [C/OL] //Proc of the 28th Int Conf on Computational Linguistics. [2023-06-16].https://aclanthology.org/2020.coling-main.392
[6]	张晓宇,李冬冬,任鹏杰,等. 基于记忆网络的知识感知医疗对话生成[J]. 计算机研究与发展,2022,59(12):2889−2900 doi: 10.7544/issn1000-1239.20210851 Zhang Xiaoyu, Li Dongdong, Ren Pengjie, et al. Memory networks based knowledge-aware medical dialogue generation[J]. Journal of Computer Research and Development, 2022, 59(12): 2889−2900(in Chinese) doi: 10.7544/issn1000-1239.20210851
[7]	Zhou Yikai, Yang Baosong, Wong D F, et al. Uncertainty-aware curriculum learning for neural machine translation [C] //Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 6934−6944
[8]	Huang Yuyun, Du Jinhua. Self-attention enhanced CNNs and collaborative curriculum learning for distantly supervised relation extraction [C] //Proc of the 24th Conf on Empirical Methods in Natural Language Processing and the 9th Int Joint Conf on Natural Language Processing (EMNLP-IJCNLP). Stroudsburg, PA: ACL, 2019: 389−398
[9]	曾维新,赵翔,唐九阳,等. 基于重排序的迭代式实体对齐[J]. 计算机研究与发展,2020,57(7):1460−1471 doi: 10.7544/issn1000-1239.2020.20190643 Zeng Weixin, Zhao Xiang, Tang Jiuyang, et al. Iterative entity alignment via re-ranking[J]. Journal of Computer Research and Development, 2020, 57(7): 1460−1471(in Chinese) doi: 10.7544/issn1000-1239.2020.20190643
[10]	Gao Tianyu, Yao Xingcheng, Chen Danqi. SimCSE: Simple contrastive learning of sentence embeddings [J]. arXiv preprint, arXiv: 2104.08821, 2021
[11]	Yan Yuanmeng, Li Rumei, Wang Sirui, et al. ConSERT: A contrastive framework for self-supervised sentence representation transfer [J]. arXiv preprint, arXiv: 2105.11741, 2021
[12]	Liu Yinhan, Ott M, Goyal N, et al. RoBERTa: A robustly optimized bert pretraining approach [J]. arXiv preprint, arXiv: 1907.11692, 2019
[13]	Wang Ruize, Tang Duyu, Duan Nan, et al. K-Adapter: Infusing knowledge into pre-trained models with adapters [J]. arXiv preprint, arXiv: 2002. 01808, 2020
[14]	Castells T, Weinzaepfel P, Revaud J. Superloss: A generic loss for robust curriculum learning [C/OL] // Proc of the 33rd Conf on Neural Information Processing Systems. [2023-06-16].https://proceedings.neurips.cc/paper/2020/hash/2cfa8f9e50e0f510ede9d12338a5f564-Abstract.html
[15]	Hazarika D, Poria S, Mihalcea R, et al. ICON: Interactive conversational memory network for multimodal emotion detection [C] //Proc of the 23rd Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2018: 2594−2604
[16]	Hazarika D, Poria S, Zadeh A, et al. Conversational memory network for emotion recognition in dyadic dialogue videos [C] //Proc of the 56th Conf Association for Computational Linguistics. Stroudsburg, PA: ACL, 2018: 2018−2122
[17]	Jiao Wenxiang, Yang Haiqin, King I, et al. Higru: Hierarchical gated recurrent units for utterance-level emotion recognition [J]. arXiv preprint, arXiv: 1904.04446, 2019
[18]	Majumder N, Poria S, Hazarika D, et al. Dialoguernn: An attentive RNN for emotion detection in conversations [C] //Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 6818−6825
[19]	Ghosal D, Majumder N, Poria S, et al. DialogueGCN: A graph convolutional neural network for emotion recognition in conversation [J]. arXiv preprint, arXiv: 1908.11540, 2019
[20]	Ishiwatari T, Yasuda Y, Miyazaki T, et al. Relation-aware graph attention networks with relational position encodings for emotion recognition in conversations [C] //Proc of the 25th Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2020: 7360−7370
[21]	Shen Weizhou, Wu Siyue, Yang Yunyi, et al. Directed acyclic graph network for conversational emotion recognition [J]. arXiv preprint, arXiv: 2105.12907, 2021
[22]	彭韬,杨亮,桑钟屹,等. 基于异构二部图的对话情感分析[J]. 中文信息学报,2021,35(11):135−142 doi: 10.3969/j.issn.1003-0077.2021.11.014 Peng Tao, Yang Liang, Sang Zhongyi, et al. Conversational sentiment analysis based on heterogeneous bipartite graphs[J]. Journal of Chinese Information Processing, 2021, 35(11): 135−142(in Chinese) doi: 10.3969/j.issn.1003-0077.2021.11.014
[23]	Li Jingye, Ji Donghong, Li Fei, et al. HiTrans: A transformer-based context-and speaker-sensitive model for emotion detection in conversations[C/OL] //Proc of the 28th Int Conf on Computational Linguistics. [2023-06-16].https://aclanthology.org/2020.coling-main.370
[24]	Devlin J, Chang Mingwei, Lee K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding [J]. arXiv preprint, arXiv: 1810.04805, 2018
[25]	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need [J]. arXiv preprint, arXiv: 1706.03762, 2017
[26]	Shen Weizhou, Chen Junqing, Quan Xiaojun, et al. DialogXL: All-in-one XLnet for multi-party conversation emotion recognition [C] //Proc of the 35th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 13789−13797
[27]	Yang Zhilin, Dai Zihang, Yang Yiming, et al. XLnet: Generalized autoregressive pretraining for language understanding [J]. arXiv preprint, arXiv: 1906.08237, 2019
[28]	Bengio Y, Louradour J, Collobert R, et al. Curriculum learning [C/OL] //Proc of the 26th Annual Int Conf on Machine Learning. [2023-06-16].https://dl.acm.org/doi/10.1145/1553374.1553380
[29]	Hacohen G, Weinshall D. On the power of curriculum learning in training deep networks [J]. arXiv preprint, arXiv: 1904.03626, 2019
[30]	Zhang Zhengyan, Han Xu, Liu Zhiyuan, et al. ERNIE: Enhanced language representation with informative entities [J]. arXiv preprint, arXiv: 1905.07129, 2019
[31]	Bordes A, Usunier N, Garcia-Duran A, et al. Translating embeddings for modeling multi-relational data [C/OL] // Proc of the 26th Conf on Neural Information Processing Systems. [2023-06-16].https://proceedings.neurips.cc/paper/2013/hash/1cecc7a77928ca8133fa24680a88d2f9-Abstract.html
[32]	Lauscher A, Vulić I, Ponti E M, et al. Informing unsupervised pretraining with external linguistic knowledge [J]. arXiv preprint, arXiv: 1909.02339, 2019
[33]	Xiong Wenhan, Du Jingfei, Wang W Y, et al. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model [J]. arXiv preprint, arXiv: 1912.09637, 2019
[34]	Khosla P, Teterwak P, Wang C, et al. Supervised contrastive learning [C/OL] // Proc of the 33rd Conf on Neural Information Processing Systems. [2023-06-16]. https://proceedings.neurips.cc/paper/2020/hash/d89a66c7c80a29b1bdbab0f2a1a94af8-Abstract.html
[35]	Gunel B, Du Jingfei, Conneau A, et al. Supervised contrastive learning for pre-trained language model fine-tuning [J]. arXiv preprint, arXiv: 2011.01403, 2020
[36]	Zhong Peixiang, Wang Di, Miao Chunyan. Knowledge-enriched transformer for emotion detection in textual conversations [J]. arXiv preprint, arXiv: 1909.10681, 2019
[37]	Song Xiaohui, Zang Liangjun, Zhang Rong, et al. EmotionFlow: Capture the dialogue level emotion transitions [C] // Proc of the 47th Int Conf on Acoustics, Speech and Signal Processing. Piscataway, NJ: IEEE, 2022: 8542−8546
[38]	Sun Yang, Yu Nan, Fu Guohong. A discourse-aware graph neural network for emotion recognition in multi-party conversation [C] // Proc of the 26th Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2021: 2949−2958

[1]	Zhang Naizhou, Cao Wei, Zhang Xiaojian, Li Shijun. Conversation Generation Based on Variational Attention Knowledge Selection and Pre-trained Language Model[J]. Journal of Computer Research and Development. DOI: 10.7544/issn1000-1239.202440551
[2]	Wang Honglin, Yang Dan, Nie Tiezheng, Kou Yue. Attributed Heterogeneous Information Network Embedding with Self-Attention Mechanism for Product Recommendation[J]. Journal of Computer Research and Development, 2022, 59(7): 1509-1521. DOI: 10.7544/issn1000-1239.20210016
[3]	Cheng Yan, Yao Leibo, Zhang Guanghe, Tang Tianwei, Xiang Guoxiong, Chen Haomai, Feng Yue, Cai Zhuang. Text Sentiment Orientation Analysis of Multi-Channels CNN and BiGRU Based on Attention Mechanism[J]. Journal of Computer Research and Development, 2020, 57(12): 2583-2595. DOI: 10.7544/issn1000-1239.2020.20190854
[4]	Wei Zhenkai, Cheng Meng, Zhou Xiabing, Li Zhifeng, Zou Bowei, Hong Yu, Yao Jianmin. Convolutional Interactive Attention Mechanism for Aspect Extraction[J]. Journal of Computer Research and Development, 2020, 57(11): 2456-2466. DOI: 10.7544/issn1000-1239.2020.20190748
[5]	Chen Yanmin, Wang Hao, Ma Jianhui, Du Dongfang, Zhao Hongke. A Hierarchical Attention Mechanism Framework for Internet Credit Evaluation[J]. Journal of Computer Research and Development, 2020, 57(8): 1755-1768. DOI: 10.7544/issn1000-1239.2020.20200217
[6]	Li Mengying, Wang Xiaodong, Ruan Shulan, Zhang Kun, Liu Qi. Student Performance Prediction Model Based on Two-Way Attention Mechanism[J]. Journal of Computer Research and Development, 2020, 57(8): 1729-1740. DOI: 10.7544/issn1000-1239.2020.20200181
[7]	Zhang Yingying, Qian Shengsheng, Fang Quan, Xu Changsheng. Multi-Modal Knowledge-Aware Attention Network for Question Answering[J]. Journal of Computer Research and Development, 2020, 57(5): 1037-1045. DOI: 10.7544/issn1000-1239.2020.20190474
[8]	Zhang Yixuan, Guo Bin, Liu Jiaqi, Ouyang Yi, Yu Zhiwen. app Popularity Prediction with Multi-Level Attention Networks[J]. Journal of Computer Research and Development, 2020, 57(5): 984-995. DOI: 10.7544/issn1000-1239.2020.20190672
[9]	Liu Ye, Huang Jinxiao, Ma Yutao. An Automatic Method Using Hybrid Neural Networks and Attention Mechanism for Software Bug Triaging[J]. Journal of Computer Research and Development, 2020, 57(3): 461-473. DOI: 10.7544/issn1000-1239.2020.20190606
[10]	Zhang Zhichang, Zhang Zhenwen, Zhang Zhiman. User Intent Classification Based on IndRNN-Attention[J]. Journal of Computer Research and Development, 2019, 56(7): 1517-1524. DOI: 10.7544/issn1000-1239.2019.20180648