A Joint Emotion-Cognition Based Approach for Moral Judgement

Wu Di; Zhao Yanyan; Qin Bing

doi:10.7544/issn1000-1239.202330812

Journal of Computer Research and Development > 2024 > 61(5): 1193-1205. > DOI: 10.7544/issn1000-1239.202330812 CSTR: 32373.14.issn1000-1239.202330812

Wu Di, Zhao Yanyan, Qin Bing. A Joint Emotion-Cognition Based Approach for Moral Judgement[J]. Journal of Computer Research and Development, 2024, 61(5): 1193-1205. DOI: 10.7544/issn1000-1239.202330812

Citation:

PDF (986 KB)

A Joint Emotion-Cognition Based Approach for Moral Judgement

Research Center for Social Computing and Information Retrieval(Harbin Institute of Technology), Harbin 150001
Key Laboratory of Cognitive Intelligence and Content Security (Harbin Institute of Technology), Ministry of Education, Harbin 150001

More Information

Author Bio:
Wu Di: born in 2000. PhD candidate. Student member of CCF. His main research interests include large language model safety, value alignment, and affective computing

Zhao Yanyan: born in 1983. PhD, professor, PhD supervisor. Member of CCF. Her main research interests include large language model safety, value alignment, and affective computing

Qin Bing: born in 1968. PhD, professor, PhD supervisor. Member of CCF. Her main research interests include large language model safety, affective computing, and text generation
Received Date: October 10, 2023
Revised Date: February 07, 2024
Available Online: March 06, 2024

Graphical Abstract

Abstract

Abstract

With the rapid development of large language models, their safety has become a growing concern among researchers and the public. To prevent potential harm in collaboration, aligning these models’ judgments with human moral values in daily scenarios is essential. A key challenge is ensuring that large language models can adaptively adjust or reassess rules in moral judgment, like humans, to maintain consistency with human morals in various contexts. Inspired by psychological and cognitive science research on the emotional and cognitive influences on human moral judgments, this study leverages the strengths of large language models in cognitive reasoning and emotional analysis. We develop an approach that emulates the interaction between emotional and cognitive judgment in human moral reasoning, thus enhancing these models’ moral judgment capabilities. Experimental results demonstrate the effectiveness of our approach in this task. Overall, this study not only presents an innovative approach to the moral judgment of large language models but also highlights the importance of integrating psychological and cognitive science theories in this field, setting a foundation for future research.
- moral judgement,
- large language model safety,
- cognitive judgment capability,
- emotional judgment capability,
- prompt learning

FullText(HTML)

References (48)

References

[1]	Tegmark M. Life 3.0: Being Human in the Age of Artificial Intelligence[M]. New York: Vintage, 2018
[2]	Russell S. Human Compatible: Artificial Intelligence and the Problem of Control[M]. London: Penguin, 2019
[3]	Asimov I. I, Robot[M]. New York: Bantam, 2008
[4]	Hendrycks D, Burns C, Basart S, et al. Aligning AI with shared human values[C]//Proc of Int Conf on Learning Representations. New Orleans, LA: OpenReview, 2020: 1−29
[5]	Kenton Z, Everitt T, Weidinger L, et al. Alignment of language agents[J]. arXiv preprint, arXiv: 2103.14659, 2021
[6]	Weidinger L, Mellor J, Rauh M, et al. Ethical and social risks of harm from language models[J]. arXiv preprint, arXiv: 2112.04359, 2021
[7]	Hendrycks D, Carlini N, Schulman J, et al. Unsolved problems in ML safety[J]. arXiv preprint, arXiv: 2109.13916, 2021
[8]	Wang Zengzhi, Xie Qiming, Ding Zixiang, et al. Is ChatGPT a good sentiment analyzer? a preliminary study[J]. arXiv preprint, arXiv: 2304.04339, 2023
[9]	Zhao Weixiang, Zhao Yanyan, Lu Xin, et al. Is ChatGPT equipped with emotional dialogue capabilities?[J]. arXiv preprint, arXiv: 2304.09582, 2023
[10]	Strawson P F. Freedom and resentment[J]. Proceedings of the British Academy, 1962, 48: 187−211
[11]	Jin Zhijing, Levine S, Gonzalez Adauto F, et al. When to make exceptions: Exploring language models as accounts of human moral judgment[C]// Advances in Neural Information Processing Systems. San Diego: Neural Information Processing Systems Foundation Inc, 2022, 35: 28458−28473
[12]	Forbes M, Hwang J D, Shwartz V, et al. Social chemistry 101: Learning to reason about social and moral norms[C]//Proc of the 2020 Conf on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg, PA: ACL, 2020: 653−670
[13]	Ziems C, Yu J, Wang Y C, et al. The moral integrity corpus: A benchmark for ethical dialogue systems[C]//Proc of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Stroudsburg, PA: ACL, 2022: 3755−3773
[14]	Emelin D, Le Bras R, Hwang J D, et al. Moral stories: Situated reasoning about norms, intents, actions, and their consequences[C]//Proc of the 2021 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2021: 698−718
[15]	Jiang Liwei, Hwang J D, Bhagavatula C, et al. Delphi: Towards machine ethics and norms[J]. arXiv preprint, arXiv: 2110.07574, 2021
[16]	Garrigan B, Adlam A L R, Langdon P E. Moral decision-making and moral development: Toward an integrative framework[J]. Developmental Review, 2018, 49: 80−100 doi: 10.1016/j.dr.2018.06.001
[17]	Crick N R, Dodge K A. A review and reformulation of social information-processing mechanisms in children’s social adjustment[J]. Psychological Bulletin, 1994, 115(1): 74−101 doi: 10.1037/0033-2909.115.1.74
[18]	Lemerise E A, Arsenio W F. An integrated model of emotion processes and cognition in social information processing[J]. Child Development, 2000, 71(1): 107−118 doi: 10.1111/1467-8624.00124
[19]	Piaget J. The Moral Judgement of the Child[M]. London: Routledge, 1932
[20]	Haidt J. The emotional dog and its rational tail: A social intuitionist approach to moral judgment[J]. Psychological Review, 2001, 108(4): 814−834 doi: 10.1037/0033-295X.108.4.814
[21]	Greene J D, Sommerville R B, Nystrom L E, et al. An fMRI investigation of emotional engagement in moral judgment[J]. Science, 2001, 293(5537): 2105−2108 doi: 10.1126/science.1062872
[22]	Sap M, Gabriel S, Qin L, et al. Social bias frames: Reasoning about social and power implications of language[C]//Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 5477−5490
[23]	Jentzsch S, Schramowski P, Rothkopf C, et al. Semantics derived automatically from language corpora contain human-like moral choices[C]//Proc of the 2019 AAAI/ACM Conf on AI, Ethics, and Society. New York: ACM, 2019: 37−44
[24]	Schramowski P, Turan C, Andersen N, et al. Large pre-trained language models contain human-like biases of what is right and wrong to do[J]. Nature Machine Intelligence, 2022, 4(3): 258−268 doi: 10.1038/s42256-022-00458-8
[25]	Kim H, Yu Y, Jiang Liwei, et al. Prosocialdialog: A prosocial backbone for conversational agents[C]//Proc of the 2022 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2022: 4005−4029
[26]	Nahian M S A, Frazier S, Riedl M, et al. Learning norms from stories: A prior for value aligned agents[C]//Proc of the AAAI/ACM Conf on AI, Ethics, and Society. New York: ACM, 2020: 124−130
[27]	Pyatkin V, Hwang J D, Srikumar V, et al. Clarifydelphi: Reinforced clarification questions with defeasibility rewards for social and moral situations[C]//Proc of the 61st Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2023: 11253−11271
[28]	Lourie N, Le Bras R, Choi Y. Scruples: A corpus of community ethical judgments on 32, 000 real-life anecdotes[C]//Proc of the AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021, 35(15): 13470−13479
[29]	Kohlberg L. Moral stages and moralization: The cognitive-development approach[J]. Moral Development and Behavior: Theory Research and Social Issues, 1976: 31−53
[30]	Rest J R, Thoma S J, Bebeau M J. Postconventional Moral Thinking: A Neo-Kohlbergian Approach[M]. Mahwah, NJ: Lawrence Erlbaum Associates, 1999
[31]	Gibbs J C. Moral Development and Reality: Beyond the Theories of Kohlberg, Hoffman, and Haidt[M]. New York: Oxford University Press, 2019
[32]	Arsenio W F, Lemerise E A. Aggression and moral development: Integrating social information processing and moral domain models[J]. Child Development, 2004, 75(4): 987−1002 doi: 10.1111/j.1467-8624.2004.00720.x
[33]	Palmer E J. Offending Behaviour[M]. London: Routledge, 2013
[34]	Levine S, Kleiman-Weiner M, Chater N, et al. The cognitive mechanisms of contractualist moral decision-making[C]// Proc of the 40th Annual Meeting of the Cognitive Science Society. Mahwah, NJ: Cognitive Science Society, 2018: 1−7
[35]	Taber-Thomas B C, Tranel D. Social and moral functioning[J]. Developmental Social Neuroscience and Childhood Brain Insult: Theory and Practice, 2012: 65−90
[36]	Anderson V, Beauchamp M. Social: A theoretical model of developmental social neuroscience[J]. Developmental Social Neuroscience and Childhood Brain Insult: Theory and Practice, 2012: 3−22
[37]	Kiley Hamlin J, Wynn K, Bloom P. Three-month-olds show a negativity bias in their social evaluations[J]. Developmental Science, 2010, 13(6): 923−929 doi: 10.1111/j.1467-7687.2010.00951.x
[38]	Rest J R. The major components of morality[J]. Morality, Moral Behavior, and Moral Development, 1984, 24: 24−36
[39]	Hoffman M L. Empathy and Moral Development: Implications for Caring and Justice[M]. Cambridge, UK: Cambridge University Press, 2001
[40]	Kovač G, Sawayama M, Portelas R, et al. Large language models as superpositions of cultural perspectives[J]. arXiv preprint, arXiv: 2307.07870, 2023
[41]	Zhou Jingyan, Hu Minda, Li Junan, et al. Rethinking machine ethics−can LLMs perform moral reasoning through the lens of moral theories?[J]. arXiv preprint, arXiv: 2308.15399, 2023
[42]	Devlin J, Chang Mingwei, Lee K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proc of the 2019 Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA: ACL, 2019: 4171−4186
[43]	Liu Yinhan, Ott M, Goyal N, et al. RoBERTa: A robustly optimized BERT pretraining approach[J]. arXiv preprint, arXiv: 1907.11692, 2019
[44]	Lan Zhenzhong, Chen Minda, Goodman S, et al. ALBERT: A lite BERT for self-supervised learning of language representations[C]// Proc of Int Conf on Learning Representations. New Orleans, LA: OpenReview, 2019: 1−17
[45]	Brown T, Mann B, Ryder N, et al. Language models are few-shot learners[C]//Advances in Neural Information Processing Systems. San Diego: Neural Information Processing Systems Foundation Inc, 2020, 33: 1877−1901
[46]	Ouyang L, Wu J, Jiang Xu, et al. Training language models to follow instructions with human feedback[C]//Advances in Neural Information Processing Systems. San Diego: Neural Information Processing Systems Foundation Inc, 2022, 35: 27730−27744
[47]	Wei J, Wang Xuezhi, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models[C]//Advances in Neural Information Processing Systems. San Diego: Neural Information Processing Systems Foundation Inc, 2022, 35: 24824−24837
[48]	Press O, Zhang Muru, Min S, et al. Measuring and narrowing the compositionality gap in language models[J]. arXiv preprint, arXiv: 2210.03350, 2022