Category Adversarial Joint Learning Method for Cross-Prompt Automated Essay Scoring

Zhang Chunyun; Zhao Hongyan; Deng Jiqin; Cui Chaoran; Dong Xiaolin; Chen Zhumin

doi:10.7544/issn1000-1239.202440266

Journal of Computer Research and Development > 2025 > Uncorrected proof > DOI: 10.7544/issn1000-1239.202440266 CSTR: 32373.14.issn1000-1239.202440266

Zhang Chunyun, Zhao Hongyan, Deng Jiqin, Cui Chaoran, Dong Xiaolin, Chen Zhumin. Category Adversarial Joint Learning Method for Cross-Prompt Automated Essay Scoring[J]. Journal of Computer Research and Development. DOI: 10.7544/issn1000-1239.202440266

Citation:

PDF (1741 KB)

Category Adversarial Joint Learning Method for Cross-Prompt Automated Essay Scoring

Zhang Chunyun^{1, 2,},
Zhao Hongyan^{1, 2},
Deng Jiqin^{1, 2},
Cui Chaoran^{1, 2, ,},
Dong Xiaolin^{1, 2},
Chen Zhumin³

1.
School of Computer Science and Technology, Shandong University of Finance and Economic, Jinan 250014
2.
Key Laboratory of Digital Media of Shandong Province (Shandong University of Finance and Economic), Jinan 250014
3.
School of Computer Science and Technology, Shandong University, Qingdao, Shandong 266237

Funds: This work was supported by the National Natural Science Foundation of China (62077033), the Shandong Provincial Natural Science Foundation (ZR2020KF015), and the Taishan Scholar Program of Shandong Province (tsqn202211199).

More Information

Author Bio:
Zhang Chunyun: born in 1986. PhD, associate professor. Member of CCF. Her main research interests include natural language processing and machine learning

Zhao Hongyan: born in 2000. Master. His main research interests include natural language processing and machine learning

Deng Jiqin: born in 1996. Master. Her main research interests include natural language processing and machine learning

Cui Chaoran: born in 1987. PhD, professor, Young Taishan scholar of Shandong province. Member of CCF. His main research interests include machine learning and data mining

Dong Xiaolin: born in 1998. Master. Her main research interest includes machine learning

Chen Zhumin: born in 1978. PhD, professor. Member of CCF. His main research interests include machine learning and natural language processing
Received Date: April 17, 2024
Revised Date: September 19, 2024
Accepted Date: October 15, 2024
Available Online: October 21, 2024

Graphical Abstract

Abstract

Abstract

Automated essay scoring (AES) can effectively alleviate the burden on teachers when evaluating student essays and provide students with objective and timely feedback. It is a crucial application of natural language processing in the field of education. Cross-prompt AES aims to develop a transferable automated scoring model that performs well on essays from a target prompt. However, existing cross-prompt AES models primarily operate in scenarios where target prompt data are available. These models align feature distributions between source and target prompts to learn invariant feature representations for transferring to the target prompt. Unfortunately, such methods cannot be applied to scenarios where target prompt data are not available. In this paper, we propose a cross-prompt AES method based on category adversarial joint learning (CAJL). First, we jointly model AES as classification and regression tasks to achieve combined performance improvement. Second, unlike existing methods that rely on prompt-agnostic features to enhance model generalization, our approach introduces a category adversarial strategy. By aligning category level features across different prompts, we can learn invariant feature representations of different prompts and further enhance model generalization. We evaluate our proposed method on ASAP (automated student assessment prize) and ASAP++ datasets, predicting both overall essay scores and trait scores. Experimental results demonstrate that our method outperforms six classical methods in terms of the QWK (quadratic weighted Kappa) metric.
- automated essay scoring (AES),
- cross-prompt,
- category adversarial,
- joint modeling,
- domain generalization

FullText(HTML)

References (45)

References

[1]	Ke Z, Ng V. Automated essay scoring: A survey of the state of the art [C] //Proc of the 28th Int Joint Conf on Artificial Intelligence. Palo Alto, CA: IJCAI, 2019: 6300−6308
[2]	丁革建,刘畅. 作文自动评分技术综述[J]. 计算机应用,2022,42(S1):386−390 Ding Gejian, Liu Chang. Survey of automated essay scoring technology[J]. Journal of Computer Applications, 2022, 42(S1): 386−390 (in Chinese)
[3]	Page E B. The imminence of grading essays by computer[J]. The Phi Delta Kappan, 1966, 47(5): 238−243
[4]	Landauer T K. Automated scoring and annotation of essays with the intelligent essay assessor [J]. Automated Essay Scoring: A Cross-Disciplinary Perspective, 2003: 87−112
[5]	Attali Y, Burstein J. Automated essay scoring with e-rater V.2[J]. Journal of Technology, Learning, and Assessment, 2006, 4(3): 1−30
[6]	Kim Y. Convolutional neural networks for sentence classification [C] // Proc of the 2014 Int Conf on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg, PA: ACL, 2014: 1746−1751
[7]	Yang Zichao, Yang Diyi, Dyer C, et al. Hierarchical attention networks for document classification [C] //Proc of the 2016 Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA: ACL, 2016: 1480−1489
[8]	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30: 5998−6008
[9]	Cao Yue, Jin Hanqi, Wan Xiaojun, et al. Domain-adaptive neural automated essay scoring [C] //Proc of the 43rd Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2020: 1011−1020
[10]	Li Xia, Chen Minping, Nie Jianyun. SEDNN: Shared and enhanced deep neural network model for cross-prompt automated essay scoring[J]. Knowledge-Based Systems, 2020, 210: 106491 doi: 10.1016/j.knosys.2020.106491
[11]	Song Wei, Zhang Kai, Fu Ruiji, et al. Multi-stage pre-training for automated Chinese essay scoring [C] //Proc of the 2020 Conf on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg, PA: ACL, 2020: 6723−6733
[12]	Ridley R, He Liang, Dai Xinyu, et al. Automated cross-prompt scoring of essay traits [C] //Proc of the 35th Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 13745−13753
[13]	Chen Yuan, Li Xia. PMAES: Prompt-mapping contrastive learning for cross-prompt automated essay scoring [C] //Proc of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Stroudsburg, PA: ACL, 2023: 1489−1503
[14]	Ridley R, He Liang, Dai Xinyu, et al. Prompt agnostic essay scorer: A domain generalization approach to cross-prompt automated essay scoring [J]. arXiv preprint, arXiv: 2008.01441, 2020
[15]	Ganin Y, Ustinova E, Ajakan H, et al. Domain-adversarial training of neural networks[J]. Journal of Machine Learning Research, 2016, 17(59): 1−35
[16]	Chen Xinyang, Wang Sinan, Wang Jianmin, et al. Representation subspace distance for domain adaptation regression [C] //Proc of the 38th Int Conf on Machine Learning. New York: ACM, 2021: 1749−1759
[17]	Mathias S, Bhattacharyya P. ASAP++: Enriching the ASAP automated essay grading dataset with essay attribute scores [C] //Proc of the 11th Int Conf on Language Resources and Evaluation (LREC 2018). Stroudsburg, PA: ACL, 2018: 1169−1173
[18]	Larkey L S. Automatic essay grading using text categorization techniques [C] //Proc of the 21st Int 47th ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 1998: 90−95
[19]	Uto M, Xie Yikuan, Ueno M. Neural automated essay scoring incorporating handcrafted features [C] //Proc of the 28th Int Conf on Computational Linguistics. Stroudsburg, PA: ACL, 2020: 6077−6088
[20]	彭星源,柯登峰,赵知,等. 基于词汇评分的汉语作文自动评分[J]. 中文信息学报,2012,26(2):102−108 Peng Xingyuan, Ke Dengfeng, Zhao Zhi, et al. Automated Chinese essay scoring based on word scores[J]. Journal of Chinese Information Science, 2012, 26(2): 102−108 (in Chinese)
[21]	Taghipour K, Ng H T. A neural approach to automated essay scoring [C] //Proc of the 2016 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2016: 1882−1891
[22]	Alikaniotis D, Yannakoudakis H, Rei M. Automatic text scoring using neural networks [C] //Proc of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Stroudsburg, PA: ACL, 2016: 715−725
[23]	王耀华,李舟军,巢文涵,等. 基于文本语义离散度的自动作文评分关键技术研究[J]. 中文信息学报,2016,30(6):173−181 Wang Yaohua, Li Zhoujun, Chao Wenhan, et al. Research on key technology of automatic essay scoring based on text semantic dispersion[J]. Journal of Chinese Information Science, 2016, 30(6): 173−181 (in Chinese)
[24]	Tay Y, Phan M C, Tuan L A, et al. SKIPFLOW: Incorporating neural coherence features for end-to-end automatic text scoring [C] //Proc of the 32nd Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 5948−5955
[25]	付瑞吉,王栋,王士进,等. 面向作文自动评分的优美句识别[J]. 中文信息学报,2018,32(06):88−97 Fu Ruiji, Wang Dong, Wang Shijin, et al. Elgart sentence recognition for automated essay scoring[J]. Journal of Chinese Information Science, 2018, 32(06): 88−97 (in Chinese)
[26]	周险兵,樊小超,任鸽,等. 基于多层次语义特征的英文作文自动评分方法[J]. 计算机应用,2021,41(08):2205−2211 Zhou Xianbing, Fan Xiaochao, Ren Ge, et al. Automated English essay scoring method based on multi-level semantic features[J]. Journal of Computer Applications, 2021, 41(08): 2205−2211 (in Chinese)
[27]	Dong Fei, Zhang Yue, Yang Jie. Attention-based recurrent convolutional neural network for automatic essay scoring [C] //Proc of the 21st Conf on Computational Natural Language Learning. Stroudsburg, PA: ACL, 2017: 153−162
[28]	Xie Jiayi, Cai Kaiwei, Kong Li, et al. Automated essay scoring via pairwise contrastive regression [C] //Proc of the 29th Int Conf on Computational Linguistics. Stroudsburg, PA: ACL, 2022: 2724−2733
[29]	Lin Szu-Yin, Kung Yun-Chi, Leu Fang-Ye. Predictive intelligence in harmful news identification by BERT-based ensemble learning model with text sentiment analysis[J]. Information Processing & Management, 2022, 59(2): 102872
[30]	Rodriguez P U, Jafari A, Ormerod C M. Language models and automated essay scoring [J]. arXiv preprint, arXiv: 1909.09482, 2019
[31]	Briskilal J, Subalalitha C N. An ensemble model for classifying idioms and literal texts using BERT and RoBERTa[J]. Information Processing & Management, 2022, 59(1): 102756
[32]	Yang Ruosong, Cao Jiannong, Wen Zhiyuan, et al. Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of regression and ranking [C]//Proc of the 58th Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 1560−1569
[33]	Ormerod C M, Malhotra A, Jafari A. Automated essay scoring using efficient transformer-based language models[J]. arXiv preprint, arXiv: 2102.13136, 2021
[34]	Sethi A, Singh K. Natural language processing based automated essay scoring with parameter-efficient transformer approach [C] //Proc of the 6th Int Conf on Computing Methodologies and Communication (ICCMC). Piscataway, NJ: IEEE, 2022: 749−756
[35]	于明诚,党亚固,吴奇林,等. 基于多尺度上下文的英文作文自动评分研究[J]. 计算机工程,2024,50(3):259−266 Yu Mingcheng, Dang Yagu, Wu Qilin, et al. Research on automatic scoring for English essay based on multi-scale context[J]. Journal of Computer Engineering, 2024, 50(3): 259−266 (in Chinese)
[36]	Jin Cancan, He Ben, Hui Kai, et al. TDNN: A two-stage deep neural network for prompt-independent automated essay scoring [C] //Proc of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Stroudsburg, PA: ACL, 2018: 1088−1097
[37]	Do H, Kim Y, Lee G G. Prompt-and trait relation-aware cross-prompt essay trait scoring[C] //Proc of the 61st Association for Computational Linguistics. Stroudsburg, PA: ACL. 2023: 1538−1551
[38]	Farahani A, Voghoei S, Rasheed K, et al. A brief review of domain adaptation [J/OL]. Advances in Data Science and Information Engineering, 2021: 877−894 [2021-10-30]. https://link.springer.com/chapter/10.1007/978-3-030-71704-9_65
[39]	Wang Jindong, Lan Cuiling, Liu Cang, et al. Generalizing to unseen domains: A survey on domain generalization[J]. IEEE Transactions on Knowledge and Data Engineering, 2022, 35(8): 35−147
[40]	Li Haoliang, Pan S J, Wang Siqi, et al. Domain generalization with adversarial feature learning [C] //Proc of the IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 5400−5409
[41]	Matsuura T, Harada T. Domain generalization using a mixture of multiple latent domains [C] //Proc of the 34th Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 11749−11756
[42]	Zhu Kaijie, Hu Xixu, Wang Jindong, et al. Improving generalization of adversarial training via robust critical fine-tuning [C] //Proc of the IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 4424−4434
[43]	Singh N D, Croce F, Hein M. Revisiting adversarial training for imagenet: Architectures, training and generalization across threat models[J]. Advances in Neural Information Processing Systems, 2024, 36: 1−14
[44]	封筠,史屹琛,高宇豪,等. 二次解耦与活体特征渐进式对齐的域自适应人脸反欺诈[J]. 计算机研究与发展,2023,60(8):1727−1739 Feng Jun, Shi Yichen, Gao Yuhao, et al. Domain adaptation for face anti-spoofing based on dual disentanglement and liveness feature progressive alignment[J]. Journal of Computer Research and Development, 2023, 60(8): 1727−1739 (in Chinese)
[45]	Chen Xilun, Cardie C. Multinomial adversarial networks for multi-domain text classification [C] //Proc of the 2018 Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Stroudsburg, PA: ACL, 2018: 1226−1240

[1]	Zhang Wenhan, Liu Xiaoming, Yang Guan, Liu Jie. Cross-Domain Named Entity Recognition of Multi-Level Structured Semantic Knowledge Enhancement[J]. Journal of Computer Research and Development, 2023, 60(12): 2864-2876. DOI: 10.7544/issn1000-1239.202220413
[2]	Jia Xibin, Zeng Meng, Mi Qing, Hu Yongli. Domain Alignment Adversarial Unsupervised Cross-Domain Text Sentiment Analysis Algorithm[J]. Journal of Computer Research and Development, 2022, 59(6): 1255-1270. DOI: 10.7544/issn1000-1239.20210039
[3]	Li Xin, Li Zhemin, Wei Juhui, Yang Yating, Wang Hongxia. Cross-Domain Adaptive Learning Model Based on Feature Separation[J]. Journal of Computer Research and Development, 2022, 59(1): 105-117. DOI: 10.7544/issn1000-1239.20200765
[4]	Weng Zejia, Chen Jingjing, Jiang Yugang. On the Generalization of Face Forgery Detection with Domain Adversarial Learning[J]. Journal of Computer Research and Development, 2021, 58(7): 1476-1489. DOI: 10.7544/issn1000-1239.2021.20200803
[5]	Jia Yingxia, Lang Congyan, Feng Songhe. A Semantic Segmentation Method of Traffic Scene Based on Categories-Aware Domain Adaptation[J]. Journal of Computer Research and Development, 2020, 57(4): 876-887. DOI: 10.7544/issn1000-1239.2020.20190475
[6]	Liu Huan, Zheng Qinghua, Luo Minnan, Zhao Hongke, Xiao Yang, Lü Yanzhang. Cross-Domain Adversarial Learning for Zero-Shot Classification[J]. Journal of Computer Research and Development, 2019, 56(12): 2521-2535. DOI: 10.7544/issn1000-1239.2019.20190614
[7]	Wei Wenhong, Wang Jiahai, Tao Ming, Yuan Huaqiang. Multi-Objective Constrained Differential Evolution Using Generalized Opposition-Based Learning[J]. Journal of Computer Research and Development, 2016, 53(6): 1410-1421. DOI: 10.7544/issn1000-1239.2016.20150806
[8]	You Feng, Zhao Ruilian, Lü Shanshan. Output Domain Based Automatic Test Case Generation[J]. Journal of Computer Research and Development, 2016, 53(3): 541-549. DOI: 10.7544/issn1000-1239.2016.20148045
[9]	Zeng Fanzi, Xiao Degui, Li Renfa, and Luo Juan. Generalization Error Bound for the Multi-Class Classification Algorithm Based on the Analytical Center of Version Space[J]. Journal of Computer Research and Development, 2009, 46(6): 1003-1008.
[10]	Shu Fengdi, Zhao Yuzhu, Wang Jizhe, Li Mingshu. User-Driven Requirements Elicitation Method with the Support of Personalized Domain Knowledge[J]. Journal of Computer Research and Development, 2007, 44(6): 1044-1052.