ISSN 1000-1239 CN 11-1777/TP

计算机研究与发展 ›› 2021, Vol. 58 ›› Issue (8): 1761-1772.doi: 10.7544/issn1000-1239.2021.20210298

所属专题: 2021人工智能前沿进展专题

• 人工智能 • 上一篇    下一篇

用于金融文本挖掘的多任务学习预训练金融语言模型

刘壮1,刘畅2,Wayne,Lin3,赵军4   

  1. 1(东北财经大学应用金融与行为科学学院 辽宁大连 116025);2(中国石油物资采购中心 沈阳 110031);3(南加州大学计算机学院 美国加利福尼亚州洛杉矶 90007);4(IBM研究院 北京 100101) (liuzhuang@dufe.edu.cn)
  • 出版日期: 2021-08-01
  • 基金资助: 
    辽宁省教育厅2021年度高等学校基本科研项目(面上项目);教育部产学合作协同育人项目(202002037015)

Pretraining Financial Language Model with Multi-Task Learning for Financial Text Mining

Liu Zhuang1, Liu Chang2, Wayne Lin3, Zhao Jun4   

  1. 1(School of Applied Finance and Behavioral Science, Dongbei University of Finance and Economics, Dalian, Liaoning 116025);2(China Petroleum Materials Procurement Center, Shenyang 110031);3(School of Computer Science, University of Southern California, Los Angeles, CA, USA 90007);4(IBM Research, Beijing 100101)
  • Online: 2021-08-01
  • Supported by: 
    This work was supported by the Basic Scientific Research Project (General Program) of Department of Education of Liaoning Province and the University-Industry Collaborative Education Program of the Ministry of Education of China (202002037015).

摘要: 近年来,机器学习,尤其是深度神经网络方法促进了金融文本挖掘研究的发展,在金融科技(Fintech)领域起着越来越重要的作用.如何从金融大数据中提取有价值的信息已经成为学术界和工业界一个非常有挑战的研究.由于深度神经网络需要大量标注数据,但是金融领域缺乏训练数据,因此,将深度学习应用于金融大数据挖掘并没有取得很好效果.为了更好地解决该问题,利用自监督学习和多任务学习等深度学习领域最新的思想和技术,提出了基于BERT模型架构的开源金融预训练语言模型F-BERT.F-BERT是一种针对特定领域(金融文本数据)的预训练语言模型,它通过在大型金融语料库上进行无监督训练得到.基于BERT架构,F-BERT可以有效地自动将知识从金融大数据中提取出并记忆在模型中,而无需进行特定于金融任务的模型结构修改,从而可以直接将其应用于下游各种金融领域任务,包括股票涨跌预测、金融情绪分类、金融智能客服等.在金融关系提取、金融情绪分类、金融智能问答任务上的大量实验表明了F-BERT模型的有效性和鲁棒性.同时,F-BERT在这3个有代表性的金融文本挖掘任务上均取得了很高的模型准确率,进一步验证了模型的性能.

关键词: BERT, 金融文本挖掘, 多任务学习, 预训练, 迁移学习, 金融科技

Abstract: Financial text mining is becoming increasingly important as the number of financial documents rapidly grows. With the progress in machine learning, extracting valuable information from financial literature has gained attention among researchers, and deep learning has boosted the development of effective financial text mining models. However, as deep learning models require a large amount of labeled training data, applying deep learning to financial text mining is often unsuccessful due to the lack of training data in financial fields. Recent researches on training contextualized language representation models on text corpora shed light on the possibility of leveraging a large number of unlabeled financial text corpora. We introduce F-BERT (BERT for financial text mining), which is a domain specific language representation model pre-trained on large-scale financial corpora. Based on the BERT architecture, F-BERT effectively transfers the knowledge from a large amount of financial texts to financial text mining models with minimal task-specific architecture modifications. The results show that our F-BERT outperforms most current state-of-the-art models, which demonstrates the effectiveness and robustness of the proposed F-BERT.

Key words: BERT, financial text mining, multi-task learning, pre-training, transfer learning, fintech

中图分类号: