高级检索
    刘壮, 刘畅, Wayne Lin, 赵军. 用于金融文本挖掘的多任务学习预训练金融语言模型[J]. 计算机研究与发展, 2021, 58(8): 1761-1772. DOI: 10.7544/issn1000-1239.2021.20210298
    引用本文: 刘壮, 刘畅, Wayne Lin, 赵军. 用于金融文本挖掘的多任务学习预训练金融语言模型[J]. 计算机研究与发展, 2021, 58(8): 1761-1772. DOI: 10.7544/issn1000-1239.2021.20210298
    Liu Zhuang, Liu Chang, Wayne Lin, Zhao Jun. Pretraining Financial Language Model with Multi-Task Learning for Financial Text Mining[J]. Journal of Computer Research and Development, 2021, 58(8): 1761-1772. DOI: 10.7544/issn1000-1239.2021.20210298
    Citation: Liu Zhuang, Liu Chang, Wayne Lin, Zhao Jun. Pretraining Financial Language Model with Multi-Task Learning for Financial Text Mining[J]. Journal of Computer Research and Development, 2021, 58(8): 1761-1772. DOI: 10.7544/issn1000-1239.2021.20210298

    用于金融文本挖掘的多任务学习预训练金融语言模型

    Pretraining Financial Language Model with Multi-Task Learning for Financial Text Mining

    • 摘要: 近年来,机器学习,尤其是深度神经网络方法促进了金融文本挖掘研究的发展,在金融科技(Fintech)领域起着越来越重要的作用.如何从金融大数据中提取有价值的信息已经成为学术界和工业界一个非常有挑战的研究.由于深度神经网络需要大量标注数据,但是金融领域缺乏训练数据,因此,将深度学习应用于金融大数据挖掘并没有取得很好效果.为了更好地解决该问题,利用自监督学习和多任务学习等深度学习领域最新的思想和技术,提出了基于BERT模型架构的开源金融预训练语言模型F-BERT.F-BERT是一种针对特定领域(金融文本数据)的预训练语言模型,它通过在大型金融语料库上进行无监督训练得到.基于BERT架构,F-BERT可以有效地自动将知识从金融大数据中提取出并记忆在模型中,而无需进行特定于金融任务的模型结构修改,从而可以直接将其应用于下游各种金融领域任务,包括股票涨跌预测、金融情绪分类、金融智能客服等.在金融关系提取、金融情绪分类、金融智能问答任务上的大量实验表明了F-BERT模型的有效性和鲁棒性.同时,F-BERT在这3个有代表性的金融文本挖掘任务上均取得了很高的模型准确率,进一步验证了模型的性能.

       

      Abstract: Financial text mining is becoming increasingly important as the number of financial documents rapidly grows. With the progress in machine learning, extracting valuable information from financial literature has gained attention among researchers, and deep learning has boosted the development of effective financial text mining models. However, as deep learning models require a large amount of labeled training data, applying deep learning to financial text mining is often unsuccessful due to the lack of training data in financial fields. Recent researches on training contextualized language representation models on text corpora shed light on the possibility of leveraging a large number of unlabeled financial text corpora. We introduce F-BERT (BERT for financial text mining), which is a domain specific language representation model pre-trained on large-scale financial corpora. Based on the BERT architecture, F-BERT effectively transfers the knowledge from a large amount of financial texts to financial text mining models with minimal task-specific architecture modifications. The results show that our F-BERT outperforms most current state-of-the-art models, which demonstrates the effectiveness and robustness of the proposed F-BERT.

       

    /

    返回文章
    返回