第15卷第4期 智能系统学报 Vol.15 No.4 2020年7月 CAAI Transactions on Intelligent Systems Jul.2020 D0L:10.11992tis.202003003 加入自注意力机制的BERT命名实体识别模型 毛明毅'，吴晨'，钟义信2，陈志成2 (1.北京工商大学计算机与信息工程学院，北京100048：2.北京邮电大学计算机学院，北京100876) 摘要：命名实体识别属于自然语言处理领域词法分析中的一部分，是计算机正确理解自然语言的基础。为了 加强模型对命名实体的识别效果，本文使用预训练模型BERT(bidirectional encoder representation from trans-- formers)作为模型的嵌入层，并针对BERT微调训练对计算机性能要求较高的问题，采用了固定参数嵌入的方 式对BERT进行应用，搭建了BERT-BiLSTM-CRF模型。并在该模型的基础上进行了两种改进实验。方法一， 继续增加自注意力(self-attention)层，实验结果显示，自注意力层的加入对模型的识别效果提升不明显。方法 二，减小BERT模型嵌入层数。实验结果显示，适度减少BERT嵌人层数能够提升模型的命名实体识别准确 性，同时又节约了模型的整体训练时间。采用9层嵌入时，在MSRA中文数据集上F1值提升至94.79%，在 Weibo中文数据集上F1值达到了68.82%。 关键词：命名实体识别：BERT;自注意力机制；深度学习；条件随机场；自然语言处理；双向长短期记忆网络；序 列标注 中图分类号：TP391文献标志码：A文章编号：1673-4785(2020)04-0772-08 中文引用格式：毛明毅，吴晨，钟义信，等.加入自注意力机制的BERT命名实体识别模型.智能系统学报，2020,15(4)： 772-779. 英文引用格式：MAO Mingyi,.WU Chen,ZHONG Yixin,.etal.BERT named entity recognition model with self--attention mechan ism[J].CAAI transactions on intelligent systems,2020,15(4):772-779. BERT named entity recognition model with self-attention mechanism MAO Mingyi',WU Chen',ZHONG Yixin',CHEN Zhicheng" (1.School of Computer and Information Engineering,Beijing Technology and Business University,Beijing 100048,China;2.School of Computer,Beijing University of Posts and Telecommunications,Beijing 100876,China) Abstract:Named entity recognition is a part of lexical analysis in the field of natural language processing.It is the basis for a computer to correctly understand natural language.In order to strengthen the recognition effect of the model on named entities,in this study,the pre-trained model BERT(bidirectional encoder representation from transformers)was used as the embedding layer of the model,and fixed parameter embedding was adopted to solve the problem of high computer performance required for BERT fine-tuning training.A BERT-BiLSTM-CRF model was built,and on the basis of this model,two improved experiments were carried out.Method one is to continue to add a self-attention layer. Experimental results show that the addition of the self-attention layer does not significantly improve the recognition ef- fect of the model.Method two is to reduce the number of embedding layers of the BERT model.Experimental results show that moderately reducing the number of BERT embedding layers can improve the model's named entity recogni- tion accuracy,while saving the overall training time of the model.When using 9-layer embedding,thevalue on the MSRA Chinese data set increased to 94.79%,and thevalue on the Weibo Chinese data set reached 68.82%. Keywords:named entity recognition;bidirectional encoder representation from transformers;self-attention mechanism; deep learning;conditional random field;natural language processing:bi-directional long short-term memory;sequence tagging 收稿日期：2020-03-02. 基金项目：北京市自然科学基金项目(4202016) 命名实体识别NER(named entity recognition) 通信作者：毛明毅.E-mail:maomy@h.btbu.edu.cn. 是自然语言处理研究领域的基础性工作之一，任DOI: 10.11992/tis.202003003 加入自注意力机制的 BERT 命名实体识别模型 毛明毅1 ，吴晨1 ，钟义信2 ，陈志成2 （1. 北京工商大学 计算机与信息工程学院，北京 100048; 2. 北京邮电大学 计算机学院，北京 100876） 摘 要：命名实体识别属于自然语言处理领域词法分析中的一部分，是计算机正确理解自然语言的基础。为了 加强模型对命名实体的识别效果，本文使用预训练模型 BERT(bidirectional encoder representation from trans￾formers) 作为模型的嵌入层，并针对 BERT 微调训练对计算机性能要求较高的问题，采用了固定参数嵌入的方 式对 BERT 进行应用，搭建了 BERT-BiLSTM-CRF 模型。并在该模型的基础上进行了两种改进实验。方法一， 继续增加自注意力 (self-attention) 层，实验结果显示，自注意力层的加入对模型的识别效果提升不明显。方法 二，减小 BERT 模型嵌入层数。实验结果显示，适度减少 BERT 嵌入层数能够提升模型的命名实体识别准确 性，同时又节约了模型的整体训练时间。采用 9 层嵌入时，在 MSRA 中文数据集上 F1 值提升至 94.79%，在 Weibo 中文数据集上 F1 值达到了 68.82%。 关键词：命名实体识别；BERT；自注意力机制；深度学习；条件随机场；自然语言处理；双向长短期记忆网络；序 列标注 中图分类号：TP391 文献标志码：A 文章编号：1673−4785(2020)04−0772−08 中文引用格式：毛明毅, 吴晨, 钟义信, 等. 加入自注意力机制的 BERT 命名实体识别模型 [J]. 智能系统学报, 2020, 15(4): 772–779. 英文引用格式：MAO Mingyi, WU Chen, ZHONG Yixin, et al. BERT named entity recognition model with self-attention mechan￾ism[J]. CAAI transactions on intelligent systems, 2020, 15(4): 772–779. BERT named entity recognition model with self-attention mechanism MAO Mingyi1 ，WU Chen1 ，ZHONG Yixin2 ，CHEN Zhicheng2 (1. School of Computer and Information Engineering, Beijing Technology and Business University, Beijing 100048, China; 2. School of Computer, Beijing University of Posts and Telecommunications, Beijing 100876, China) Abstract: Named entity recognition is a part of lexical analysis in the field of natural language processing. It is the basis for a computer to correctly understand natural language. In order to strengthen the recognition effect of the model on named entities, in this study, the pre-trained model BERT (bidirectional encoder representation from transformers) was used as the embedding layer of the model, and fixed parameter embedding was adopted to solve the problem of high computer performance required for BERT fine-tuning training. A BERT-BiLSTM-CRF model was built, and on the basis of this model, two improved experiments were carried out. Method one is to continue to add a self-attention layer. Experimental results show that the addition of the self-attention layer does not significantly improve the recognition ef￾fect of the model. Method two is to reduce the number of embedding layers of the BERT model. Experimental results show that moderately reducing the number of BERT embedding layers can improve the model’s named entity recogni￾tion accuracy, while saving the overall training time of the model. When using 9-layer embedding, thevalue on the MSRA Chinese data set increased to 94.79%, and thevalue on the Weibo Chinese data set reached 68.82%. Keywords: named entity recognition; bidirectional encoder representation from transformers; self-attention mechanism; deep learning; conditional random field; natural language processing; bi-directional long short-term memory; sequence tagging 命名实体识别 NER(named entity recognition) 是自然语言处理研究领域的基础性工作之一，任 收稿日期：2020−03−02. 基金项目：北京市自然科学基金项目 (4202016). 通信作者：毛明毅. E-mail：maomy@th.btbu.edu.cn. 第 15 卷第 4 期 智 能 系 统 学 报 Vol.15 No.4 2020 年 7 月 CAAI Transactions on Intelligent Systems Jul. 2020