ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, Haifeng Wang
Baidu Inc., Beijing, China
{sunyu02, wangshuohuan, tianhao, wu hua,wanghaifeng}@baidu.com
Abstract
Recently pre-trained models have achieved state-of-the-art
results in various language understanding tasks. Current pre-
training procedures usually focus on training the model with
several simple tasks to grasp the co-occurrence of words or
sentences. However, besides co-occurring information, there
exists other valuable lexical, syntactic and semantic infor-
mation in training corpora, such as named entities, semantic
closeness and discourse relations. In order to extract the lexi-
cal, syntactic and semantic information from training corpora,
we propose a continual pre-training framework named ERNIE
2.0 which incrementally builds pre-training tasks and then
learn pre-trained models on these constructed tasks via contin-
ual multi-task learning. Based on this framework, we construct
several tasks and train the ERNIE 2.0 model to capture lexical,
syntactic and semantic aspects of information in the train-
ing data. Experimental results demonstrate that ERNIE 2.0
model outperforms BERT and XLNet on 16 tasks including
English tasks on GLUE benchmarks and several similar tasks
in Chinese. The source codes and pre-trained models have
been released at https://github.com/PaddlePaddle/ERNIE.
Introduction
Pre-trained language representations such as ELMo(Peters et
al
.
2018), OpenAI GPT(Radford et al
.
2018), BERT (Devlin
et al
.
2018), ERNIE 1.0 (Sun et al
.
2019)
1
and XLNet(Yang
et al
.
2019) have been proven to be effective for improving
the performances of various natural language understanding
tasks including sentiment classification (Socher et al
.
2013),
natural language inference (Bowman et al
.
2015), named
entity recognition (Sang and De Meulder 2003) and so on.
Generally the pre-training of models often train the model
based on the co-occurrence of words and sentences. While
in fact, there are other lexical, syntactic and semantic infor-
mation worth examining in training corpora other than co-
occurrence. For example, named entities like person names,
location names, and organization names, may contain con-
ceptual information. Information like sentence order and
Copyright
c
2020, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.
1
In order to distinguish ERNIE 2.0 framework and the ERNIE
model, the latter is referred to as ERNIE 1.0.(Sun et al. 2019)
sentence proximity enables the models to learn structure-
aware representations. And semantic similarity at the doc-
ument level or discourse relations among sentences allow
the models to learn semantic-aware representations. In or-
der to discover all valuable information in training corpora,
be it lexical, syntactic or semantic representations, we pro-
pose a continual pre-training framework named ERNIE 2.0
which could incrementally build and train a large variety of
pre-training tasks through continual multi-task learning.
Our ERNIE framework supports the introduction of vari-
ous customized tasks continually, which is realized through
continual multi-task learning. When given one or more new
tasks, the continual multi-task learning method simultane-
ously trains the newly-introduced tasks together with the
original tasks in an efficient way, without forgetting previ-
ously learned knowledge. In this way, our framework can
incrementally train the distributed representations based on
the previously trained parameters that it grasped. Moreover,
in this framework, all the tasks share the same encoding net-
works, thus making the encoding of lexical, syntactic and
semantic information across different tasks possible.
In summary, our contributions are as follows:
•
We propose a continual pre-training framework ERNIE
2.0, which efficiently supports customized training tasks
and continual multi-task learning in an incremental way.
•
We construct three kinds of unsupervised language pro-
cessing tasks to verify the effectiveness of the proposed
framework. Experimental results demonstrate that ERNIE
2.0 achieves significant improvements over BERT and XL-
Net on 16 tasks including English GLUE benchmarks and
several Chinese tasks.
•
Our fine-tuning code of ERNIE 2.0 and models pre-trained
on English corpora are available at https://github.com/
PaddlePaddle/ERNIE.
Related Work
Unsupervised Learning for Language
Representation
It is effective to learn general language representation by
pre-training a language model with a large amount of unan-
arXiv:1907.12412v2 [cs.CL] 21 Nov 2019
剩余7页未读，继续阅读
资源评论
lizzy05
- 粉丝: 77
- 资源: 21
最新资源
- 2015-2024年上市公司商道融绿esg评级数据（年度）
- DeepSeek：通用人工智能从入门到精通的技术解析与应用指南
- 离散扩展龙伯格观测器：扰动补偿功能下的鲁棒性能优化及动态响应增强策略,离散扩展龙伯格观测器：具有扰动补偿功能的高鲁棒性预测控制系统,一种具有扰动补偿功能的离散扩展龙伯格观测器，有较好的参数摄动扰动抑制
- 无刷直流电机BLDC三闭环控制系统的Matlab Simulink仿真模型搭建：原理、波形记录与参数详解,无刷直流电机BLDC三闭环控制系统的Matlab Simulink仿真模型搭建：原理、波形记录
- 基于Python的Django-vue基于spark的短视频推荐系统的设计与实现源码-说明文档-演示视频.zip
- DeepSeek写的重力球迷宫手机小游戏
- 单相变压器绕组与铁芯振动形变仿真模型：洛伦兹力与磁致伸缩效应下的动态响应分析,COMSOL单相变压器绕组与铁芯振动形变仿真模型：基于洛伦兹力与磁致伸缩效应的时域分析,comsol的单相变压器绕组及铁芯
- 新兴经济体二氧化碳排放报告2024.pdf
- 激光熔覆技术：COMSOL模拟建模与视频教程服务，助力激光研究人员与工程师的专业提升,激光熔覆技术：COMSOL软件下的建模与视频教程应用指南,COMSOL 激光 激光熔覆 名称：激光熔覆 适用人群：
- 2000-2023年上市公司价值链升级数据（含原始数据+计算代码+结果）
- COMSOL仿真下的钢架无损超声检测：焊接区域及周边缺陷识别技术，角钢梁纵波转横波检测原理揭秘,Comsol仿真技术下的钢架无损超声检测：角钢梁缺陷的精准识别与定位,Comsol仿真钢架无损超声检测
- 基于FPGA的图像坏点像素修复算法实现及Matlab辅助验证：探索其原理、测试与使用视频教程 注：标题中的“可刀”一词在此上下文中并无实际意义，因此未被包含在标题中 标题长度符合要求，并尽量简洁明了
- 2008-2022年各省环境污染指数数据（原始数据+结果）.xlsx
- zhaopin_mzhan.apk
- 权威科研机构发布钢轨表面缺陷检测数据集，含400张图像和8种类别缺陷，mAP达0.8，附赠lunwen,钢轨表面缺陷检测数据集：包含400张图片与八种缺陷类别，适用于目标检测算法训练与研究 ,钢轨表面
- C形永磁辅助同步磁阻电机Maxwell参数化模型：转子手绘设计及关键参数优化分析,基于Maxwell参数化模型的C形永磁辅助同步磁阻电机研究：转子手绘非UDP模块的参数化设计及优化分析,C形永磁辅助同
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
