BERT: Pre-training of Deep Bidirectional Transformers for
Language Understanding
Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova
Google AI Language
{jacobdevlin,mingweichang,kentonl,kristout}@google.com
Abstract
We introduce a new language representa-
tion model called BERT, which stands for
Bidirectional Encoder Representations from
Transformers. Unlike recent language repre-
sentation models (Peters et al., 2018; Radford
et al., 2018), BERT is designed to pre-train
deep bidirectional representations by jointly
conditioning on both left and right context in
all layers. As a result, the pre-trained BERT
representations can be fine-tuned with just one
additional output layer to create state-of-the-
art models for a wide range of tasks, such
as question answering and language inference,
without substantial task-specific architecture
modifications.
BERT is conceptually simple and empirically
powerful. It obtains new state-of-the-art re-
sults on eleven natural language processing
tasks, including pushing the GLUE bench-
mark to 80.4% (7.6% absolute improvement),
MultiNLI accuracy to 86.7% (5.6% abso-
lute improvement) and the SQuAD v1.1 ques-
tion answering Test F1 to 93.2 (1.5 absolute
improvement), outperforming human perfor-
mance by 2.0.
1 Introduction
Language model pre-training has shown to be ef-
fective for improving many natural language pro-
cessing tasks (Dai and Le, 2015; Peters et al.,
2017, 2018; Radford et al., 2018; Howard and
Ruder, 2018). These tasks include sentence-level
tasks such as natural language inference (Bow-
man et al., 2015; Williams et al., 2018) and para-
phrasing (Dolan and Brockett, 2005), which aim
to predict the relationships between sentences by
analyzing them holistically, as well as token-level
tasks such as named entity recognition (Tjong
Kim Sang and De Meulder, 2003) and SQuAD
question answering (Rajpurkar et al., 2016), where
models are required to produce fine-grained output
at the token-level.
There are two existing strategies for apply-
ing pre-trained language representations to down-
stream tasks: feature-based and fine-tuning. The
feature-based approach, such as ELMo (Peters
et al., 2018), uses tasks-specific architectures that
include the pre-trained representations as addi-
tional features. The fine-tuning approach, such as
the Generative Pre-trained Transformer (OpenAI
GPT) (Radford et al., 2018), introduces minimal
task-specific parameters, and is trained on the
downstream tasks by simply fine-tuning the pre-
trained parameters. In previous work, both ap-
proaches share the same objective function dur-
ing pre-training, where they use unidirectional lan-
guage models to learn general language represen-
tations.
We argue that current techniques severely re-
strict the power of the pre-trained representations,
especially for the fine-tuning approaches. The ma-
jor limitation is that standard language models are
unidirectional, and this limits the choice of archi-
tectures that can be used during pre-training. For
example, in OpenAI GPT, the authors use a left-
to-right architecture, where every token can only
attended to previous tokens in the self-attention
layers of the Transformer (Vaswani et al., 2017).
Such restrictions are sub-optimal for sentence-
level tasks, and could be devastating when ap-
plying fine-tuning based approaches to token-level
tasks such as SQuAD question answering (Ra-
jpurkar et al., 2016), where it is crucial to incor-
porate context from both directions.
In this paper, we improve the fine-tuning based
approaches by proposing BERT: Bidirectional
Encoder Representations from Transformers.
BERT addresses the previously mentioned uni-
directional constraints by proposing a new
pre-training objective: the “masked language
arXiv:1810.04805v1 [cs.CL] 11 Oct 2018
剩余13页未读，继续阅读
资源评论
