# BERT
**\*\*\*\*\* New November 15th, 2018: SOTA SQuAD 2.0 System \*\*\*\*\***
We released code changes to reproduce our 83% F1 SQuAD 2.0 system, which is
currently 1st place on the leaderboard by 3%. See the SQuAD 2.0 section of the
README for details.
**\*\*\*\*\* New November 5th, 2018: Third-party PyTorch and Chainer versions of
BERT available \*\*\*\*\***
NLP researchers from HuggingFace made a
[PyTorch version of BERT available](https://github.com/huggingface/pytorch-pretrained-BERT)
which is compatible with our pre-trained checkpoints and is able to reproduce
our results. Sosuke Kobayashi also made a
[Chainer version of BERT available](https://github.com/soskek/bert-chainer)
(Thanks!) We were not involved in the creation or maintenance of the PyTorch
implementation so please direct any questions towards the authors of that
repository.
**\*\*\*\*\* New November 3rd, 2018: Multilingual and Chinese models available
\*\*\*\*\***
We have made two new BERT models available:
* **[`BERT-Base, Multilingual`](https://storage.googleapis.com/bert_models/2018_11_03/multilingual_L-12_H-768_A-12.zip)**:
102 languages, 12-layer, 768-hidden, 12-heads, 110M parameters
* **[`BERT-Base, Chinese`](https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip)**:
Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M
parameters
We use character-based tokenization for Chinese, and WordPiece tokenization for
all other languages. Both models should work out-of-the-box without any code
changes. We did update the implementation of `BasicTokenizer` in
`tokenization.py` to support Chinese character tokenization, so please update if
you forked it. However, we did not change the tokenization API.
For more, see the
[Multilingual README](https://github.com/google-research/bert/blob/master/multilingual.md).
**\*\*\*\*\* End new information \*\*\*\*\***
## Introduction
**BERT**, or **B**idirectional **E**ncoder **R**epresentations from
**T**ransformers, is a new method of pre-training language representations which
obtains state-of-the-art results on a wide array of Natural Language Processing
(NLP) tasks.
Our academic paper which describes BERT in detail and provides full results on a
number of tasks can be found here:
[https://arxiv.org/abs/1810.04805](https://arxiv.org/abs/1810.04805).
To give a few numbers, here are the results on the
[SQuAD v1.1](https://rajpurkar.github.io/SQuAD-explorer/) question answering
task:
SQuAD v1.1 Leaderboard (Oct 8th 2018) | Test EM | Test F1
------------------------------------- | :------: | :------:
1st Place Ensemble - BERT | **87.4** | **93.2**
2nd Place Ensemble - nlnet | 86.0 | 91.7
1st Place Single Model - BERT | **85.1** | **91.8**
2nd Place Single Model - nlnet | 83.5 | 90.1
And several natural language inference tasks:
System | MultiNLI | Question NLI | SWAG
----------------------- | :------: | :----------: | :------:
BERT | **86.7** | **91.1** | **86.3**
OpenAI GPT (Prev. SOTA) | 82.2 | 88.1 | 75.0
Plus many other tasks.
Moreover, these results were all obtained with almost no task-specific neural
network architecture design.
If you already know what BERT is and you just want to get started, you can
[download the pre-trained models](#pre-trained-models) and
[run a state-of-the-art fine-tuning](#fine-tuning-with-bert) in only a few
minutes.
## What is BERT?
BERT is a method of pre-training language representations, meaning that we train
a general-purpose "language understanding" model on a large text corpus (like
Wikipedia), and then use that model for downstream NLP tasks that we care about
(like question answering). BERT outperforms previous methods because it is the
first *unsupervised*, *deeply bidirectional* system for pre-training NLP.
*Unsupervised* means that BERT was trained using only a plain text corpus, which
is important because an enormous amount of plain text data is publicly available
on the web in many languages.
Pre-trained representations can also either be *context-free* or *contextual*,
and contextual representations can further be *unidirectional* or
*bidirectional*. Context-free models such as
[word2vec](https://www.tensorflow.org/tutorials/representation/word2vec) or
[GloVe](https://nlp.stanford.edu/projects/glove/) generate a single "word
embedding" representation for each word in the vocabulary, so `bank` would have
the same representation in `bank deposit` and `river bank`. Contextual models
instead generate a representation of each word that is based on the other words
in the sentence.
BERT was built upon recent work in pre-training contextual representations —
including [Semi-supervised Sequence Learning](https://arxiv.org/abs/1511.01432),
[Generative Pre-Training](https://blog.openai.com/language-unsupervised/),
[ELMo](https://allennlp.org/elmo), and
[ULMFit](http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html)
— but crucially these models are all *unidirectional* or *shallowly
bidirectional*. This means that each word is only contextualized using the words
to its left (or right). For example, in the sentence `I made a bank deposit` the
unidirectional representation of `bank` is only based on `I made a` but not
`deposit`. Some previous work does combine the representations from separate
left-context and right-context models, but only in a "shallow" manner. BERT
represents "bank" using both its left and right context — `I made a ... deposit`
— starting from the very bottom of a deep neural network, so it is *deeply
bidirectional*.
BERT uses a simple approach for this: We mask out 15% of the words in the input,
run the entire sequence through a deep bidirectional
[Transformer](https://arxiv.org/abs/1706.03762) encoder, and then predict only
the masked words. For example:
```
Input: the man went to the [MASK1] . he bought a [MASK2] of milk.
Labels: [MASK1] = store; [MASK2] = gallon
```
In order to learn relationships between sentences, we also train on a simple
task which can be generated from any monolingual corpus: Given two sentences `A`
and `B`, is `B` the actual next sentence that comes after `A`, or just a random
sentence from the corpus?
```
Sentence A: the man went to the store .
Sentence B: he bought a gallon of milk .
Label: IsNextSentence
```
```
Sentence A: the man went to the store .
Sentence B: penguins are flightless .
Label: NotNextSentence
```
We then train a large model (12-layer to 24-layer Transformer) on a large corpus
(Wikipedia + [BookCorpus](http://yknzhu.wixsite.com/mbweb)) for a long time (1M
update steps), and that's BERT.
Using BERT has two stages: *Pre-training* and *fine-tuning*.
**Pre-training** is fairly expensive (four days on 4 to 16 Cloud TPUs), but is a
one-time procedure for each language (current models are English-only, but
multilingual models will be released in the near future). We are releasing a
number of pre-trained models from the paper which were pre-trained at Google.
Most NLP researchers will never need to pre-train their own model from scratch.
**Fine-tuning** is inexpensive. All of the results in the paper can be
replicated in at most 1 hour on a single Cloud TPU, or a few hours on a GPU,
starting from the exact same pre-trained model. SQuAD, for example, can be
trained in around 30 minutes on a single Cloud TPU to achieve a Dev F1 score of
91.0%, which is the single system state-of-the-art.
The other important aspect of BERT is that it can be adapted to many types of
NLP tasks very easily. In the paper, we demonstrate state-of-the-art results on
sentence-level (e.g., SST-2), sentence-pair-level (e.g., MultiNLI), word-level
(e.g., NER), and span-level (e.g., SQuAD) tasks with almost no task-specific
modifications.
## What has been released in this repository?
We are releasing the following:
* TensorFlow code for the BERT model architecture (
没有合适的资源?快使用搜索试试~ 我知道了~
基于bert的lstm-crf-ner模型.zip
共64个文件
py:26个
pyc:14个
ipynb:7个
需积分: 5 1 下载量 31 浏览量
2024-05-22
13:16:14
上传
评论
收藏 227KB ZIP 举报
温馨提示
LSTM (Long Short-Term Memory) 是一种特殊的循环神经网络(RNN)架构,用于处理具有长期依赖关系的序列数据。传统的RNN在处理长序列时往往会遇到梯度消失或梯度爆炸的问题,导致无法有效地捕捉长期依赖。LSTM通过引入门控机制(Gating Mechanism)和记忆单元(Memory Cell)来克服这些问题。 以下是LSTM的基本结构和主要组件: 记忆单元(Memory Cell):记忆单元是LSTM的核心,用于存储长期信息。它像一个传送带一样,在整个链上运行,只有一些小的线性交互。信息很容易地在其上保持不变。 输入门(Input Gate):输入门决定了哪些新的信息会被加入到记忆单元中。它由当前时刻的输入和上一时刻的隐藏状态共同决定。 遗忘门(Forget Gate):遗忘门决定了哪些信息会从记忆单元中被丢弃或遗忘。它也由当前时刻的输入和上一时刻的隐藏状态共同决定。 输出门(Output Gate):输出门决定了哪些信息会从记忆单元中输出到当前时刻的隐藏状态中。同样地,它也由当前时刻的输入和上一时刻的隐藏状态共同决定。 LSTM的计算过程可以大致描述为: 通过遗忘门决定从记忆单元中丢弃哪些信息。 通过输入门决定哪些新的信息会被加入到记忆单元中。 更新记忆单元的状态。 通过输出门决定哪些信息会从记忆单元中输出到当前时刻的隐藏状态中。 由于LSTM能够有效地处理长期依赖关系,它在许多序列建模任务中都取得了很好的效果,如语音识别、文本生成、机器翻译、时序预测等。
资源推荐
资源详情
资源评论
收起资源包目录
基于bert的lstm-crf-ner模型.zip (64个子文件)
content
__init__.py 0B
train
tf_metrics.pyc 8KB
mymetrics.py 965B
__init__.py 0B
bert_lstm_ner_cg_estimator.pyc 22KB
tf_metrics.py 8KB
models.pyc 9KB
conlleval.py 11KB
models.py 10KB
conlleval.pl 13KB
train_helper.pyc 4KB
train_helper.py 5KB
lstm_crf_layer.pyc 6KB
lstm_crf_layer.py 7KB
__init__.pyc 153B
bert_lstm_ner_cg_estimator.py 37KB
bert_lstm_ner_cg_variable.py 38KB
__pycache__
lstm_crf_layer.cpython-36.pyc 5KB
train_helper.cpython-36.pyc 3KB
bert_lstm_ner.cpython-36.pyc 20KB
conlleval.cpython-36.pyc 7KB
__init__.cpython-36.pyc 144B
models.cpython-36.pyc 7KB
bert_lstm_ner_cg_estimator.cpython-36.pyc 19KB
tf_metrics.cpython-36.pyc 8KB
test.py 370B
test_slice.ipynb 2KB
run.py 3KB
data_demo
dev.txt 13KB
test.txt 13KB
train.txt 13KB
bert
modeling_test.py 9KB
__init__.py 616B
input.txt 17B
sentence2vec.py 14KB
extract_features.py 14KB
LICENSE 11KB
sentence2veclib.py 13KB
run_pretraining.py 18KB
sample_text.txt 4KB
CONTRIBUTING.md 1KB
cmd.sh 748B
optimization_test.py 2KB
modeling.py 37KB
optimization.py 6KB
output.json 14KB
tokenization_test.py 4KB
tokenization.py 10KB
requirements.txt 110B
create_pretraining_data.py 15KB
.gitignore 1KB
run.sh 482B
README.md 40KB
multilingual.md 11KB
run_classifier.py 31KB
sentence2vec.json 452B
run_squad.py 45KB
.gitignore 127B
test_tensor.ipynb 10KB
.ipynb_checkpoints
test_tensor-checkpoint.ipynb 72B
test_slice-checkpoint.ipynb 72B
tune_model-checkpoint.ipynb 72B
bert-lstm-crf-ner-checkpoint.ipynb 72B
tune_model.ipynb 89KB
共 64 条
- 1
资源评论
生瓜蛋子
- 粉丝: 3916
- 资源: 7441
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功