# BERT
**\*\*\*\*\* New May 31st, 2019: Whole Word Masking Models \*\*\*\*\***
This is a release of several new models which were the result of an improvement
the pre-processing code.
In the original pre-processing code, we randomly select WordPiece tokens to
mask. For example:
`Input Text: the man jumped up , put his basket on phil ##am ##mon ' s head`
`Original Masked Input: [MASK] man [MASK] up , put his [MASK] on phil
[MASK] ##mon ' s head`
The new technique is called Whole Word Masking. In this case, we always mask
*all* of the the tokens corresponding to a word at once. The overall masking
rate remains the same.
`Whole Word Masked Input: the man [MASK] up , put his basket on [MASK] [MASK]
[MASK] ' s head`
The training is identical -- we still predict each masked WordPiece token
independently. The improvement comes from the fact that the original prediction
task was too 'easy' for words that had been split into multiple WordPieces.
This can be enabled during data generation by passing the flag
`--do_whole_word_mask=True` to `create_pretraining_data.py`.
Pre-trained models with Whole Word Masking are linked below. The data and
training were otherwise identical, and the models have identical structure and
vocab to the original models. We only include BERT-Large models. When using
these models, please make it clear in the paper that you are using the Whole
Word Masking variant of BERT-Large.
* **[`BERT-Large, Uncased (Whole Word Masking)`](https://storage.googleapis.com/bert_models/2019_05_30/wwm_uncased_L-24_H-1024_A-16.zip)**:
24-layer, 1024-hidden, 16-heads, 340M parameters
* **[`BERT-Large, Cased (Whole Word Masking)`](https://storage.googleapis.com/bert_models/2019_05_30/wwm_cased_L-24_H-1024_A-16.zip)**:
24-layer, 1024-hidden, 16-heads, 340M parameters
Model | SQUAD 1.1 F1/EM | Multi NLI Accuracy
---------------------------------------- | :-------------: | :----------------:
BERT-Large, Uncased (Original) | 91.0/84.3 | 86.05
BERT-Large, Uncased (Whole Word Masking) | 92.8/86.7 | 87.07
BERT-Large, Cased (Original) | 91.5/84.8 | 86.09
BERT-Large, Cased (Whole Word Masking) | 92.9/86.7 | 86.46
**\*\*\*\*\* New February 7th, 2019: TfHub Module \*\*\*\*\***
BERT has been uploaded to [TensorFlow Hub](https://tfhub.dev). See
`run_classifier_with_tfhub.py` for an example of how to use the TF Hub module,
or run an example in the browser on
[Colab](https://colab.sandbox.google.com/github/google-research/bert/blob/master/predicting_movie_reviews_with_bert_on_tf_hub.ipynb).
**\*\*\*\*\* New November 23rd, 2018: Un-normalized multilingual model + Thai +
Mongolian \*\*\*\*\***
We uploaded a new multilingual model which does *not* perform any normalization
on the input (no lower casing, accent stripping, or Unicode normalization), and
additionally inclues Thai and Mongolian.
**It is recommended to use this version for developing multilingual models,
especially on languages with non-Latin alphabets.**
This does not require any code changes, and can be downloaded here:
* **[`BERT-Base, Multilingual Cased`](https://storage.googleapis.com/bert_models/2018_11_23/multi_cased_L-12_H-768_A-12.zip)**:
104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters
**\*\*\*\*\* New November 15th, 2018: SOTA SQuAD 2.0 System \*\*\*\*\***
We released code changes to reproduce our 83% F1 SQuAD 2.0 system, which is
currently 1st place on the leaderboard by 3%. See the SQuAD 2.0 section of the
README for details.
**\*\*\*\*\* New November 5th, 2018: Third-party PyTorch and Chainer versions of
BERT available \*\*\*\*\***
NLP researchers from HuggingFace made a
[PyTorch version of BERT available](https://github.com/huggingface/pytorch-pretrained-BERT)
which is compatible with our pre-trained checkpoints and is able to reproduce
our results. Sosuke Kobayashi also made a
[Chainer version of BERT available](https://github.com/soskek/bert-chainer)
(Thanks!) We were not involved in the creation or maintenance of the PyTorch
implementation so please direct any questions towards the authors of that
repository.
**\*\*\*\*\* New November 3rd, 2018: Multilingual and Chinese models available
\*\*\*\*\***
We have made two new BERT models available:
* **[`BERT-Base, Multilingual`](https://storage.googleapis.com/bert_models/2018_11_03/multilingual_L-12_H-768_A-12.zip)
(Not recommended, use `Multilingual Cased` instead)**: 102 languages,
12-layer, 768-hidden, 12-heads, 110M parameters
* **[`BERT-Base, Chinese`](https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip)**:
Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M
parameters
We use character-based tokenization for Chinese, and WordPiece tokenization for
all other languages. Both models should work out-of-the-box without any code
changes. We did update the implementation of `BasicTokenizer` in
`tokenization.py` to support Chinese character tokenization, so please update if
you forked it. However, we did not change the tokenization API.
For more, see the
[Multilingual README](https://github.com/google-research/bert/blob/master/multilingual.md).
**\*\*\*\*\* End new information \*\*\*\*\***
## Introduction
**BERT**, or **B**idirectional **E**ncoder **R**epresentations from
**T**ransformers, is a new method of pre-training language representations which
obtains state-of-the-art results on a wide array of Natural Language Processing
(NLP) tasks.
Our academic paper which describes BERT in detail and provides full results on a
number of tasks can be found here:
[https://arxiv.org/abs/1810.04805](https://arxiv.org/abs/1810.04805).
To give a few numbers, here are the results on the
[SQuAD v1.1](https://rajpurkar.github.io/SQuAD-explorer/) question answering
task:
SQuAD v1.1 Leaderboard (Oct 8th 2018) | Test EM | Test F1
------------------------------------- | :------: | :------:
1st Place Ensemble - BERT | **87.4** | **93.2**
2nd Place Ensemble - nlnet | 86.0 | 91.7
1st Place Single Model - BERT | **85.1** | **91.8**
2nd Place Single Model - nlnet | 83.5 | 90.1
And several natural language inference tasks:
System | MultiNLI | Question NLI | SWAG
----------------------- | :------: | :----------: | :------:
BERT | **86.7** | **91.1** | **86.3**
OpenAI GPT (Prev. SOTA) | 82.2 | 88.1 | 75.0
Plus many other tasks.
Moreover, these results were all obtained with almost no task-specific neural
network architecture design.
If you already know what BERT is and you just want to get started, you can
[download the pre-trained models](#pre-trained-models) and
[run a state-of-the-art fine-tuning](#fine-tuning-with-bert) in only a few
minutes.
## What is BERT?
BERT is a method of pre-training language representations, meaning that we train
a general-purpose "language understanding" model on a large text corpus (like
Wikipedia), and then use that model for downstream NLP tasks that we care about
(like question answering). BERT outperforms previous methods because it is the
first *unsupervised*, *deeply bidirectional* system for pre-training NLP.
*Unsupervised* means that BERT was trained using only a plain text corpus, which
is important because an enormous amount of plain text data is publicly available
on the web in many languages.
Pre-trained representations can also either be *context-free* or *contextual*,
and contextual representations can further be *unidirectional* or
*bidirectional*. Context-free models such as
[word2vec](https://www.tensorflow.org/tutorials/representation/word2vec) or
[GloVe](https://nlp.stanford.edu/projects/glove/) generate a single "word
embedding" representation for each word in the vocabulary, so `bank` would have
the same representation in `bank deposit` and `river bank`. Contextual models
instead generate a representati
没有合适的资源?快使用搜索试试~ 我知道了~
整理文本分类的比赛和解决方案.zip
共85个文件
py:50个
txt:13个
json:6个
需积分: 5 0 下载量 22 浏览量
2023-10-01
15:38:09
上传
评论
收藏 16.94MB ZIP 举报
温馨提示
整理文本分类的比赛和解决方案
资源推荐
资源详情
资源评论
收起资源包目录
整理文本分类的比赛和解决方案.zip (85个子文件)
TextClassification-master
ElementsRecognition
TraditionalMethods
multi_label_models.py 7KB
__init__.py 0B
main.py 3KB
baseline
svm.py 12KB
multi_label_test.py 35KB
process_data.py 9KB
predictor
__init__.py 28B
predictor.zip 5.63MB
main.py 3KB
predictor.py 3KB
tags
loan
tags.txt 91B
divorce
tags.txt 91B
labor
tags.txt 91B
my_predictor.py 3KB
test_classifier.py 1KB
bert-multilabel
main.py 16KB
stopwords_modify.txt 11KB
off_line_predict_judge.py 21KB
judger.py 6KB
run_predict_judge.sh 160B
run_merge_data_train.sh 206B
run_merge_data_predict_judge.sh 201B
run_train.sh 168B
bert_multilabel_eval.py 17KB
bert-multilabel-classification.py 36KB
radam.py 8KB
data_process.py 5KB
data
loan
data_small_selected.json 1.16MB
tags.txt 91B
divorce
data_small_selected.json 1.93MB
tags.txt 91B
CAIL2019-FE-Small
data_small
loan
data_small_selected.json 1.16MB
tags.txt 90B
divorce
data_small_selected.json 1.93MB
tags.txt 90B
labor
data_small_selected.json 1.04MB
tags.txt 90B
CAIL2019-FE-Small.zip 1007KB
labor
data_small_selected.json 1.04MB
tags.txt 91B
bert_tensorflow_multi_label
utils.py 35KB
multi-label-classification-bert.py 11KB
eval.py 3KB
test_tfserving.py 14KB
configuration.py 3KB
main.py 8KB
smaller_checkpoint_data.py 595B
stopwords_modify.txt 11KB
judger.py 6KB
freeze_graph.py 18KB
freeze_graph_test.py 13KB
ensemble_berts.py 5KB
convert_ckpt_to_pb.py 2KB
bert
modeling_test.py 9KB
__init__.py 616B
run_classifier.zip 8.13MB
extract_features.py 14KB
run_classifier_train.sh 985B
LICENSE 11KB
run_pretraining.py 18KB
sample_text.txt 4KB
CONTRIBUTING.md 1KB
optimization_test.py 2KB
modeling.py 37KB
optimization.py 6KB
tokenization_test.py 4KB
tokenization.py 12KB
requirements.txt 110B
predicting_movie_reviews_with_bert_on_tf_hub.ipynb 65KB
create_pretraining_data.py 16KB
.gitignore 1KB
run_classifier_with_tfhub.py 11KB
README.md 44KB
multilingual.md 11KB
run_classifier.py 38KB
run_squad.py 45KB
offline_main.py 4KB
myeval.py 3KB
judger.py 5KB
readme.md 4KB
data_tools
__init__.py 0B
data_analysis_process.py 18KB
data_process_config.py 3KB
data_read_write.py 3KB
README.md 359B
共 85 条
- 1
资源评论
天天501
- 粉丝: 598
- 资源: 4666
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功