# JointBERT
(Unofficial) Pytorch implementation of `JointBERT`: [BERT for Joint Intent Classification and Slot Filling](https://arxiv.org/abs/1902.10909)
## Model Architecture
<p float="left" align="center">
<img width="600" src="https://user-images.githubusercontent.com/28896432/68875755-b2f92900-0746-11ea-8819-401d60e4185f.png" />
</p>
- Predict `intent` and `slot` at the same time from **one BERT model** (=Joint model)
- total_loss = intent_loss + coef \* slot_loss (Change coef with `--slot_loss_coef` option)
- **If you want to use CRF layer, give `--use_crf` option**
## Dependencies
- python>=3.6
- torch==1.6.0
- transformers==3.0.2
- seqeval==0.0.12
- pytorch-crf==0.7.2
## Dataset
| | Train | Dev | Test | Intent Labels | Slot Labels |
| ----- | ------ | --- | ---- | ------------- | ----------- |
| ATIS | 4,478 | 500 | 893 | 21 | 120 |
| Snips | 13,084 | 700 | 700 | 7 | 72 |
- The number of labels are based on the _train_ dataset.
- Add `UNK` for labels (For intent and slot labels which are only shown in _dev_ and _test_ dataset)
- Add `PAD` for slot label
## Training & Evaluation
```bash
$ python3 main.py --task {task_name} \
--model_type {model_type} \
--model_dir {model_dir_name} \
--do_train --do_eval \
--use_crf
# For ATIS
$ python3 main.py --task atis \
--model_type bert \
--model_dir atis_model \
--do_train --do_eval
# For Snips
$ python3 main.py --task snips \
--model_type bert \
--model_dir snips_model \
--do_train --do_eval
```
## Prediction
```bash
$ python3 predict.py --input_file {INPUT_FILE_PATH} --output_file {OUTPUT_FILE_PATH} --model_dir {SAVED_CKPT_PATH}
```
## Results
- Run 5 ~ 10 epochs (Record the best result)
- Only test with `uncased` model
- ALBERT xxlarge sometimes can't converge well for slot prediction.
| | | Intent acc (%) | Slot F1 (%) | Sentence acc (%) |
| --------- | ---------------- | -------------- | ----------- | ---------------- |
| **Snips** | BERT | **99.14** | 96.90 | 93.00 |
| | BERT + CRF | 98.57 | **97.24** | **93.57** |
| | DistilBERT | 98.00 | 96.10 | 91.00 |
| | DistilBERT + CRF | 98.57 | 96.46 | 91.85 |
| | ALBERT | 98.43 | 97.16 | 93.29 |
| | ALBERT + CRF | 99.00 | 96.55 | 92.57 |
| **ATIS** | BERT | 97.87 | 95.59 | 88.24 |
| | BERT + CRF | **97.98** | 95.93 | 88.58 |
| | DistilBERT | 97.76 | 95.50 | 87.68 |
| | DistilBERT + CRF | 97.65 | 95.89 | 88.24 |
| | ALBERT | 97.64 | 95.78 | 88.13 |
| | ALBERT + CRF | 97.42 | **96.32** | **88.69** |
## Updates
- 2019/12/03: Add DistilBert and RoBERTa result
- 2019/12/14: Add Albert (large v1) result
- 2019/12/22: Available to predict sentences
- 2019/12/26: Add Albert (xxlarge v1) result
- 2019/12/29: Add CRF option
- 2019/12/30: Available to check `sentence-level semantic frame accuracy`
- 2020/01/23: Only show the result related with uncased model
- 2020/04/03: Update with new prediction code
## References
- [Huggingface Transformers](https://github.com/huggingface/transformers)
- [pytorch-crf](https://github.com/kmkurn/pytorch-crf)
没有合适的资源?快使用搜索试试~ 我知道了~
JointBert-demo-chinese.rar
共89个文件
png:31个
py:11个
txt:10个
需积分: 32 4 下载量 66 浏览量
2021-12-03
10:07:17
上传
评论
收藏 372.43MB RAR 举报
温馨提示
BERT for Joint Intent Classifification and Slot Filling的中文实现
资源详情
资源评论
资源推荐
收起资源包目录
JointBert-demo-chinese.rar (89个子文件)
JointBert-demo-chinese
text
Transformer.ppt 10.48MB
seq2seq
img_11.png 45KB
img_14.png 315KB
img_23.png 18KB
img_9.png 131KB
seq2seq.md 20KB
img_16.png 157KB
img_7.png 129KB
img_15.png 406KB
img_26.png 122KB
img_28.png 121KB
img_12.png 39KB
img_29.png 102KB
img_27.png 99KB
img_2.png 77KB
img_10.png 37KB
img_1.png 36KB
img.png 69KB
img_17.png 86KB
img_24.png 22KB
img_4.png 81KB
img_25.png 30KB
img_20.png 196KB
img_8.png 32KB
img_18.png 90KB
elmo.png 176KB
img_22.png 180KB
img_13.png 55KB
img_6.png 74KB
img_21.png 259KB
img_19.png 116KB
img_5.png 83KB
img_3.png 62KB
模型代码解读.md 223B
main.py 5KB
utils.py 4KB
data
sample
train
seq.in 29KB
label 7KB
seq.out 68KB
intent_label.txt 45B
test
seq.in 6KB
label 1KB
seq.out 15KB
slot_label.txt 1KB
dev
seq.in 3KB
label 632B
seq.out 7KB
vocab_process.py 2KB
atis
train
seq.in 283KB
label 56KB
seq.out 406KB
intent_label.txt 380B
test
seq.in 51KB
label 11KB
seq.out 79KB
slot_label.txt 2KB
dev
seq.in 32KB
label 6KB
seq.out 47KB
snips
train
seq.in 603KB
label 180KB
seq.out 929KB
intent_label.txt 103B
test
seq.in 32KB
label 10KB
seq.out 50KB
slot_label.txt 1002B
dev
seq.in 33KB
label 10KB
seq.out 49KB
model
module.py 872B
modeling_jointalbert.py 3KB
__init__.py 142B
modeling_jointdistilbert.py 3KB
modeling_jointbert.py 4KB
data_loader.py 14KB
sample_pred_out.txt 754B
predict.py 9KB
predict.sh 221B
bert-base-chinese
vocab.txt 107KB
pytorch_model.bin 392.55MB
config.json 647B
LICENSE 11KB
trainer.py 11KB
requirements.txt 67B
.gitignore 1KB
README.md 4KB
sample.sh 166B
sample_pred_in.txt 152B
共 89 条
- 1
weixin_53420606
- 粉丝: 0
- 资源: 3
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
最新资源
- 论文(最终)_20240430235101.pdf
- 基于python编写的Keras深度学习框架开发,利用卷积神经网络CNN,快速识别图片并进行分类
- 最全空间计量实证方法(空间杜宾模型和检验以及结果解释文档).txt
- 5uonly.apk
- 蓝桥杯Python组的历年真题
- 2023-04-06-项目笔记 - 第一百十九阶段 - 4.4.2.117全局变量的作用域-117 -2024.04.30
- 2023-04-06-项目笔记 - 第一百十九阶段 - 4.4.2.117全局变量的作用域-117 -2024.04.30
- 前端开发技术实验报告:内含4四实验&实验报告
- Highlight Plus v20.0.1
- 林周瑜-论文.docx
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功
评论0