# Implicit-Discourse-Relation-Recognition
Identifying IDRR in paragraph sentences using BERT models - 基于 BERT 类模型的隐式语篇关系识别(NLP)
<br>
## Abstract
Implicit discourse relationship identification aims to discover the semantic relationship between two sentences with missing discourse connectives. The discourse relationship between two text fragments plays a key role in many natural language processing tasks. Conjunctives strongly indicate the meaning of discourse relations, while discourses without conjunctives are called implicit discourse. Compared with explicit relations, implicit relations are harder to detect, and the key to their prediction is to correctly model the semantics of two discourse arguments, as well as the contextual interaction between them. This paper proposes several frameworks based on the BERT encoder model, including BERT base, ALBERT, DistilBERT base, RoBERTa and other pre-trained models.
<br>
## Background
A discourse usually contains one or more sentences to describe daily events and for people to exchange ideas and opinions. Since a chapter is usually composed of multiple text fragments, the relationship between text fragments should be considered in order to accurately understand the theme of the chapter. There are often connectives used to express relationships in the text, but there are no connectives between some text fragments, but there is an implicit relationship between them. The task of implicit textual relation recognition is to detect implicit relations and classify their meanings between two text fragments without connectives. The IDRR task has played a crucial role in various downstream natural language processing tasks such as text summarization, machine translation, etc. This paper provides solutions to various BERT pre-trained models for the implicit discourse relationship recognition task.
<br>
## Problem Description
Discourse Relation Recognition (DRR) aims to identify whether there is a logical relationship between two text fragments in a discourse; if it exists, it further classifies the relationship meaning into some predefined types, such as temporal (temporal), causal ( Contingency), comparison (comparison) and extension (expansion) and other relations. An article may contain one or more connective words between text fragments. A connective word is a vocabulary that can directly express the meaning of a certain relationship. For example, the connective word "because" usually expresses a causal relationship between two text fragments. If there are connectives in the discourse relation recognition, it is called Explicit Discourse Relation Recognition (EDRR), otherwise it is called Implicit Discourse Relation Recognition (IDRR). This experiment uses the Penn Discourse Tree Bank 2.0 (PDTB-2.0) corpus as the input sample of the model.
<br>
## Solution
Neural network experiments successively adopted different types of BERT models to train the PDTB-2.0 dataset and classify the implicit discourse relations of the chapters. Firstly, BERT BASE, ALBERT BASE, DistilBERT BASE and other models were used for training in the experiment, but the effect was not good, and the F1 values were 56.31, 56.81, and 50.19 respectively. After improvement, RoBERTa BASE, a model based on tag-dependent perceptual sequence generation, was used to achieve an F1 value of 62.08; and then tried to use the RoBERTa LARGE model, and through parameter adjustment, an F1 value of 69.76 was achieved. Experiments have verified that BERT can improve the accuracy and F1 value of many natural language processing and language modeling tasks, and it plays an important role in IDRR tasks.
<br>
## Experimental setup
Environment: Visual Studio Code 1.74.1 + Python 3.7.7 + NVDIA GeForce RTX 3090
Data: Penn Discourse Tree Bank 2.0(PDTB-2.0)
<br>
## Algorithm flowchart
![flowchart](README/flowchart.png)
<br>
## Result
| Model | Results |
| :---------------: | :------------------------------------------------------------: |
| BERT BASE | ![bertbase-output](README/bertbase-output.png) |
| ALBERT BASE | ![albertbase-output](README/albertbase-output.png) |
| DistilBERT BASE | ![distilbert-output](README/distilbert-output.png) |
| RoBERTa BASE | ![robertabase-output](README/robertabase-output.png) |
| RoBERTa LARGE | ![roberta-large](README/roberta-large.png) |
<br>
As shown in table above, it can be observed that the effect of using DistilBERT BASE at the beginning is not good, only reaching the F1 value of 50.19. This is due to the streamlined architecture of the DistilBERT model, which performs knowledge distillation in the pre-training phase, resulting in a 40% reduction in model size compared to the BERT model, a reduction in the number of layers, and much fewer trainable parameters. Among them, BERT's word segmentation embedding and pooling layers were deleted at the same time, and finally the number of DistilBERT BASE layers was reduced by 2 times. After trying to train the model on BERT BASE and ALBERT BASE, the experimental effect has been slightly improved, that is, the F1 values of 56.31 and 56.81 were respectively achieved, and the training time is equivalent to twice that of the DistilBERT BASE model.
In the experiment, the RoBERTa model was used again for IDRR task training. The BASE version achieved a higher F1 value of 62.08, while the performance improvement of the LARGE version was more significant, reaching an F1 value of 69.76. RoBERTa works much better than other BERT-like models because the model uses dynamic masking, where different parts of the sentence are masked at each iteration, making the model more performant; other BERT-like models use static masking. The batch size used by RoBERTa is much larger than that of BERT, and the training data set is also much larger than that of BERT, which is about 160 GB; while BERT only trains about 16 GB of data, so RoBERTa is more robust on IDRR tasks than Other BERT-like models are high.
<br>
## Conclusion
This project emphasizes the usage of various BERT-like models to train the Implicit Discourse Relation Discrimination (IDRR) task, and a deep understanding on variables such as model framework, number of parameters, model size, hyperparameters, etc. influence on the training effect. Compared with DistilBERT BASE, ALBERT BASE, BERT BASE, and RoBERTa BASE, RoBERTa LARGE model training has reached the highest F1 value, improving the overall model performance and accuracy.
没有合适的资源?快使用搜索试试~ 我知道了~
温馨提示
项目工程资源经过严格测试可直接运行成功且功能正常的情况才上传,可轻松复刻,拿到资料包后可轻松复现出一样的项目,本人系统开发经验充足(全领域),有任何使用问题欢迎随时与我联系,我会及时为您解惑,提供帮助。 【资源内容】:包含完整源码+工程文件+说明(如有)等。答辩评审平均分达到96分,放心下载使用!可轻松复现,设计报告也可借鉴此项目,该资源内项目代码都经过测试运行成功,功能ok的情况下才上传的。 【提供帮助】:有任何使用问题欢迎随时与我联系,我会及时解答解惑,提供帮助 【附带帮助】:若还需要相关开发工具、学习资料等,我会提供帮助,提供资料,鼓励学习进步 【项目价值】:可用在相关项目设计中,皆可应用在项目、毕业设计、课程设计、期末/期中/大作业、工程实训、大创等学科竞赛比赛、初期项目立项、学习/练手等方面,可借鉴此优质项目实现复刻,设计报告也可借鉴此项目,也可基于此项目来扩展开发出更多功能 下载后请首先打开README文件(如有),项目工程可直接复现复刻,如果基础还行,也可在此程序基础上进行修改,以实现其它功能。供开源学习/技术交流/学习参考,勿用于商业用途。质量优质,放心下载使用。
资源推荐
资源详情
资源评论
收起资源包目录
Identifying IDRR in paragraph sentences using BERT models - 基于 BERT 类模型的隐式语篇关系识别(NLP).zip (50个子文件)
DSqrs942240909
utils.py 6KB
roberta-large.py 8KB
preprocess.py 10KB
README
roberta-large.png 62KB
bertbase-output.png 58KB
flowchart.png 186KB
robertabase-output.png 60KB
distilbert-output.png 67KB
albertbase-output.png 63KB
pdtb2.py 16KB
roberta-base.py 8KB
distilbert-base-uncased.py 8KB
gcn.py 1012B
PDTB
Ji
data
dev.txt 345KB
conn.txt 1001B
class11.txt 233B
class4.txt 41B
test.txt 294KB
train.txt 3.64MB
saved_dict
saved_dict.txt 0B
log
distilbert-base-uncased.log 4KB
roberta-large.log 4KB
bert-base-uncased.log 4KB
roberta.log 2KB
log.txt 0B
albert-base-v2.log 2KB
roberta-base.log 2KB
.gitattributes 66B
label_graph.g 107KB
bert-base-uncased.py 9KB
output
roberta-base.txt 51KB
distilbert-output.txt 49KB
albertbase-output.txt 51KB
roberta-large.txt 50KB
bertbase-output.txt 51KB
dataset
implicit_train.csv 2.8MB
implicit_test.json 328KB
labels.json 261B
implicit_dev.json 384KB
implicit_train.json 4.13MB
隐式数据中的conn是人工标注的伪连接词,测试时也是不可作为输入的(内空).txt 0B
explicit.json 4.86MB
pdtb2.csv 56.29MB
albert-base-v2.py 8KB
run.py 7KB
preprocess.ipynb 35KB
train_mutual_learning.py 13KB
transformer.py 10KB
__pycache__
roberta-large.cpython-37.pyc 5KB
README.md 6KB
共 50 条
- 1
资源评论
热爱技术。
- 粉丝: 2814
- 资源: 7860
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
最新资源
- 西电微机原理实验四:微机原理实验中8255可编程并行接口的应用实践
- 西电微机原理课程实验指南-理论与实践相结合提升技术素养
- 基于go+gin+vue+element admin 后台管理系统,支持用户管理,认证,内容管理等详细文档+优秀项目+全部资料.zip
- 基于go-kratos +Ant Design Pro的前后端分离微服务管理系统后端模块详细文档+优秀项目+全部资料.zip
- 基于Golang 的后台管理系统(基础版)详细文档+优秀项目+全部资料.zip
- CR750CR751 控制器操作说明书(故障排除).pdf
- 基于goframe搭建的电商前后台API系统详细文档+优秀项目+全部资料.zip
- linux常用命令大全.txt
- 基于golang的分布式即时通讯系统详细文档+优秀项目+全部资料.zip
- linux常用命令大全.txt
- 基于Golang的个人简易博客系统详细文档+优秀项目+全部资料.zip
- 基于Golang实现的单点登录系统(go-sso),实现手机号注册、手机号+验证码登录、手机号+密码登录、账号登出等功能,用户认证采用cookie和jwt两种方式详细文档+优秀项目+全部资料.zip
- 基于Golang+Markdown的博客系统详细文档+优秀项目+全部资料.zip
- 基于golang实现的分布式聊天系统,支持i一对一聊天,聊天室等详细文档+优秀项目+全部资料.zip
- 基于Golang的开源社区系统。简洁对话,高效互动,详细文档+优秀项目+全部资料.zip
- 基于Golang重构考试系统详细文档+优秀项目+全部资料.zip
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功