使用所学知识库和多模式对齐生成放射报告_Radiology Report Generation with a Learned K
版权申诉
117 浏览量
2022-01-14
23:27:18
上传
评论
收藏 906KB PDF 举报
Radiology Report Generation with a Learned Knowledge Base
and Multi-modal Alignment
Shuxin Yang
1,5
, Xian Wu
3
, Shen Ge
3
, Xingwang Wu
4
, S. Kevin Zhou
1,2
, Li Xiao
1,5
1
The Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology,
CAS, Beijing, 100190, China
2
School of Biomedical Engineering & Suzhou Institute for Advanced Research Center for Medical Imaging, Robotics, and
Analytic Computing & LEarning (MIRACLE) University of Science and Technology of China, Suzhou 215123, China
3
Tencent Medical AI Lab, Beijing, 100094, China
4
The First Affiliated Hospital of Anhui Medical University, HeFei, 230022, China
5
University of Chinese Academy of Sciences, Beijing, 100049, China
Abstract
In clinics, a radiology report is crucial for guiding a patient’s
treatment. Unfortunately, report writing imposes a heavy bur-
den on radiologists. To effectively reduce such a burden, we
hereby present an automatic, multi-modal approach for re-
port generation from chest x-ray. Our approach, motivated by
the observation that the descriptions in radiology reports are
highly correlated with the x-ray images, features two distinct
modules: (i) Learned knowledge base. To absorb the knowl-
edge embedded in the above-mentioned correlation, we auto-
matically build a knowledge base based on textual embedding.
(ii) Multi-modal alignment. To promote the semantic align-
ment among reports, disease labels and images, we explicitly
utilize textual embedding to guide the learning of the visual
feature space. We evaluate the performance of the proposed
model using metrics from both natural language generation
and clinic efficacy on the public IU and MIMIC-CXR datasets.
Our ablation study shows that each module contributes to im-
proving the quality of generated reports. Furthermore, with
the aid of both modules, our approach clearly outperforms
state-of-the-art methods.
Introduction
The radiology report is crucial for assisting clinic decision
making (Zhou, Rueckert, and Fichtinger 2019). It describes
some observations on images such as diseases’ degree, size,
and location. However, the process of writing reports is time-
consuming and tedious for radiologists (Bruno, Walker, and
Abujudeh 2015). With the advancement of deep learning
and natural language processing, automatic radiology report
generation has attracted growing research interests.
Many radiology report generation approaches follow the
practice of image captioning models (Xu et al. 2015; Lu et al.
2017; Anderson et al. 2018). For example, (Jing, Xie, and
Xing 2018; Yuan et al. 2019) employ the encoder-decoder
architecture and propose the hierarchical generator as well
as the attention mechanism to generate long reports. How-
ever, radiology report generation task is different from image
captioning task. In image captioning, the model is required
to cover the details of the input image, while for radiology
report generation, the model is required to focus on the ab-
normal regions and infer potential diseases. Therefore, to
Copyright © 2022, All rights reserved.
generate a correct radiology report, the model needs to iden-
tify the abnormal regions and provide proper descriptions.
To this end, the medical background knowledge needs to be
included in modeling.
Recently, some works attempt to integrate medical knowl-
edge in modeling: MKG (Zhang et al. 2020) and PPKED (Liu
et al. 2021) incorporate manual pre-constructed knowledge
graphs to enhance the generation, HRGR (Li et al. 2018)
builds a template database based on prior knowledge by man-
ually filtering a set of sentences in the training corpus. These
methods achieve improved performance over image caption-
ing models. However, these models need to build the knowl-
edge graph or template database in advance which is still
laborious. In addition, when applying these models to images
of other diseases, the knowledge graph or template database
needs to be updated as well.
In this paper, we propose a knowledge base updating mech-
anism to store medical knowledge automatically. It learns
a knowledge base from training data. Firstly, we initialize
a memory as a knowledge base and use CNN/BERT model
to extract visual features and textual embeddings from the
input images and corresponding reference reports. Next, the
knowledge base is updated by the report embeddings during
the training phase. At the end of training, we fix the knowl-
edge base as the model’s parameter and use it for report
generation. To acquire the related knowledge of the input
image, we propose a visual-knowledge attention module that
queries knowledge base with visual features. Finally, we em-
ploy the standard Transformer model with the help of the
visual features and acquired knowledge to generate radiology
reports.
Since the critical clinical information usually comes from
descriptions of abnormalities, where such sentences are rare
and diverse in radiology datasets, we need to enable the
knowledge base to focus on the knowledge of abnormalities.
To this end, we propose a multi-modal alignment mecha-
nism. It consists of visual-textual alignment and visual-label
alignment. The intuition is that the reports and disease la-
bels describe the same observations on the images, so the
semantic features among images, reports, and disease labels
should be consistent. Specifically, we adapt the triplet mar-
gin loss (Balntas et al. 2016) to align the visual features and
arXiv:2112.15011v1 [eess.IV] 30 Dec 2021
资源评论
易小侠
- 粉丝: 6509
- 资源: 9万+
最新资源
- 科大讯飞语音引擎 jar包 demo,科大讯飞语音合成引擎3.0，支持4.0系统以上，文字转语音输出.zip
- Java架构面试笔试专题资料及经验（含答案）SpringBoot面试Linux面试专题及答案 合集.zip
- 头歌c语言实验答案tion-model-for-ne开发笔记
- docker配置使用-model-for-networK开发demo
- docker配置使用vaWeb-mas笔记
- c语言连接两个字符串-mas开发笔记
- 俄罗斯引擎yandex进入x-master 笔记
- 头歌c语言实验答案el-for-network-ids-ma笔记
- 一个delphi写的连连看
- 植物大战僵尸-杂交版 最新版本
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
