没有合适的资源?快使用搜索试试~ 我知道了~
利用 GPT-4 识别癌症表型..pdf
1.该资源内容由用户上传,如若侵权请联系客服进行举报
2.虚拟产品一经售出概不退款(资源遇到问题,请及时私信上传者)
2.虚拟产品一经售出概不退款(资源遇到问题,请及时私信上传者)
版权申诉
0 下载量 59 浏览量
2024-04-12
15:53:18
上传
评论
收藏 2.91MB PDF 举报
温馨提示
试读
26页
利用 GPT-4 识别癌症表型..pdf
资源推荐
资源详情
资源评论
Leveraging GPT-4 for Identifying Cancer Phenotypes in Electronic Health
Records: A Performance Comparison between GPT-4, GPT-3.5-turbo, Flan-
T5 and spaCy’s Rule-based & Machine Learning-based methods
Kriti Bhattarai, BA
1,2
, Inez Y. Oh, PhD
1
, Jonathan Moran Sierra, BS
3
, Jonathan Tang, MD
4
,
Philip R.O. Payne, PhD
1,2
, Zachary B. Abrams, PhD
1
, Albert M. Lai, PhD
1,2
1
Institute for Informatics, Data Science & Biostatistics, Washington University School of
Medicine, St. Louis, Missouri, USA
2
Department of Computer Science, Washington University in St. Louis, Missouri, USA
3
Medical Scientist Training Program, Washington University School of Medicine, St. Louis,
Missouri, USA
4
Department of Internal Medicine, Washington University School of Medicine, St. Louis,
Missouri, USA
Corresponding Author:
Kriti Bhattarai
Department of Computer Science
Institute for Informatics, Data Science & Biostatistics
Washington University in St. Louis
660 S. Euclid Ave, 6
th
Floor
St. Louis, MO, 63110, USA
kriti.bhattarai@wustl.edu
Keywords: generative pre-trained transformer (GPT), natural language processing, large
language models, clinical phenotype extraction, electronic health records
Word Count:3862
was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprint (whichthis version posted April 6, 2024. ; https://doi.org/10.1101/2023.09.27.559788doi: bioRxiv preprint
ABSTRACT
Objective: Accurately identifying clinical phenotypes from Electronic Health Records (EHRs)
provides additional insights into patients’ health, especially when such information is unavailable
in structured data. This study evaluates the application of OpenAI's Generative Pre-trained
Transformer (GPT)-4 model to identify clinical phenotypes from EHR text in non-small cell lung
cancer (NSCLC) patients. The goal was to identify disease stages, treatments and progression
utilizing GPT-4, and compare its performance against GPT-3.5-turbo, Flan-T5-xl, Flan-T5-xxl,
and two rule-based and machine learning-based methods, namely, scispaCy and medspaCy.
Materials and Methods: Phenotypes such as initial cancer stage, initial treatment, evidence of
cancer recurrence, and affected organs during recurrence were identified from 13,646 records for
63 NSCLC patients from Washington University in St. Louis, Missouri. The performance of the
GPT-4 model is evaluated against GPT-3.5-turbo, Flan-T5-xxl, Flan-T5-xl, medspaCy and
scispaCy by comparing precision, recall, and micro- F1 scores.
Results: GPT-4 achieved higher F1 score, precision, and recall compared to Flan-T5-xl, Flan-
T5-xxl, medspaCy and scispaCy’s models. GPT-3.5-turbo performed similarly to that of GPT-4.
GPT and Flan-T5 models were not constrained by explicit rule requirements for contextual
pattern recognition. SpaCy models relied on predefined patterns, leading to their suboptimal
performance.
Discussion and Conclusion: GPT-4 improves clinical phenotype identification due to its robust
pre-training and remarkable pattern recognition capability on the embedded tokens. It
demonstrates data-driven effectiveness even with limited context in the input. While rule-based
models remain useful for some tasks, GPT models offer improved contextual understanding of
the text, and robust clinical phenotype extraction.
was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprint (whichthis version posted April 6, 2024. ; https://doi.org/10.1101/2023.09.27.559788doi: bioRxiv preprint
BACKGROUND AND SIGNIFICANCE
Introduction
Extracting clinical phenotypes from unstructured Electronic Health Records (EHRs) is a
critical task in natural language processing (NLP). Accurately identifying relevant phenotypes
from unstructured text utilizing NLP techniques provides additional insights into patients’ health,
especially when such information is unavailable in structured data. NLP extraction techniques
facilitate this process by mapping unstructured text to a structured representation, making it
easier to evaluate patients’ disease progression, treatment modalities, and treatment
effectiveness. This is particularly evident when analyzing data from non-small cell lung cancer
patients, where unstructured text is abundant. Accurately identifying disease stage, treatments
and progression from cancer text will contribute to continued research efforts aimed at
improving treatment strategies for non-small lung cancer patients, assessing disease progression,
and improving lung cancer-related outcomes.
Background
Clinical phenotype extraction is an ongoing research area where the type of extraction
tasks and target phenotypes vary across different clinical domains. Rule-based, machine
learning-based, and deep-learning models have been applied to phenotype extraction.
1-7
While
rule-based models extract phenotypes based on pre-defined patterns, most machine learning and
deep-learning approaches are trained on sentences or documents labeled with the relevant
phenotypes and the model subsequently classifies texts into these phenotypes.
5,8
SpaCy models,
including MedspaCy
7
and scispaCy
9
are two recent and frequently used hybrid frameworks that
utilize statistical and machine-learning named entity recognition methods in conjunction with
rule-based NLP to identify clinical phenotypes. There are studies that have utilized medspaCy
was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprint (whichthis version posted April 6, 2024. ; https://doi.org/10.1101/2023.09.27.559788doi: bioRxiv preprint
and scispaCy to identify specific sections within EHR text for NER, extract phenotypes from
relation extraction documents, and generate text embeddings.
10-14
Although extracting clinical phenotypes is essential, several gaps remain in the literature.
There is no effective model for direct extraction, as most of these models require additional
training and fine-tuning.
15,16
Moreover, current methods often lack robustness, leading to
suboptimal performance.
15- 19
In addition, limited availability of labeled, publicly accessible
cancer EHR text leaves an important domain underexplored for NLP.
Pre-trained transformer-based language models have recently been studied for tasks such
as question answering, text generation, and machine translation.
20,21
Despite the success of
transformer-based language model in such tasks, their application in the context of clinical
phenotype extraction remain underexplored, opening numerous avenues of research. Recent
research has demonstrated the use of large language models (LLMs) for entity extraction.
22-25
However, it is essential to investigate these recent transformer-based methods in specific clinical
domains and compare their performance to previously recognized machine learning and rule-
based models to generate insights into their potential benefits for clinical phenotype extraction.
OBJECTIVES
The aim of this study was to investigate the most recent transformer-based language
models as they remain underexplored for cancer phenotype extraction from real-world EHR text.
We evaluated the application of OpenAI’s Generative Pre-Trained Transformer (GPT)-4 model
25
for clinical phenotype extraction in an EHR retrospective study focusing on non-small cell lung
cancer patients as a specific case study. We used GPT-4 to identify individual words or tokens in
a data sequence as distinct phenotypes. Specifically, we measure the prevalence of specific lung
cancer phenotypes, including cancer stage, treatment modalities, cancer recurrence instance, and
was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprint (whichthis version posted April 6, 2024. ; https://doi.org/10.1101/2023.09.27.559788doi: bioRxiv preprint
organs affected by cancer recurrence. These phenotypes are important for informing treatment
decisions and assessing disease progression in non-small cell lung cancer patients.
We built the model framework using a clinical text dataset from Washington University
in St. Louis, Missouri, for a patient population diagnosed with non-small cell lung cancer. To
evaluate the effectiveness of GPT-4, we compared its results against 2 subject matter experts’
manual annotation. We also conducted a comparative analysis with GPT-3.5-turbo
26
, Flan-T5
27
(Flan-T5-xl, Flan-T5-xxl), and spaCy (medspaCy, scispaCy), currently frequently used rule-
based and machine learning approaches in clinical phenotype extraction. While Flan-T5 models
are LLMs, spacy models are two recent and hybrid frameworks that utilize statistical and
machine-learning methods in conjunction with rule-based NLP to identify clinical phenotypes.
We selected these baseline models based on their inherent capacity for rapid extraction, and their
ability to generate results without requiring training or additional fine-tuning.
Our comparison between scispaCy, medspaCy, Flan-T5-xl, Flan-T5-xxl, GPT-3.5-turbo
and GPT-4 aims to highlight the strengths and weaknesses of each approach for cancer
phenotype extraction from unstructured clinical text, providing valuable insights into their
effectiveness and potential use for cases in cancer phenotype extraction from EHR. In evaluating
these current approaches for phenotype extraction, we also note their limitations.
MATERIALS AND METHODS
To extract a detailed representation of specific lung cancer phenotypes, we used GPT-4,
available through Microsoft’s Azure OpenAI Service. We compared and evaluated the
performance of the current models by comparing true positives (recall) and false positives at the
patient-level. The following subsections discuss the datasets, annotation methods, and
methodologies used for extracted information, baseline comparison techniques, and evaluation
was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprint (whichthis version posted April 6, 2024. ; https://doi.org/10.1101/2023.09.27.559788doi: bioRxiv preprint
剩余25页未读,继续阅读
资源评论
百态老人
- 粉丝: 1995
- 资源: 2万+
下载权益
C知道特权
VIP文章
课程特权
开通VIP
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功