使用监督机器学习的可靠性..doc资源-CSDN文库

版权申诉

113 浏览量 2024-04-03 10:27:47 上传评论收藏 1.44MB DOC 举报

### 使用监督机器学习的可靠性研究：基于合成数据在医疗保健领域的应用 #### 摘要与背景本文探讨了在医疗保健领域利用合成数据进行监督机器学习模型训练的可靠性和隐私保护问题。随着数据科学的发展，如何有效利用敏感度高的医疗数据成为了一个重要的课题。为了在不泄露患者隐私的情况下最大化数据价值，研究者们开始探索合成数据的生成及应用。目前，虽然已有一些用于生成合成数据的方法被提出，但针对这些方法的有效性和通用性的评估研究仍然较少。 #### 研究目标本研究旨在通过对比分析监督机器学习模型在真实数据与合成数据上的表现差异，来理解合成数据在医疗保健领域中的潜力和局限性。具体而言，本研究将采用多种流行的合成数据生成技术，并将其与实际数据集相结合，以此评估这些技术在保持隐私的同时能否有效地支持机器学习任务。 #### 实验设计 1. **数据集选择**：实验选用了19个公开的医疗保健数据集，这些数据集包含了分类和数值型数据。 2. **合成数据生成**：使用了三种不同的合成数据生成器，分别是基于分类和回归树（CART）、参数化方法以及贝叶斯网络的技术。这三种方法分别代表了不同类型的合成数据生成策略，有助于全面评估合成数据的质量。 3. **监督机器学习模型**：选择了五种经典的监督学习算法来进行模型训练，包括随机梯度下降（SGD）、决策树、K近邻算法（KNN）、随机森林和支持向量机（SVM）。这些算法广泛应用于分类和回归任务中，具有较好的代表性。 4. **性能评估**：所有模型均只在真实数据上进行测试，以此来衡量在合成数据上训练出的模型是否能够达到与真实数据相同或接近的性能水平。 #### 合成数据生成方法简介 1. **基于分类和回归树（CART）**：这种方法通过构建一棵或多棵决策树来模拟数据的分布特征。它能够较好地处理分类和数值型数据，并且可以捕捉到数据之间的复杂关系。 2. **参数化方法**：此类方法假设数据服从某种概率分布（如正态分布），并基于此分布生成新的样本。这种方法简单直观，但在处理非线性关系时可能存在局限性。 3. **贝叶斯网络**：贝叶斯网络是一种基于概率图模型的数据生成方法，它能够捕捉变量间的条件依赖关系，适用于处理具有复杂结构的数据集。 #### 监督机器学习模型概述 1. **随机梯度下降（SGD）**：这是一种用于优化线性和非线性模型的迭代方法，特别适用于大规模数据集。 2. **决策树**：该算法通过构建一个树形结构来表示特征间的逻辑关系，易于理解和解释。 3. **K近邻算法（KNN）**：KNN是一种基于实例的学习方法，通过计算新样本与已有样本之间的距离来预测其类别。 4. **随机森林**：随机森林由多个决策树组成，通过集成学习的方式提高模型的准确性和稳定性。 5. **支持向量机（SVM）**：SVM是一种二分类模型，通过寻找最优超平面来实现对数据的最大间隔分类。 #### 结论与展望通过比较真实数据和合成数据上训练出的模型性能，可以为未来的研究提供有价值的见解。此外，研究还可能揭示哪些特定类型的数据或任务更适合使用合成数据，以及如何进一步改进合成数据生成技术以更好地服务于医疗保健领域的需求。这项工作不仅有助于推动合成数据在保护隐私方面的应用，也为利用大数据改善医疗服务提供了新的可能性。

资源推荐

资源详情

资源评论

Original Paper

Debbie Rankin

PhD, Corresponding author, d.rankin1@ulster.ac.uk, +4428 7167 5841

Michaela Black

PhD, mm.black@ulster.ac.uk

Raymond Bond

PhD, rb.bond@ulster.ac.uk

Jonathan Wallace

MSc, jg.wallace@ulster.ac.uk

Maurice Mulvenna

PhD, md.mulvenna@ulster.ac.uk

Gorka Epelde

3,4

PhD, gepelde@vicomtech.org

School of Computing, Engineering and Intelligent Systems, Ulster University,

Derry~Londonderry, Northern Ireland, United Kingdom

School of Computing, Ulster University, Jordanstown, Northern Ireland, United

Kingdom

Vicomtech Foundation, Basque Research and Technology Alliance (BRTA), Donostia-

San Sebastián, Spain

Biodonostia Health Research Institute, eHealth Group, Donostia-San Sebastián, Spain

Reliability of Supervised Machine Learning Using Synthetic Data

in Healthcare: A Model to Preserve Privacy for Data Sharing

Abstract

Background:

The exploitation of synthetic data in healthcare is at an early stage. Synthetic data

generation could unlock the vast potential within healthcare datasets that are too

sensitive for release due to privacy concerns. Several synthetic data generators have

been developed to date, however studies evaluating their efficacy and

generalisability are scarce.

Objective:

This work sets out to understand the difference in performance of supervised

machine learning models trained on synthetic data compared with those trained on

real data.

Methods:

A total of 19 open healthcare datasets containing both categorical and numerical

data have been selected for experimental work. Synthetic data is generated using

three popular synthetic data generators that apply Classification and Regression

Trees, parametric and Bayesian network approaches. Real and synthetic data are

used (separately) to train five supervised machine learning models: stochastic

gradient descent, decision tree, k-nearest neighbors, random forest and support

vector machine. Models are tested only on real data to determine whether a model

developed by training on synthetic data can be put into use by healthcare

departments and used to accurately classify new, real examples. Evaluation metrics

are computed and differentials in these scores are compared. The impact of

statistical disclosure control on model performance is also assessed.

Results:

The accuracy of ML models trained on synthetic data is lower than models trained

on real data in 92% of cases. Tree-based models trained on synthetic data have

deviations in accuracy from models trained on real data of 17.7-19.3%, whilst other

models have lower deviations of 5.8-7.2%. The winning classifier when trained and

tested on real data versus models trained on synthetic data and tested on real data

is the same in 26.3% of cases for CART and parametric synthetic data, and in 21.1%

of cases for Bayesian network generated synthetic data. Tree-based models perform

best with real data and are the winning classifier in 94.7% of cases. This is not the

case for models trained on synthetic data. When tree-based models are not

considered, the winning classifier for real and synthetic data is matched in 73.7%,

52.6% and 68.4% of cases for CART, parametric and Bayesian network synthetic

data, respectively. Statistical disclosure control methods did not have a notable

impact on data utility.

Conclusions:

The results of this study are promising with small decreases in accuracy observed in

models trained with synthetic data compared to models trained with real data,

where both are tested on real data. Such deviations are expected and manageable.

Tree-based classifiers have some sensitivity to synthetic data and the underlying

cause requires further investigation. This study highlights the potential of synthetic

data and the need for further evaluation its robustness. Synthetic data must ensure

individual privacy and data utility is preserved in order to instil confidence in

healthcare departments when utilising such data to inform policy decision-making.

Keywords: Synthetic Data; Supervised Machine Learning; Data Utility; Healthcare;

Decision Support; Statistical Disclosure Control

Introduction

Background

National Healthcare Departments hold vast volumes of data on patients and the

population that is not being used to its full potential due to valid privacy concerns.

Machine learning (ML) has the potential to vastly improve decisions and outcomes

in healthcare and yet these improvements have not yet been fully realised. The

reason may be in part related to an issue that faces many data scientists and

researchers in the area: the limited availability of or access to data, or the readiness

for healthcare institutions to share data. Privacy concerns over personal data, and in

particular healthcare data, means that although the data exists, it is deemed too

sensitive for public release [1], even in the case of serious research.

One way to overcome the issue of data availability is to use fully synthetic data as an

alternative to real data. The exploitation of synthetic data in healthcare is at an early

stage and is gaining increasing attention. Synthetic data is data that is simulated

from real data by using the underlying statistical properties of the real data to

produce synthetic datasets that exhibit these same statistical properties. Synthetic

data can represent the population in the original data whilst avoiding any

divulgence of real, potentially personal, confidential and sensitive data. In the case

of health-related data, this would ensure that actual patient records are not

disclosed thus avoiding governance and confidentiality issues. There are three types

of synthetic data: fully synthetic, partially synthetic, and hybrid synthetic. This work

considers fully synthetic data which does not contain original data.

Synthetic data can be used in two ways: to augment an existing dataset thus

increasing its size, for times when a dataset is unbalanced due to the limited

occurrence of an event or when more examples are required [2,3]; and to generate a

fully synthetic dataset that is representative of the original dataset, for times when

data is not available due to its sensitive nature [4]. The latter is considered in this

work as a key requirement for healthcare data sharing.

Traditionally, data perturbation techniques such as data swapping, data masking,

cell suppression and adding noise, have been applied to real data to modify and thus

protect the data from disclosure prior to releasing it. However, such methods do not

eliminate disclosure risk and can impact the utility of the data, particularly if

multivariate relationships are not considered [5]. Synthetic data was first proposed

by Rubin [6] and Little [7]. Raghunathan, Reiter and Rubin [8] implemented and

extended upon this, pioneering the multiple imputation approach to synthetic data

generation, exemplified in a range of studies [9-14]. Reiter [15] then introduced an

alternative method of synthesising data through a non-parametric tree-based

technique that utilises Classification and Regression Trees (CART). A more recent

technique proposes a Bayesian network approach for synthetic data generation

[16]. Synthetic data is considered a secure approach for enabling public release of

sensitive data as it goes beyond traditional de-identification methods by generating

a fake dataset that does not contain any of the original, identifiable information from

which it was generated, whilst retaining the valid statistical properties of the real

data. Therefore, the risk of disclosure of a real person or reverse engineering is

considered to be unlikely [17].

Whilst a number of synthetic data generators have been developed, empirical

evidence of their efficacy has not been fully explored. This work extends a

preliminary study [18] and investigates whether fully synthetic data can preserve

the hidden complex patterns that supervised ML can uncover from real data, and

therefore whether it can be used as a valid alternative to real data when developing

eHealth applications and healthcare policy making solutions. This will be achieved

by experimenting with a range of open healthcare datasets. Synthetic data will be

generated using three well known synthetic data generation techniques. Supervised

ML algorithms will be used to validate the performance of the synthetic datasets.

Statistical disclosure control (SDC) methods that can further decrease the disclosure

risk associated with synthetic data will also be considered.

Overview

To inform the viability of the use of synthetic data as a valid and reliable alternative

to real data in the healthcare domain we will answer the following research

questions:

1) What is the differential in performance when using synthetic data versus real

data for training and testing supervised ML models?

2) What is the variance of absolute difference of accuracies between ML models

training on real and synthetic datasets?

3) How often does the winning ML technique change when training using real

data to training using synthetic data?

4) What is the impact of statistical disclosure control (i.e. privacy protection)

measures on the utility of synthetic data (i.e. similarity to real data)?

To answer these questions, 19 open healthcare datasets containing both categorical

and numerical data have been selected for experimentation [19]. Synthetic datasets

are generated for each of these 19 datasets using three popular synthetic data

generators that apply CART [15,17], parametric [8,17] and Bayesian network [16]

approaches, respectively, to enable a robust comparison of the three synthetic data

generation techniques across a broad range of data.

Initially we analyse whether the multivariate relationships that exist in the real data

are preserved in the synthetic versions of the data, for data generated using each of

the three synthetic data generation techniques, by computing pairwise mutual

information scores for each variable pair combination in each dataset [16]. It is

important that such relationships are retained when data is synthesised.

To evaluate the utility of synthetic data for Machine Learning, we then investigate

the performance of supervised ML models trained on synthetic data and tested on

real data, compared with models trained on real data and also tested on the real

data. This allows us to determine if a model developed using synthetic data can

classify real data examples as accurately and reliably as a model developed using

real data. We consider five different supervised machine learning models to

compare performance and determine if there are differences in robustness across

each of these models. Standard evaluation metrics are computed for models trained

on real and synthetic data, for each ML model, and for each dataset [20]. The

differences in accuracy for models trained on synthetic data versus models trained

on real data are computed to analyse the extent to which synthetic data causes a

degradation in model performance, if any.

It is pertinent that the optimal ML model built using synthetic data matches the

optimal ML model that would be selected if real data were used in the model

training process. This would provide stakeholders in healthcare with confidence in

the use of synthetic data for model development. Thus, we consider how often the

best ML classifier built using synthetic data matches the best ML model built using

real data.

Finally, the impact of a number of statistical disclosure control methods on model

performance is assessed. Statistical disclosure control methods seek to further

enhance data privacy; however, this can lead to a loss in usefulness of the data [21]

and we consider the extent to which performance degradation occurs as a result of

SDC.

This large-scale assessment of the reliability of synthetic data when used for

supervised ML, utilising 19 healthcare datasets and 3 synthetic data generation

techniques, provides an important contribution in relation to the trust and

confidence that stakeholders in healthcare can have in synthetic data. We also

propose a pipeline to illustrate how synthetic data can potentially fit within the

healthcare provider context. This work demonstrates the promising performance of

synthetic data whilst highlighting its limitations and future work directions to

overcome them.

Synthetic Data: Present and Future Use

The validity and disclosure risk associated with synthetic data has been under

investigation by the U.S. Census Bureau since 2003 for the purpose of creating

public use data from a combination of sensitive data from the Census Bureau’s

Survey of Income and Program Participation (SIPP), the Internal Revenue Service’s

(IRS) individual lifetime earnings data, and the Social Security Administration’s

(SSA) individual benefit data [22,23]. The goal was to enable the release of

synthesised person-level records containing personal and financial characteristics

from confidential datasets, whilst preserving privacy. Successful results have led to

the release of public use synthetic data files. Researchers can have their work

validated against the Gold Standard (real) data by the Census Bureau, thus enabling

them to determine the impact of synthetic data on their exploratory analyses and

model development and have confidence in their results, whilst also allowing the

Census Bureau to continuously improve their synthesis techniques. The public

release of this data has provided significant benefit to the research community and

general population, enabling more extensive economic policy research to be

performed by groups who could not previously access useful data [24-29]. This

work led to the release of further synthetic datasets by the Census Bureau. The

Synthetic Longitudinal Business Database (SynLBD) comprises data from an annual

economic census of establishments in the U.S. [30]. This dataset provides broad

access to rich data that supports the research and policy-making communities in

business and employment related topics. OnTheMap is a tool utilising synthetic data

to provide workforce related maps, demographic profiles and reports of U.S.

citizens, as well as disaster event information and the impact of such events on

workers and employers [31]. Similarly, synthetic data has also been under

investigation in the UK as a means to provide public access to rich data from UK

Longitudinal Studies [32-34] that contain highly sensitive data linking national

census data to administrative data for individuals and their families.

These datasets enable researchers to explore data and develop and test code and

models outside the secure environment where real data resides with no restrictions,

剩余27页未读，继续阅读

评论收藏

内容反馈

版权申诉

百态老人

粉丝: 6831
资源: 2万+

使用监督机器学习的可靠性..doc

可信机器学习的公平性综述.pdf

基于机器学习的SMN可靠性分析及量测误差建模.pdf

2021年人工智能复习新版题库.doc

(完整word版)人工智能复习题库.doc

资产定价动态因子模型的深度学习.doc

计算机网络技术简答.doc

重点设备异常状况.DOC

大数据考试题含答案.doc

会计学综合练习.doc

数据挖掘试卷一.doc

装卸工安全作业交底内容.doc

人工智能的发展及未来畅想.doc

计算机网络管理技术及发展展望.doc

大数据考试题含答案(word文档良心出品).doc

公司安全检查表.doc

【大作业】基于模式识别的智能检测-共9页.doc

放射科技术组岗位职责说明.doc

论输电线路巡检管理系统的研发.doc

2021-2022年收藏的精品资料二O0九年湖南省会计资格考试试卷会计基础试题及答案.doc

基于-weka的数据分类和聚类分析实验报告.doc

计算机控制技术发展综述报告.doc

大数据人工智能论文.doc 对人工智能进行详细解读，引入了一些当前科技前沿

06574 电力公司专业安全培训教材（DOC 7页）.doc

公司经营管理方针.doc

[2012年国赛MATLAB创新奖A题]桂林理工大学-葡萄酒的评价.doc

计算机控制技术基础知识复习(共12页).doc

计算机控制技术试卷及答案(仅供参考).doc

基于大数据与云计算的配电网设备状态监测与故障诊断关键技术研究申请书及可研报告.doc

税控收款机国标GB18240.1-3电子文档

最新资源