FedDG：医学图像分割的联邦域泛化通过连续频率空间中的情景学习资源-CSDN文库

108 浏览量 2024-07-23 08:49:58 上传评论收藏 15.86MB PDF 举报

资源推荐

资源详情

资源评论

FedDG: Federated Domain Generalization on Medical Image Segmentation

via Episodic Learning in Continuous Frequency Space

Quande Liu

, Cheng Chen

, Jing Qin

, Qi Dou

, Pheng-Ann Heng

Department of Computer Science and Engineering, The Chinese University of Hong Kong

School of Nursing, The Hong Kong Polytechnic University

{qdliu, cchen, qdou, pheng}@cse.cuhk.edu.hk, harry.qin@polyu.edu.hk

Abstract

Federated learning allows distributed medical institu-

tions to collaboratively learn a shared prediction model

with privacy protection. While at clinical deployment, the

models trained in federated learning can still suffer from

performance drop when applied to completely unseen hos-

pitals outside the federation. In this paper, we point out and

solve a novel problem setting of federated domain gener-

alization (FedDG), which aims to learn a federated model

from multiple distributed source domains such that it can

directly generalize to unseen target domains. We present

a novel approach, named as Episodic Learning in Contin-

uous Frequency Space (ELCFS), for this problem by en-

abling each client to exploit multi-source data distributions

under the challenging constraint of data decentralization.

Our approach transmits the distribution information across

clients in a privacy-protecting way through an effective con-

tinuous frequency space interpolation mechanism. With the

transferred multi-source distributions, we further carefully

design a boundary-oriented episodic learning paradigm to

expose the local learning to domain distribution shifts and

particularly meet the challenges of model generalization in

medical image segmentation scenario. The effectiveness

of our method is demonstrated with superior performance

over state-of-the-arts and in-depth ablation experiments on

two medical image segmentation tasks. The code is avail-

able at https://github.com/liuquande/FedDG-ELCFS.

1. Introduction

Data collaboration across multiple medical institutions is

increasingly desired to build accurate and robust data-driven

deep networks for medical image segmentation [7, 18, 50].

Federated learning (FL) [20] has recently opened the door

for a promising privacy-preserving solution, which allows

training a model on distributed datasets while keeping data

Corresponding author

(a) (b)

...

: Source domain

: Unseen domain

: Data parivacy

: Central server

: Model

...

Amplitude spectrum

(frequency space)

Local client k

Local episodic learning

Local data

Transformed data with

multi-source distributions

Figure 1. (a) The novel problem setting of federated domain gen-

eralization (FedDG), which aims to learn a federated model from

multiple decentralized source domains such that it can directly

generalize to completely unseen target domains. (b) Our main

idea to tackle FedDG by transferring distribution information in

frequency space and episodic learning at each local client.

locally. The paradigm works in a way that each local client

(e.g., hospital) learns from their own data, and only aggre-

gates the model parameters at a certain frequency at the cen-

tral server to generate a global model. All the data samples

are kept within each local client during federated training.

Although FL has witnessed some pilot progress on med-

ical image segmentation tasks [4, 44, 49], all existing works

only focus on improving model performance on the internal

clients, while neglecting model generalizability onto unseen

domains outside the federation. This is a crucial problem

impeding wide applicability of FL models in real practice.

The testing medical images encountered in unseen hospi-

tals can differ signiﬁcantly from the source clients in terms

of data distributions, due to the variations in imaging scan-

ners and protocols. How to generalize the federated model

under such distribution shifts is technically challenging yet

unexplored so far. In this work, we identify the novel prob-

lem setting of Federated Domain Generalization (FedDG),

which aims to learn a federated model from multiple decen-

tralized source domains such that it can directly generalize

to completely unseen domains, as illustrated in Fig. 1 (a).

Unseen domain generalization (DG) is an active research

topic with different methods being proposed [3, 8, 11, 24,

25, 26, 29, 37, 43], but the federated paradigm with dis-

1013

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

DOI 10.1109/CVPR46437.2021.00107

Authorized licensed use limited to: Xi'an Univ of Posts & Telecom. Downloaded on July 15,2022 at 16:41:56 UTC from IEEE Xplore. Restrictions apply.

FedDG: Federated Domain Generalization on Medical Image Segmentation

via Episodic Learning in Continuous Frequency Space

Quande Liu

, Cheng Chen

, Jing Qin

, Qi Dou

, Pheng-Ann Heng

Department of Computer Science and Engineering, The Chinese University of Hong Kong

School of Nursing, The Hong Kong Polytechnic University

{qdliu, cchen, qdou, pheng}@cse.cuhk.edu.hk, harry.qin@polyu.edu.hk

Abstract

Federated learning allows distributed medical institu-

tions to collaboratively learn a shared prediction model

with privacy protection. While at clinical deployment, the

models trained in federated learning can still suffer from

performance drop when applied to completely unseen hos-

pitals outside the federation. In this paper, we point out and

solve a novel problem setting of federated domain gener-

alization (FedDG), which aims to learn a federated model

from multiple distributed source domains such that it can

directly generalize to unseen target domains. We present

a novel approach, named as Episodic Learning in Contin-

uous Frequency Space (ELCFS), for this problem by en-

abling each client to exploit multi-source data distributions

under the challenging constraint of data decentralization.

Our approach transmits the distribution information across

clients in a privacy-protecting way through an effective con-

tinuous frequency space interpolation mechanism. With the

transferred multi-source distributions, we further carefully

design a boundary-oriented episodic learning paradigm to

expose the local learning to domain distribution shifts and

particularly meet the challenges of model generalization in

medical image segmentation scenario. The effectiveness

of our method is demonstrated with superior performance

over state-of-the-arts and in-depth ablation experiments on

two medical image segmentation tasks. The code is avail-

able at https://github.com/liuquande/FedDG-ELCFS.

1. Introduction

Data collaboration across multiple medical institutions is

increasingly desired to build accurate and robust data-driven

deep networks for medical image segmentation [7, 18, 50].

Federated learning (FL) [20] has recently opened the door

for a promising privacy-preserving solution, which allows

training a model on distributed datasets while keeping data

Corresponding author

(a) (b)

...

: Source domain

: Unseen domain

: Data parivacy

: Central server

: Model

...

Amplitude spectrum

(frequency space)

Local client k

Local episodic learning

Local data

Transformed data with

multi-source distributions

Figure 1. (a) The novel problem setting of federated domain gen-

eralization (FedDG), which aims to learn a federated model from

multiple decentralized source domains such that it can directly

generalize to completely unseen target domains. (b) Our main

idea to tackle FedDG by transferring distribution information in

frequency space and episodic learning at each local client.

locally. The paradigm works in a way that each local client

(e.g., hospital) learns from their own data, and only aggre-

gates the model parameters at a certain frequency at the cen-

tral server to generate a global model. All the data samples

are kept within each local client during federated training.

Although FL has witnessed some pilot progress on med-

ical image segmentation tasks [4, 44, 49], all existing works

only focus on improving model performance on the internal

clients, while neglecting model generalizability onto unseen

domains outside the federation. This is a crucial problem

impeding wide applicability of FL models in real practice.

The testing medical images encountered in unseen hospi-

tals can differ signiﬁcantly from the source clients in terms

of data distributions, due to the variations in imaging scan-

ners and protocols. How to generalize the federated model

under such distribution shifts is technically challenging yet

unexplored so far. In this work, we identify the novel prob-

lem setting of Federated Domain Generalization (FedDG),

which aims to learn a federated model from multiple decen-

tralized source domains such that it can directly generalize

to completely unseen domains, as illustrated in Fig. 1 (a).

Unseen domain generalization (DG) is an active research

topic with different methods being proposed [3, 8, 11, 24,

25, 26, 29, 37, 43], but the federated paradigm with dis-

1013

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

DOI 10.1109/CVPR46437.2021.00107

Authorized licensed use limited to: Xi'an Univ of Posts & Telecom. Downloaded on July 15,2022 at 16:41:56 UTC from IEEE Xplore. Restrictions apply.

FedDG：医学图像分割的联邦域泛化

通过连续频率空间中的情景学习

刘全德1,程辰1,景勤2,齐斗1,*,彭安恒1

1香港中文大学计算机科学与工程系

2香港理工大学护理学院

联邦学习允许分布式医疗机构

协作学习具有隐私保护的共享预测模型。在临床部署

时，在联邦学习中训练的模型在应用于联邦以外完全

看不见的医院时仍然会出现性能下降。在本文中，我

们指出并解决了联邦域泛化（FedDG）的新问题设置

，旨在从多个分布式源域中学习联邦模型，使其可以

直接泛化到看不见的目标域。我们提出了一种新颖的

方法，称为连续频率空间中的情节学习（ELCFS），

通过使每个客户端能够在数据分散的挑战性约束下利

用多源数据分布来解决这个问题。我们的方法通过有

效的连续频率空间插值机制以保护隐私的方式在客户

端之间传输分布信息。通过转移的多源分布，我们进

一步精心设计了一种面向边界的情景学习范式，以将

局部学习暴露于域分布的变化，特别是应对医学图像

分割场景中模型泛化的挑战。

成效

在两个医学图像分割任务上，我们的方法的性能优于

最先进的和深入的消融实验。该代码可在https：github.

comliuquandeFedDG-ELCFS获得。

跨多个医疗机构的数据协作是

越来越需要为医学图像分割[7,18,50]建立准确和强大的

数据驱动深度网络。联邦学习(FL)[20]最近为一种有前

途的隐私保护解决方案打开了大门，该解决方案允许

在分布式数据集上训练模型，同时保留数据

*通讯作者

:源域

:看不见的域

:中央服务器

幅度谱（频率空间）

本地客户端k

局部情景学习

本地数据

具有多源分布的转换数据

图1.(a)联邦域泛化(FedDG)的新问题设置，旨在从多个分散的

源域中学习联邦模型，使其可以直接泛化到完全看不见的目

标域。(b)我们通过在每个本地客户端传输频率空间中的分布

信息和情景学习来解决FedDG的主要想法。

本地。该范式的工作方式是每个本地客户端（例如医

院）从自己的数据中学习，并且仅在中央服务器上以

特定频率聚合模型参数以生成全局模型。在联合训练

期间，所有数据样本都保存在每个本地客户端中。

尽管FL见证了一些医学试点进展

ical图像分割任务[4,44,49]，所有现有的工作都只专注于

提高内部客户端的模型性能，而忽略了模型在联邦以

外看不见的域上的泛化性。这是阻碍FL模型在实际实

践中广泛适用的关键问题。由于成像扫描仪和协议的

差异，在未见过的医院中遇到的测试医学图像在数据

分布方面可能与源客户端有很大不同。如何在这种分

布变化下推广联合模型在技术上具有挑战性，但迄今

为止尚未探索。在这项工作中，我们确定了联邦域泛

化(FedDG)的新问题设置，其旨在从多个分散的源域中

学习一个联邦模型，以便它可以直接泛化到完全看不

见的域，如图1(a)所示.

看不见的领域泛化（DG）是一项积极的研究

提出了不同方法的主题[3,8,11,24,25,26,29,37,43]，但联邦范

式与dis-

2021IEEECVF计算机视觉和模式识别会议(CVPR)

计

算

机

视

觉

和

模

式

识

别

会

议

-4

31.

00

:10

.11

7.2

1.0

授权许可使用仅限于：西安邮电大学。从IEEEXplore于2022年7月15日16:41:56UTC下载。有限制。

tributed data sources poses new challenges for DG. With

the goal to extract representations that are robust to distri-

bution shift, existing DG approaches usually require access

to multi-source distributions in the learning process. For in-

stance, adversarial feature alignment methods [26, 29] have

to train the domain discriminator with samples from differ-

ent source datasets. Meta-learning based methods [8, 24]

need to use multi-source data of different distributions to

construct virtual training and virtual testing domains within

each minibatch. Whereas in federated paradigm, data are

stored distributedly and the learning at each client can only

access its local data. Therefore, current DG methods are

typically not applicable in FedDG scenario. In addition, the

local optimization would make model biased to its own data

distribution, thus less generalizable to new target domains.

To solve this FedDG problem, our insight is to enable

each client to access multi-source data distributions in a

privacy-protecting way. The idea is motivated by the knowl-

edge that the low-level distributions (i.e., style) and high-

level semantics of an image can be respectively captured

by amplitude and phase spectrum in the frequency space, as

revealed by visual psychophysics [13, 42, 57]. We can con-

sider exchanging these amplitude spectrum across clients to

transmit the distribution information (cf. Fig. 1 (b)), while

keeping the phase spectrum with core semantics locally for

privacy protection. Based on this, we also devise a con-

tinuous frequency space interpolation mechanism, which

interpolates between the local and transferred distributions

for enriching the established multi-domain distributions for

each local client. This promotes the local training to gain

domain-invariance beneﬁting from a dedicated dense distri-

bution space. With these established distributions, we ex-

pose the local learning to domain distribution shifts via an

episodic training paradigm to enhance the generalizability

of local parameters. A novel meta-update objective function

is designed to guide cross-domain optimization attending to

the boundary area. This is notably important for medical

image segmentation applications where generalization er-

rors often come from imprecise predictions at ambiguous

boundary of anatomies.

Our main contributions are highlighted as follows:

• We tackle the novel and practical problem of Feder-

ated Domain Generalization. To the best of our knowl-

edge, this is the ﬁrst work to improve generalizability

on completely unseen domains for federated models.

• We propose a privacy-preserving solution to learn the

generalizable FL model under decentralized datasets,

through an effective continuous frequency space inter-

polation mechanism across clients.

• We present a novel boundary-oriented episodic learn-

ing scheme for the local training at a client, which ex-

poses local optimization to domain shifts and enhances

model generalizability at ambiguous boundary area.

• We conduct extensive experiments on two typical med-

ical image segmentation tasks, i.e., retinal fundus im-

age segmentation (four datasets) and prostate MRI seg-

mentation (six datasets). Our achieved superior per-

formance over state-of-the-arts and in-depth analytical

experiments demonstrate the efﬁcacy of our approach.

2. Related Work

2.1. Federated Learning in Medical Imaging

Federated learning [15, 20, 36, 56] provides a promis-

ing privacy-preserving solution for multi-site data collabo-

ration, which develops a global model from decentralized

datasets by aggregating the parameters of each local client

while keeping data locally. Representatively, McMahan et

al. [36] propose the popular federated averaging algorithm

for communication-efﬁcient federated training of deep net-

works. With the advantage of privacy protection, FL has

recently drawn increasing interests in medical image appli-

cations [4, 18, 22, 27, 45, 49, 51]. Sheller et al. [49] is a pilot

study to investigate the collaborative model training without

sharing patient data for the multi-site brain tumor segmenta-

tion. Later on, Li et al. [27] further compare several weights

sharing strategies in FL to alleviate the effect of data imbal-

ance among different hospitals. However, these works all

focus on improving performance on internal clients, with-

out considering the generalization issue for unseen domains

outside the federation, which is crucial for wide clinical us-

ability. Latest literature has studied a related problem of

unsupervised domain adaptation in FL paradigm [28, 41],

whereas these methods typically require data from the target

domain to adapt the model. In practice, it would be time-

consuming or even impractical to collect data from each

new hospital before model deployment. Instead, our tack-

led new problem setting of FedDG aims to directly gener-

alize the federated model to completely unseen domains, in

which no prior knowledge from the target domain is needed.

2.2. Domain Generalization

Domain generalization [5, 9, 12, 14, 43, 47, 58, 59]

aims to learn a model from multiple source domains

such that it can directly generalize to unseen target do-

mains. Among previous efforts, some methods aim to

learn domain-invariant representations by minimizing the

domain discrepancy across multiple source domains [11,

16, 26, 29, 32, 37, 38, 55]. For example, Motiian et

al. [37] utilize a contrastive loss to minimize the distance

between samples from the same class but different domains.

Some other DG methods are based on meta-learning, which

is an episodic training paradigm by creating meta-train

and meta-test splits at each iteration to stimulate domain

shift [1, 8, 24, 30]. Li et al. [30] employ meta-learning

1014

Authorized licensed use limited to: Xi'an Univ of Posts & Telecom. Downloaded on July 15,2022 at 16:41:56 UTC from IEEE Xplore. Restrictions apply.

tributed data sources poses new challenges for DG. With

the goal to extract representations that are robust to distri-

bution shift, existing DG approaches usually require access

to multi-source distributions in the learning process. For in-

stance, adversarial feature alignment methods [26, 29] have

to train the domain discriminator with samples from differ-

ent source datasets. Meta-learning based methods [8, 24]

need to use multi-source data of different distributions to

construct virtual training and virtual testing domains within

each minibatch. Whereas in federated paradigm, data are

stored distributedly and the learning at each client can only

access its local data. Therefore, current DG methods are

typically not applicable in FedDG scenario. In addition, the

local optimization would make model biased to its own data

distribution, thus less generalizable to new target domains.

To solve this FedDG problem, our insight is to enable

each client to access multi-source data distributions in a

privacy-protecting way. The idea is motivated by the knowl-

edge that the low-level distributions (i.e., style) and high-

level semantics of an image can be respectively captured

by amplitude and phase spectrum in the frequency space, as

revealed by visual psychophysics [13, 42, 57]. We can con-

sider exchanging these amplitude spectrum across clients to

transmit the distribution information (cf. Fig. 1 (b)), while

keeping the phase spectrum with core semantics locally for

privacy protection. Based on this, we also devise a con-

tinuous frequency space interpolation mechanism, which

interpolates between the local and transferred distributions

for enriching the established multi-domain distributions for

each local client. This promotes the local training to gain

domain-invariance beneﬁting from a dedicated dense distri-

bution space. With these established distributions, we ex-

pose the local learning to domain distribution shifts via an

episodic training paradigm to enhance the generalizability

of local parameters. A novel meta-update objective function

is designed to guide cross-domain optimization attending to

the boundary area. This is notably important for medical

image segmentation applications where generalization er-

rors often come from imprecise predictions at ambiguous

boundary of anatomies.

Our main contributions are highlighted as follows:

• We tackle the novel and practical problem of Feder-

ated Domain Generalization. To the best of our knowl-

edge, this is the ﬁrst work to improve generalizability

on completely unseen domains for federated models.

• We propose a privacy-preserving solution to learn the

generalizable FL model under decentralized datasets,

through an effective continuous frequency space inter-

polation mechanism across clients.

• We present a novel boundary-oriented episodic learn-

ing scheme for the local training at a client, which ex-

poses local optimization to domain shifts and enhances

model generalizability at ambiguous boundary area.

• We conduct extensive experiments on two typical med-

ical image segmentation tasks, i.e., retinal fundus im-

age segmentation (four datasets) and prostate MRI seg-

mentation (six datasets). Our achieved superior per-

formance over state-of-the-arts and in-depth analytical

experiments demonstrate the efﬁcacy of our approach.

2. Related Work

2.1. Federated Learning in Medical Imaging

Federated learning [15, 20, 36, 56] provides a promis-

ing privacy-preserving solution for multi-site data collabo-

ration, which develops a global model from decentralized

datasets by aggregating the parameters of each local client

while keeping data locally. Representatively, McMahan et

al. [36] propose the popular federated averaging algorithm

for communication-efﬁcient federated training of deep net-

works. With the advantage of privacy protection, FL has

recently drawn increasing interests in medical image appli-

cations [4, 18, 22, 27, 45, 49, 51]. Sheller et al. [49] is a pilot

study to investigate the collaborative model training without

sharing patient data for the multi-site brain tumor segmenta-

tion. Later on, Li et al. [27] further compare several weights

sharing strategies in FL to alleviate the effect of data imbal-

ance among different hospitals. However, these works all

focus on improving performance on internal clients, with-

out considering the generalization issue for unseen domains

outside the federation, which is crucial for wide clinical us-

ability. Latest literature has studied a related problem of

unsupervised domain adaptation in FL paradigm [28, 41],

whereas these methods typically require data from the target

domain to adapt the model. In practice, it would be time-

consuming or even impractical to collect data from each

new hospital before model deployment. Instead, our tack-

led new problem setting of FedDG aims to directly gener-

alize the federated model to completely unseen domains, in

which no prior knowledge from the target domain is needed.

2.2. Domain Generalization

Domain generalization [5, 9, 12, 14, 43, 47, 58, 59]

aims to learn a model from multiple source domains

such that it can directly generalize to unseen target do-

mains. Among previous efforts, some methods aim to

learn domain-invariant representations by minimizing the

domain discrepancy across multiple source domains [11,

16, 26, 29, 32, 37, 38, 55]. For example, Motiian et

al. [37] utilize a contrastive loss to minimize the distance

between samples from the same class but different domains.

Some other DG methods are based on meta-learning, which

is an episodic training paradigm by creating meta-train

and meta-test splits at each iteration to stimulate domain

shift [1, 8, 24, 30]. Li et al. [30] employ meta-learning

1014

Authorized licensed use limited to: Xi'an Univ of Posts & Telecom. Downloaded on July 15,2022 at 16:41:56 UTC from IEEE Xplore. Restrictions apply.

分布式数据源对DG提出了新的挑战。为了提取对分布

变化具有鲁棒性的表示，现有的DG方法通常需要在学

习过程中访问多源分布。例如，对抗性特征对齐方法[

26,29]必须使用来自不同源数据集的样本来训练域鉴别

器。基于元学习的方法[8,24]需要使用不同分布的多源

数据在每个minibatch中构建虚拟训练和虚拟测试域。

而在联邦范式中，数据是分布式存储的，每个客户端

的学习只能访问其本地数据。因此，当前的DG方法通

常不适用于FedDG场景。此外，局部优化会使模型偏

向于其自身的数据分布，因此对新目标域的泛化性较

差。

为了解决这个FedDG问题，我们的见解是启用

每个客户端以保护隐私的方式访问多源数据分布。这

个想法的动机是这样的知识，即图像的低级分布（即

风格）和高级语义可以通过频率空间中的幅度和相位

谱分别捕获，正如视觉心理物理学所揭示的那样[13,42,

57].我们可以考虑在客户端之间交换这些幅度谱以传输

分布信息（参见图1（b）），同时在本地保持具有核

心语义的相位谱以保护隐私。基于此，我们还设计了

一种连续频率空间插值机制，在本地分布和转移分布

之间进行插值，以丰富每个本地客户端已建立的多域

分布。这促进了局部训练以获得受益于专用密集分布

空间的域不变性。通过这些已建立的分布，我们通过

情景训练范式将局部学习暴露于域分布变化，以增强

局部参数的泛化性。设计了一种新颖的元更新目标函

数来指导关注边界区域的跨域优化。这对于医学图像

分割应用非常重要，因为在这些应用中，泛化错误通

常来自对解剖结构模糊边界的不精确预测。

我们的主要贡献如下：

我们解决Feder-的新颖而实际的问题

ed域泛化。据我们所知，这是第一项提高联邦模

型在完全看不见的领域的泛化性的工作。

我们提出了一个隐私保护解决方案来学习

通过跨客户端的有效连续频率空间插值机制，可

在分散数据集下推广FL模型。

我们提出了一种新的面向边界的情景学习——

用于在客户端进行本地培训的方案，该方案将本

地优化暴露给域转移并增强

模糊边界区域的模型泛化性。

我们对两种典型的药物进行了广泛的实验——

ical图像分割任务，即视网膜眼底图像分割（四个

数据集）和前列腺MRI分割（六个数据集）。我

们在最先进的技术和深入的分析实验中取得了卓

越的性能，证明了我们方法的有效性。

二、相关工作

2.1。医学影像中的联邦学习

联邦学习[15,20,36,56]提供了一个承诺

用于多站点数据协作的隐私保护解决方案，该解决方

案通过聚合每个本地客户端的参数，同时将数据保存

在本地，从分散的数据集开发全局模型。代表性的是

，麦克马汉等人。[36]提出了流行的联合平均算法，用

于深度网络的通信高效联合训练。凭借隐私保护的优

势，FL最近在医学图像应用中引起了越来越多的兴趣[

4,18,22,27,45,49,51]。谢勒等人。[49]是一项试点研究，旨

在调查协作模型训练，而无需共享患者数据以进行多

部位脑肿瘤分割。后来，李等人。[27]进一步比较了FL

中的几种权重共享策略，以减轻不同医院之间数据不

平衡的影响。然而，这些工作都专注于提高内部客户

端的性能，没有考虑联邦以外看不见的域的泛化问题

，这对于广泛的临床可用性至关重要。最新文献研究

了FL范式中无监督域适应的相关问题[28,41]，而这些方

法通常需要来自目标域的数据来适应模型。在实践中

，在模型部署之前从每家新医院收集数据会很耗时，

甚至不切实际。相反，我们解决的FedDG新问题设置

旨在将联合模型直接推广到完全看不见的领域，其中

不需要来自目标领域的先验知识。

2.2.领域泛化

领域泛化[5,9,12,14,43,47,58,59]

旨在从多个源域学习模型，以便它可以直接泛化到看

不见的目标域。

在以前的努力中，一些方法旨在

通过最小化多个源域[11,16,26,29,32,37,38,55]之间的域差异

来学习域不变表示。

例如，Motiian等

人。[37]利用对比损失来最小化来自同一类但不同域的

样本之间的距离。其他一些DG方法基于元学习，这是

一种情景式训练范式，通过在每次迭代中创建元训练

和元测试拆分来刺激域转移[1、8、24、30]。李等人。

[30]采用元学习

授权许可使用仅限于：西安邮电大学。从IEEEXplore于2022年7月15日16:41:56UTC下载。有限制。

to learn an auxiliary loss that guides the feature extractor

to learn more generalized features. However, these meth-

ods typically require centralizing multi-domain data in one

place for learning, which violates privacy protection in fed-

erated learning setting with decentralized datasets.

There are other methods tackling DG by manipulating

deep neural network architectures [19, 23, 35], leveraging

self-supervision signals [3, 54], designing training heuris-

tics [17, 25], or conducting data augmentations [48, 53, 60,

61], which are free from requirement of data centralization.

Representatively, Carlucci et al. [3] adopt self-supervised

learning by solving jigsaw puzzles. Zhang et al. [60] con-

duct extensive data augmentations on each source domain

by stacking a series of transformations. These approaches,

when applied in FL paradigm, can helpfully act as regular-

izations for the local training with individual source domain

data, yet hardly exploit the rich data distributions across

domains. Our method instead, aims to transfer the distri-

bution information across clients to make full use of the

multi-source distributions towards FedDG. We also experi-

mentally compare with these typical methods under the FL

setting with superior performance demonstrated.

3. Method

We start with the formulation for federated domain gen-

eralization and its challenges in medical image segmen-

tation scenario. We then describe the proposed method

Episodic Learning in Continuous Frequency Space (EL-

CFS) to explicitly address these challenges. An overview

of the method is shown in Fig. 2.

3.1. Federated Domain Generalization

Preliminaries: In FedDG, we denote (X , Y) as the joint

image and label space of a task, S = {S

, S

, ..., S

} as

the set of K distributed source domains involved in feder-

ated learning. Each domain contains data and label pairs

of S

= {(x

, y

)}

i=1

, which are sampled from a domain-

speciﬁc distribution (X

, Y). The goal of FedDG is to learn

a model f

: X → Y using the K distributed source do-

mains, such that it can directly generalize to a completely

unseen testing domain T with a high performance.

Standard federated learning paradigm involves the com-

munication between a central server and the K local clients.

At each federated round t, every client k will receive the

same global model weights θ from the central server and

update the model with their local data S

for E epochs.

The central server then collects the local parameters θ

from

all clients and aggregates them to update the global model.

This process repeats until the global model converges. In

this work, we consider the most popular federated averaging

algorithm (FedAvg) [36], which aggregates the local param-

eters with weights in proportional to the size of each local

dataset to update the global model, i.e., θ =

k=1

where N =

k=1

. It is worth noting that our method

can also be ﬂexibly incorporated to other FL backbones.

Challenges: With the goal of unseen domain general-

ization, a model is expected to thoroughly investigate the

multi-source data distributions to pursue domain-invariance

of its learned latent space. However, the federated setting

in the speciﬁc medical image segmentation scenario poses

several challenges for that. First, the multi-source data in

FL are stored distributedly and the learning at each client

can only access its individual local distribution, which con-

strains to make full use of the multi-source distributions to

learn generalizable parameters. Second, though FL has col-

laborated multi-source data, the medical images acquired

from different clinical sites can present large heterogeneity.

This leads to distinct distributions among the collaborative

datasets, which is insufﬁcient to ensure domain invariance

in a more continuous distribution space to attain good gen-

eralizability in complex clinical environments. Third, the

structure of medical anatomises usually present high ambi-

guity around its boundary region, raising challenge for pre-

vious DG techniques that typically lacks assurance for the

domain-invariance of features in such ambiguous region.

3.2. Continuous Frequency Space Interpolation

To address the restriction of decentralized datasets, the

foundation of our solution is to exchange the distribution in-

formation across clients, such that each local client can get

access to multi-source data distributions for learning gener-

alizable parameters. Considering that sharing raw images is

forbidden, we propose to exploit the information inherent in

the frequency space, which enables to separate the distribu-

tion (i.e. style) information from the original images to be

shared between clients without privacy leakage.

Speciﬁcally, given a sample x

∈ R

H×W ×C

(C = 3 for

RGB image and C = 1 for grey-scale image) from the k-th

client, we can obtain its frequency space signal through fast

Fourier transform [39] as:

F(x

)(u, v, c) =

H−1

h=0

W −1

w=0

(h, w, c)e

−j2π(

. (1)

This frequency space signal F(x

) can be further decom-

posed to an amplitude spectrum A

∈ R

H×W ×C

and a

phase spectrum P

∈ R

H×W ×C

, which respectively reﬂect

the low-level distributions (e.g. style) and high-level seman-

tics (e.g. object) of the image. To exchange the distribution

information across clients, we ﬁrst construct a distribution

bank A = [A

, ..., A

], where each A

= {A

}

i=1

con-

tains all amplitude spectrum of images from the k-th client,

representing the distribution of X

. This bank is then made

accessible to all clients as shared distribution knowledge.

Next, we design a continuous interpolation mechanism

within the frequency space, aiming to transmit multi-source

1015

Authorized licensed use limited to: Xi'an Univ of Posts & Telecom. Downloaded on July 15,2022 at 16:41:56 UTC from IEEE Xplore. Restrictions apply.

to learn an auxiliary loss that guides the feature extractor

to learn more generalized features. However, these meth-

ods typically require centralizing multi-domain data in one

place for learning, which violates privacy protection in fed-

erated learning setting with decentralized datasets.

There are other methods tackling DG by manipulating

deep neural network architectures [19, 23, 35], leveraging

self-supervision signals [3, 54], designing training heuris-

tics [17, 25], or conducting data augmentations [48, 53, 60,

61], which are free from requirement of data centralization.

Representatively, Carlucci et al. [3] adopt self-supervised

learning by solving jigsaw puzzles. Zhang et al. [60] con-

duct extensive data augmentations on each source domain

by stacking a series of transformations. These approaches,

when applied in FL paradigm, can helpfully act as regular-

izations for the local training with individual source domain

data, yet hardly exploit the rich data distributions across

domains. Our method instead, aims to transfer the distri-

bution information across clients to make full use of the

multi-source distributions towards FedDG. We also experi-

mentally compare with these typical methods under the FL

setting with superior performance demonstrated.

3. Method

We start with the formulation for federated domain gen-

eralization and its challenges in medical image segmen-

tation scenario. We then describe the proposed method

Episodic Learning in Continuous Frequency Space (EL-

CFS) to explicitly address these challenges. An overview

of the method is shown in Fig. 2.

3.1. Federated Domain Generalization

Preliminaries: In FedDG, we denote (X , Y) as the joint

image and label space of a task, S = {S

, S

, ..., S

} as

the set of K distributed source domains involved in feder-

ated learning. Each domain contains data and label pairs

of S

= {(x

, y

)}

i=1

, which are sampled from a domain-

speciﬁc distribution (X

, Y). The goal of FedDG is to learn

a model f

: X → Y using the K distributed source do-

mains, such that it can directly generalize to a completely

unseen testing domain T with a high performance.

Standard federated learning paradigm involves the com-

munication between a central server and the K local clients.

At each federated round t, every client k will receive the

same global model weights θ from the central server and

update the model with their local data S

for E epochs.

The central server then collects the local parameters θ

from

all clients and aggregates them to update the global model.

This process repeats until the global model converges. In

this work, we consider the most popular federated averaging

algorithm (FedAvg) [36], which aggregates the local param-

eters with weights in proportional to the size of each local

dataset to update the global model, i.e., θ =

k=1

where N =

k=1

. It is worth noting that our method

can also be ﬂexibly incorporated to other FL backbones.

Challenges: With the goal of unseen domain general-

ization, a model is expected to thoroughly investigate the

multi-source data distributions to pursue domain-invariance

of its learned latent space. However, the federated setting

in the speciﬁc medical image segmentation scenario poses

several challenges for that. First, the multi-source data in

FL are stored distributedly and the learning at each client

can only access its individual local distribution, which con-

strains to make full use of the multi-source distributions to

learn generalizable parameters. Second, though FL has col-

laborated multi-source data, the medical images acquired

from different clinical sites can present large heterogeneity.

This leads to distinct distributions among the collaborative

datasets, which is insufﬁcient to ensure domain invariance

in a more continuous distribution space to attain good gen-

eralizability in complex clinical environments. Third, the

structure of medical anatomises usually present high ambi-

guity around its boundary region, raising challenge for pre-

vious DG techniques that typically lacks assurance for the

domain-invariance of features in such ambiguous region.

3.2. Continuous Frequency Space Interpolation

To address the restriction of decentralized datasets, the

foundation of our solution is to exchange the distribution in-

formation across clients, such that each local client can get

access to multi-source data distributions for learning gener-

alizable parameters. Considering that sharing raw images is

forbidden, we propose to exploit the information inherent in

the frequency space, which enables to separate the distribu-

tion (i.e. style) information from the original images to be

shared between clients without privacy leakage.

Speciﬁcally, given a sample x

∈ R

H×W ×C

(C = 3 for

RGB image and C = 1 for grey-scale image) from the k-th

client, we can obtain its frequency space signal through fast

Fourier transform [39] as:

F(x

)(u, v, c) =

H−1

h=0

W −1

w=0

(h, w, c)e

−j2π(

. (1)

This frequency space signal F(x

) can be further decom-

posed to an amplitude spectrum A

∈ R

H×W ×C

and a

phase spectrum P

∈ R

H×W ×C

, which respectively reﬂect

the low-level distributions (e.g. style) and high-level seman-

tics (e.g. object) of the image. To exchange the distribution

information across clients, we ﬁrst construct a distribution

bank A = [A

, ..., A

], where each A

= {A

}

i=1

con-

tains all amplitude spectrum of images from the k-th client,

representing the distribution of X

. This bank is then made

accessible to all clients as shared distribution knowledge.

Next, we design a continuous interpolation mechanism

within the frequency space, aiming to transmit multi-source

1015

Authorized licensed use limited to: Xi'an Univ of Posts & Telecom. Downloaded on July 15,2022 at 16:41:56 UTC from IEEE Xplore. Restrictions apply.

学习引导特征提取器学习更多广义特征的辅助损失。

然而，这些方法通常需要将多域数据集中在一个地方

进行学习，这违反了具有分散数据集的联邦学习设置

中的隐私保护。

还有其他通过操纵来处理DG的方法

深度神经网络架构[19,23,35]，利用自我监督信号[3,54]，

设计训练启发式[17,25]，或进行数据增强[48,53,60,61]，

这些都没有数据集中化的要求。代表性的是，Carlucci

等人。[3]通过解决拼图采用自我监督学习。张等人。[

60]通过堆叠一系列转换对每个源域进行广泛的数据增

强。这些方法在FL范式中应用时，可以有助于作为具

有单个源域数据的本地训练的正则化，但几乎不能利

用跨域的丰富数据分布。相反，我们的方法旨在跨客

户端传输分布信息，以充分利用对FedDG的多源分布

。我们还在FL设置下通过实验与这些典型方法进行了

比较，并展示了卓越的性能。

我们从联邦域生成的公式开始

医学图像分割场景中的迭代及其挑战。

然后我们描述所提出的方法

连续频率空间(ELCFS)中的情景学习，以明确解决这些

挑战。该方法的概要如图2所示。

3.1。联合域泛化

预备：在FedDG中，我们将(X,Y)表示为联合

任务的图像和标签空间，S={S1,S2,...,SK}作为联邦学习中

涉及的K个分布式源域的集合。每个域包含Sk={(xk

i=1，它们是从域中采样的-

具体分布（Xk，Y）。FedDG的目标是使用K个分布式

源域学习一个模型fθ:X→Y，以便它可以直接泛化到

具有高性能的完全看不见的测试域T。

标准联邦学习范式涉及

中央服务器和K个本地客户端之间的通信。在每个联邦

轮t中，每个客户端k将从中央服务器接收相同的全局

模型权重θ，并使用其本地数据Sk更新模型Eepoch。

然后中央服务器从所有客户端收集本地参数θk并聚合

它们以更新全局模型。这个过程一直重复，直到全局

模型收敛。在这项工作中，我们考虑最流行的联合平

均算法（FedAvg）[36]，它聚合局部参数，其权重与每

个局部的大小成正比

更新全局模型的数据集，即θ=K

k=1Nk。值得注意的是，我们的方法

也可以灵活地结合到其他FL主链中。

挑战：以看不见的领域通用为目标-

化，一个模型有望彻底研究多源数据分布，以追求其

学习的潜在空间的域不变性。然而，特定医学图像分

割场景中的联合设置对此提出了一些挑战。首先，FL

中的多源数据是分布式存储的，每个客户端的学习只

能访问其各自的局部分布，这限制了充分利用多源分

布来学习泛化参数。其次，虽然FL已经协作了多源数

据，但从不同临床站点获取的医学图像可能呈现出很

大的异质性。这导致协作数据集之间的分布不同，这

不足以确保在更连续的分布空间中的域不变性，从而

在复杂的临床环境中获得良好的泛化性。第三，医学

解剖结构通常在其边界区域周围呈现高度模糊性，这

对以前的DG技术提出了挑战，这些技术通常缺乏对此

类模糊区域中特征的域不变性的保证。

3.2.连续频率空间插值

为了解决分散数据集的限制，

我们的解决方案的基础是在客户端之间交换分布信息

，这样每个本地客户端都可以访问多源数据分布以学

习可泛化的参数。考虑到共享原始图像是被禁止的，

我们建议利用频率空间中固有的信息，这使得能够将

分布（即风格）信息与原始图像分离，以便在客户端

之间共享而不会泄露隐私。

具体来说，给定一个样本xk

来自第k个客户端的RGB图像和灰度图像C=1），我们

可以通过快速傅里叶变换[39]获得其频率空间信号：

这个频率空间信号F(xk i)可以进一步分解

对幅度谱Ak构成

相位谱Pk i∈RH×W×C，分别反映

图像的低级分布（例如样式）和高级语义（例如对象

）。为了在客户之间交换分销信息，我们首先构建一

个分销银行A=[A1,...,AK]，其中每个Ak={Ak

包含来自第k个客户端的图像的所有幅度谱，表示Xk

的分布。然后，所有客户都可以使用该银行作为共享

的分销知识。

接下来，我们设计一个连续插值机制

在频率空间内，旨在传输多源

授权许可使用仅限于：西安邮电大学。从IEEEXplore于2022年7月15日16:41:56UTC下载。有限制。

剩余10页未读，继续阅读

评论收藏

内容反馈

AI智博信息

粉丝: 1493
资源: 238

FedDG：医学图像分割的联邦域泛化通过连续频率空间中的情景学习

最新资源