【论文+代码】Swin-UnetUnet-LikePureTransformerforMedicalImageSe资源-CSDN文库

共2个文件

zip：1个

pdf：1个

深度学习

transformer

171 浏览量 2024-05-11 16:43:57 上传评论 1 收藏 3.84MB ZIP 举报

资源推荐

资源详情

资源评论

收起资源包目录

【论文+代码】Swin-Unet Unet-Like Pure Transformer for Medical Image Segmentation.zip （2个子文件）

Swin-Unet-main.zip 28KB

【8-2022】Swin-Unet Unet-Like Pure Transformer for Medical Image Segmentation.pdf 3.97MB

Swin-Unet: Unet-Like Pure Transformer

for Medical Image Segmentation

Hu Cao

, Yueyue Wang

, Joy Chen

, Dongsheng Jiang

)

, Xiaopeng Zhang

Qi Tian

)

, and Manning Wang

)

Technische Universit¨at M¨unchen, M¨unchen, Germany

hu.cao@tum.de

Fudan University, Shanghai, China

{yywang17,mnwang}@fudan.edu.cn

Johns Hopkins University, Baltimore, MD, USA

Huawei Technologies, Shanghai, China

dongsheng

jiang@outlook.com, tian.qi1@huawei.com

Abstract. In the past few years, convolutional neural networks (CNNs)

have achieved milestones in medical image analysis. In particular, deep

neural networks based on U-shaped architecture and skip-connections

have been widely applied in various medical image tasks. However,

although CNN has achieved excellent performance, it cannot learn global

semantic information interaction well due to the locality of convolution

operation. In this paper, we propose Swin-Unet, which is an Unet-like

pure Transformer for medical image segmentation. The tokenized image

patches are fed into the Transformer-based U-shaped Encoder-Decoder

architecture with skip-connections for local-global semantic feature learn-

ing. Speciﬁcally, we use a hierarchical Swin Transformer with shifted

windows as the encoder to extract context features. And a symmet-

ric Swin Transformer-based decoder with a patch expanding layer is

designed to perform the up-sampling operation to restore the spatial

resolution of the feature maps. Under the direct down-sampling and up-

sampling of the inputs and outputs by 4×, experiments on multi-organ

and cardiac segmentation tasks demonstrate that the pure Transformer-

based U-shaped Encoder-Decoder network outperforms those methods

with full-convolution or the combination of transformer and convolu-

tion. The codes have been publicly available at the link (https://github.

com/HuCaoFighting/Swin-Unet).

Keywords: Transformer

· Self-attention · Medical image segmentation

1 Introduction

Beneﬁting from the development of deep learning, computer vision technology

has been widely used in medical image analysis. Image segmentation is an impor-

H. Cao and Y. Wang—Work done as an intern in Huawei Technologies.

 The Author(s), under exclusive license to Springer Nature Switzerland AG 2023

L. Karlinsky et al. (Eds.): ECCV 2022 Workshops, LNCS 13803, pp. 205–218, 2023.

https://doi.org/10.1007/978-3-031-25066-8

206 H. Cao et al.

tant part of medical image analysis. Accurate and robust medical image segmen-

tation can play a cornerstone role in computer-aided diagnosis and image-guided

clinical surgery [2,11].

Existing medical image segmentation methods mainly rely on fully convolu-

tional neural networks (FCNNs) with U-shaped structure [7,16,21]. The typical

U-shaped network, U-Net [21], consists of a symmetric Encoder-Decoder with

skip connections. In the encoder, a series of convolutional layers and continuous

down-sampling layers are used to extract deep features with large receptive ﬁelds.

Then, the decoder up-samples the extracted deep features to the input resolution

for pixel-level semantic prediction, and the high-resolution features of diﬀerent

scale from the encoder are fused with skip connections to alleviate the loss of

spatial information caused by down-sampling. With such an elegant structural

design, U-Net has achieved great success in various medical imaging applica-

tions. Following this technical route, many algorithms such as 3D U-Net [4],

Res-UNet [31], U-Net++ [35] and UNet3+ [15] have been developed for image

and volumetric segmentation. The excellent performance of these FCNN-based

methods in cardiac segmentation, organ segmentation and lesion segmentation

proves that CNN has a strong ability of learning discriminating features.

Currently, although the CNN-based methods have achieved excellent perfor-

mance in the ﬁeld of medical image segmentation, there still cannot fully meet

the strict requirements of medical applications for segmentation accuracy. Image

segmentation is still a challenge task in medical image analysis. Since the intrin-

sic locality of convolution operation, it is diﬃcult for CNN-based approaches to

learn explicit global semantic information interaction [2]. Some studies have tried

to address this problem by using atrous convolutional layers [3,9], self-attention

mechanisms [22,30], and image pyramids [34]. However, these methods still have

limitations in modeling long - range dependencies. Recently, inspired by Trans-

former’s great success in the nature language processing (NLP) domain [27],

researchers have tried to bring Transformer into the vision domain [1]. In [6],

vision transformer (ViT) is proposed to perform the image recognition task.

Taking 2D image patches with positional embeddings as inputs and pre-training

on large dataset, ViT achieved comparable performance with the CNN-based

methods. Besides, data-eﬃcient image transformer (DeiT) is presented in [23],

which indicates that Transformer can be trained on mid-size datasets and that

a more robust Transformer can be obtained by combining it with the distilla-

tion method. In [18], a hierarchical Swin Transformer is developed. Take Swin

Transformer as vision backbone, the authors of [18] achieved state-of-the-art

performance on Image classiﬁcation, object detection and semantic segmenta-

tion. The success of ViT, DeiT and Swin Transformer in image recognition task

demonstrates the potential for Transformer to be applied in the vision domain.

Motivated by the Swin Transformer’s [18] success, we propose Swin-Unet

to leverage the power of Transformer for 2D medical image segmentation in

this work. To our best knowledge, Swin-Unet is a ﬁrst pure Transformer-based

U-shaped architecture that consists of encoder, bottleneck, decoder, and skip

connections. Encoder, bottleneck and decoder are all built based on Swin Trans-

former block [18]. The input medical images are split into non-overlapping image

patches. Each patch is treated as a token and fed into the Transformer-based

Swin-Unet 207

encoder to learn deep feature representations. The extracted context features

are then up-sampled by the decoder with patch expanding layer, and fused

with the multi-scale features from the encoder via skip connections, so as to

restore the spatial resolution of the feature maps and further perform segmenta-

tion prediction. Extensive experiments on multi-organ and cardiac segmentation

datasets indicate that the proposed method has excellent segmentation accuracy

and robust generalization ability. Concretely, our contributions can be summa-

rized as: (1) Based on Swin Transformer block, we build a symmetric Encoder-

Decoder architecture with skip connections. In the encoder, self-attention from

local to global is realized; in the decoder, the global features are up-sampled

to the input resolution for corresponding pixel-level segmentation prediction.

(2) A patch expanding layer is developed to achieve up-sampling and feature

dimension increase without using convolution or interpolation operation. (3) It

is found in the experiment that skip connection is eﬀective for Transformer,

so a pure Transformer-based U-shaped Encoder-Decoder architecture with skip

connection is ﬁnally constructed, named Swin-Unet.

2 Related Work

CNN-Based Methods: Early medical image segmentation methods are

mainly contour-based and traditional machine learning-based algorithms [12,25].

With the development of deep CNNs, U-Net is proposed in [21] for medical

image segmentation. Due to the simplicity and superior performance of the U-

shaped structure, various Unet-like methods are constantly emerging, such as

Res-UNet [31], Dense-UNet [17], U-Net++ [35] and UNet3+ [15]. And it is also

introduced into the ﬁeld of 3D medical image segmentation, such as 3D-Unet [4]

and V-Net [19]. Currently, CNN-based methods have achieved tremendous suc-

cess in the ﬁeld of medical image segmentation due to its powerful representation

ability.

Vision Transformers: Transformer was ﬁrst proposed for the machine trans-

lation task in [27]. In the NLP domain, the Transformer-based methods have

achieved the state-of-the-art performance in various tasks [5]. Driven by Trans-

former’s success, the researchers introduced a pioneering vision transformer

(ViT) in [6], which achieved the impressive speed-accuracy trade-oﬀ on image

recognition task. Compared with CNN-based methods, the drawback of ViT is

that it requires pre-training on its own large dataset. To alleviate the diﬃculty

in training ViT, Deit [23] describes several training strategies that allow ViT to

train well on ImageNet. Recently, several excellent works have been done based

on ViT [10,18,28]. It is worth mentioning that an eﬃcient and eﬀective hierarchi-

cal vision Transformer, called Swin Transformer, is proposed as a vision backbone

in [18]. Based on the shifted windows mechanism, Swin Transformer achieved

the state-of-the-art performance on various vision tasks including image classiﬁ-

cation, object detection and semantic segmentation. In this work, we attempt to

use Swin Transformer block as basic unit to build a U-shaped Encoder-Decoder

208 H. Cao et al.

architecture with skip connections for medical image segmentation, thus provid-

ing a benchmark comparison for the development of Transformer in the medical

image ﬁeld.

Self-attention/Transformer to Complement CNNs: In recent years,

researchers have tried to introduce self-attention mechanism into CNN to

improve the performance of the network [30]. In [22], the skip connections with

additive attention gate are integrated in U-shaped architecture to perform medi-

cal image segmentation. However, this is still the CNN-based method. Currently,

some eﬀorts are being made to combine CNN and Transformer to break the

dominance of CNNs in medical image segmentation [2,11,26]. In [2], the authors

combined Transformer with CNN to constitute a strong encoder for 2D medical

image segmentation. Similar to [2,26] and [33] use the complementarity of Trans-

former and CNN to improve the segmentation capability of the model. Currently,

various combinations of Transformer with CNN are applied in multi-modal brain

tumor segmentation [29] and 3D medical image segmentation [11,32]. Diﬀerent

from the above methods, we try to explore the application potential of pure

Transformer in medical image segmentation.

3Method

3.1 Architecture Overview

The overall architecture of the proposed Swin-Unet is presented in Fig. 1. Swin-

Unet consists of encoder, bottleneck, decoder and skip connections. The basic

unit of Swin-Unet is Swin Transformer block [18]. For the encoder, to trans-

form the inputs into sequence embeddings, the medical images are split into

non-overlapping patches with patch size of 4 × 4. By such partition approach,

the feature dimension of each patch becomes to 4 × 4 × 3 = 48. Furthermore, a

linear embedding layer is applied to projected feature dimension into arbitrary

dimension (represented as C). The transformed patch tokens pass through several

Swin Transformer blocks and patch merging layers to generate the hierarchical

feature representations. Speciﬁcally, patch merging layer is responsible for down-

sampling and increasing dimension, and Swin Transformer block is responsible

for feature representation learning. Inspired by U-Net [21], we design a symmet-

ric transformer-based decoder. The decoder is composed of Swin Transformer

block and patch expanding layer. The extracted context features are fused with

multiscale features from encoder via skip connections to complement the loss

of spatial information caused by down-sampling. In contrast to patch merging

layer, a patch expanding layer is specially designed to perform up-sampling.

The patch expanding layer reshapes feature maps of adjacent dimensions into

a large feature maps with 2× up-sampling of resolution. In the end, the last

patch expanding layer is used to perform 4× up-sampling to restore the reso-

lution of the feature maps to the input resolution (W × H), and then a linear

projection layer is applied on these up-sampled features to output the pixel-level

segmentation predictions. We would elaborate each block in the following.

Swin-Unet 209

Fig. 1. The architecture of Swin-Unet, which is composed of encoder, bottleneck,

decoder and skip connections. Encoder, bottleneck and decoder are all constructed

based on Swin transformer block.

3.2 Swin Transformer Block

Diﬀerent from the conventional multi-head self attention (MSA) module, Swin

transformer block [18] is constructed based on shifted windows. In Fig. 2, two

consecutive Swin transformer blocks are presented. Each Swin transformer block

is composed of LayerNorm (LN) layer, multi-head self attention module, residual

connection and 2-layer MLP with GELU non-linearity. The window-based multi-

head self attention (W-MSA) module and the shifted window-based multi-head

self attention (SW-MSA) module are applied in the two successive transformer

blocks, respectively. Based on such window partitioning mechanism, continuous

swin transformer blocks can be formulated as:

ˆz

= W -MSA(LN(z

l−1

)) + z

l−1

, (1)

评论收藏

内容反馈

明初啥都能学会

粉丝: 1319
资源: 13

【论文+代码】Swin-Unet Unet-Like Pure Transformer for Medical Image Se

Swin-Unet-Transformer网络-用于语义分割-二分类

图像去噪-基于Swin-Transformer+UNet实现的图像去噪算法-效果佳-附项目源码-优质项目实战.zip

基于Swin-Unet自适应多尺度训练、多类别分割、迁移学习：心脏左心房冠状面分割【包含切片好的数据、代码、训练好的结果等等】

基于Swin-Transformer和Unet 自适应多尺度训练、多类别分割、迁移学习：医学图像子宫颈细胞核分割

Swing transformer Unet源代码，能直接运行

TransUnet-transformer 用于语义分割-二分类

基于Swin-Transformer和Unet 自适应多尺度训练、多类别分割：BraTS 3d脑肿瘤图像切分的2D图片分割项目

基于Swin-Transformer和Unet 分割项目、自适应多尺度训练、多类别分割、迁移学习：遥感道路二值分割项目

基于Swin-Transformer和Unet 项目、自适应多尺度训练、多类别分割：腹部多脏器5类别分割数据集

2021 自然语言处理峰会 共35份

YOLOv8-deepsort 实现智能车辆目标检测+车辆跟踪+车辆计数

YOLOv8网络结构图，自制visio文件，yolov8.vsds，需要的自取，在原有的基础上直接改就行了

yolov8(2023年8月版本),已经下好yolov8s.pt和yolov8n.pt

Transformer模型实现长期预测并可视化结果（附代码+数据集+原理介绍）

社交平台上经济类话题的文章热度信息，数据是真实的，但不是真实日期

行人跌倒数据集（VOC格式）

YOLOV5 + 双目相机实现三维测距（新版本）

Unet眼底血管图像分割数据集+代码+模型+系统界面+教学视频.zip

全新的SOTA模型YOLOv9

基于YOLOv8-Pose的姿态识别项目，带数据集可直接跑通的源码

YOLOV5口罩检测数据集+代码+模型 2000张标注好的数据+教学视频.zip

pycharm连接autodl服务器（yolov8训练自己的数据集）

Deep Learning Tuning Playbook（中译版）

zotero翻译插件.xpi

labelme v5.3.1 （2023年8月新版本，双击打开即用）

Informer模型实战案例(代码+数据集+参数讲解)

YOLOv5 人脸口罩图片数据集

YOLOv8改进，融合Gold-YOLO Neck

YOLOv8目标追踪实战全套资源包 - 源码与数据集完整分享

最新资源

2021 自然语言处理峰会共35份