Deepplug-and-playpriorsforspectralsnapshotcompressiveimaging

66 浏览量 2021-01-25 21:49:04 上传评论收藏 4.13MB PDF 举报

资源推荐

资源详情

资源评论

Deep plug-and-play priors for spectral snapshot

compressive imaging

SIMING ZHENG,

1,2,†

YANG LIU,

3,†

ZIYI MENG,

MU QIAO,

ZHISHEN TONG,

6,7

XIAOYU YANG,

1,2

SHENSHENG HAN,

6,7

AND XIN YUAN

Computer Network Information Center, Chinese Academy of Sciences, Beijing 100190, China

University of Chinese Academy of Sciences, Beijing 100049, China

Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139, USA

Beijing University of Posts and Telecommunications, Beijing 100876, China

New Jersey Institute of Technology, Newark, New Jersey 07102, USA

Key Laboratory for Quantum Optics and Center for Cold Atom Physics of CAS, Shanghai Institute of Optics and Fine Mechanics,

Chinese Academy of Sciences, Shanghai 201800, China

Center of Materials Science and Optoelectronics Engineering, University of Chinese Academy of Sciences, Beijing 100049, China

Nokia Bell Labs, Murray Hill, New Jersey 07974, USA

*Corresponding author: xyuan@bell-labs.com

Received 5 October 2020; revised 20 November 2020; accepted 23 November 2020; posted 23 November 2020 (Doc. ID 411745);

published 21 January 2021

We propose a plug-and-play (PnP) method that uses deep-learning-based denoisers as regularization priors for

spectral snapshot compressive imaging (SCI). Our method is efficient in terms of reconstruction quality and speed

trade-off, and flexible enough to be ready to use for different compressive coding mechanisms. We demonstrate

the efficiency and flexibility in both simulations and five different spectral SCI systems and show that the pro-

posed deep PnP prior could achieve state-of-the-art results with a simple plug-in based on the optimization frame-

work. This paves the way for capturing and recovering multi- or hyperspectral information in one snapshot,

which might inspire intriguing application s in remote sensing, biomedical science, and material science. Our

code is available at: https://github.com/zsm12 11/PnP-CASSI.

https://doi.org/10.1364/PRJ.411745

1. INTRODUCTION

Real scenes are spectrally rich. Capturing the color, and thus the

spectral information, has been a central issue since the dawn of

photography. Correspondingly, many strategies have been con-

sidered. Since the advent of solid-state imaging, the color filter

array and especially the red–green–blue (RGB) bayer filter have

been the dominant strategy [1]. These filter arrays usually only

capture red, green, and blue bands and thus limit the spectral

resolution. When the number of sampled wavelengths becomes

large, bandpass filters, push-room, and other strategies may be

desirable. These systems usually have limited temporal resolu-

tion due to the inherent scanning procedure. Advances in

photonics and 2D materials give rise to compact solutions

to single-shot spectrometers at a high spectral resolution

[2–5]. More recently, it has been applied for spectral imaging

via combining stacking [6], optical parallelization [7], and com-

pressive sampling [8] strategies, where the trade-off between the

spatial pixel and spectral resolution still remains a challenge.

Thanks to compressive sensing (CS) [9 –11] and the advent

of decompressive inference algorithms over the past couple

of decades, there is substantial interest in hyperspectral color

filter arrays [12–14]. Such sampling strategies capture localized

coded image features and are well-matched to sparsity-based

inference algorithms [15–17]. With these advanced algorithms,

this technique has led to single-shot imaging for hyperspectral

images (HSIs), and we dub it snapshot compressive imaging

(SCI) [16,18]. In this paper, we focus on the s pectral SCI,

which aims to measure the x, y, λ data cube.

Spectral SCI is a hardware encoder plus software decoder

system, where the hardware encoder denotes the optical system,

which compresses the 3D x, y, λ data cube to a snapshot mea-

surement on the 2D detector, and the software decoder denotes

the reconstruction algorithms used to recover the 3D data cube

from the snapshot measurement.

The underlying principle of the spectral SCI hardware is to

modulate different bands (corresponding to different wave-

lengths) in the spectral data cube by different weights and then

integrate the light to the sensor. To perform the modulation,

which should be different for different spectral bands, various

techniques have been used. The pioneer work of coded aperture

snapshot spectral imaging (CASSI) [12] used a fixed mask (coded

aperture) and two dispersers to implement the band-wise

B18

Vol. 9, No. 2 / February 2021 / Photonics Research

Research Article

modulation, termed DD-CASSI; here DD means dual disperser.

Following this, the single-disperser (SD) CASSI was developed

[19], which achieves modulation by removing a disperser.

Following CASSI, various spectral SCI systems have been built

using disperser/prism and masks [20–24]. Recently, motivated

by the spectral variant responses of other media, spatial light mod-

ulators [25], ground-glass-based light field modulation [26], and

scatters [27] have also been employed for spectral SCI. In addi-

tion, some compact systems have also been built [28,29].

The software decoder, i.e., the reconstruction algorithm,

plays a pivotal role in spectral SCI as it outputs the desired data

cube. At the beginning, optimization-based algorithms devel-

oped for inverse problems such as CS were employed. Since

spectral SCI is an ill-posed problem, regularizers or priors

are generally used, such as the sparsity [30] and total variation

[15]. Later, the patch-based methods such as dictionary learn-

ing [25,31] and Gaussian mixture models [32] were developed

for the reconstruction of spectral SCI. Recently, by utilizing the

nonlocal similarity in the spectral data cube, group sparsity [17]

and low-rank models [16] have been developed to achieve state-

of-the-art results. The main bottleneck of these high perfor-

mance iterative optimization-based algorithms is the low

reconstruction speed. Since the spectral data cube is usually

large-scale, sometimes it needs hours to reconstruct a spectral

data cube from a snapshot measurement. This precludes the

real applications of spectral SCI systems.

To address the above speed issue in optimization algorithms,

and inspired by the performance of deep-learning approaches

for other inverse problems [33,34], convolutional neural net-

works (CNNs) have been used to solve the inverse problem of

spectral SCI for the sake of high speed [35–39]. These net-

works have led to better results than their optimization counter-

parts, given sufficient training data and time, which usually take

days or weeks. After training, the network can output the

reconstruction instantaneously and thus lead to end-to-end

spectral SCI sampling and reconstruction [39]. However, these

networks are usually system-specific. For example, different

numbers of spectral bands exist in different spectral SCI sys-

tems. Further, due to the different designs of masks, the trained

CNNs cannot be used in other systems, while retraining a new

network from scratch would take a long time.

Bearing the above concerns in mind, i.e., optimization-

based and deep-learning-based algorithms each have their

own pros and cons, it is desirable to develop a fast, flexible,

and high accuracy algorithm for spectral SCI. Fortunately,

the plug-and-play (PnP) framework [40,41] has been proposed

for inverse problems with provable convergence [42,43]. The

idea of PnP is intuitive, since the goal is to use the state-of-the-

art denoiser as a simple plug-in for recovery. The rationale here

is to employ recent advanced deep denoisers [44–

46] in the

iterative optimization algo rithm to speed up the reconstruction

process. Since these denoisers are pretrained with a wide range

of noise levels, the PnP algorithm is very efficient and usually

only tens or hundreds of iterations would provide promising

results [18]. More importantly, no training is required for dif-

ferent tasks and thus the same denoising network can be di-

rectly used in different systems. Therefore, PnP is a good

trade-off for reconstruction quality, speed, and flexibility.

However, since most existing flexible denoising networks are

designed for natural images, i.e., the gray-scale or RGB images,

directly using these networks into spectral SCI systems would

not lead to good results. To address this issue, in this paper, we

propose training a flexible denoising network for multispectral/

HSIs and then apply it to the PnP framework to solve the

reconstruction problem of spectral SCI.

Our proposed approach enjoys the advantages of speed, flex-

ibility, and high accuracy. We apply the proposed method in five

different real systems (three SD-CASSI systems [39,47,48], one

mutispectral endomicroscopy system [36], and one ghost imag-

ing spectral system [26]) and all of them have achieved promising

results. To compare with other state-of-the-art algorithms, sim-

ulations are also conducted to provide quantitative analysis.

Spectral sensor design and fabrication [2,4–8]maybenefitfrom

our method by taking inspiration from the coding mechanisms

and the simple plug-in for recovery.

Note that the PnP framework has been used in other inverse

problems such as video CS [18], which emphasized the theo-

retical analysis of PnP for SCI problems in general and used an

off-the-shelf denoiser (FFDNet) [46] to demonstrate its

capability in video SCI. No spectral SCI results have been

shown therein because spectral SCI is more challenging in

terms of its various coding mechanisms and no off-the-shelf

denoiser could provide a fast, flexible, and high-accuracy sol-

ution. As a matter of fact, this observation serves as the initial

motivation for this paper. Towards this end, the novelty of this

paper is twofold. First, we propose a CNN-based deep spectral

denoising network as the spatio-spectral prior, which is flexible

in terms of data size and the input noise levels. Second, we

summarize the image-plane and aperture-plane coding mech-

anisms for spectral SCI and use the PnP method combined

with our proposed deep spectral denoising prior for both

simulations and five different spectral SCI systems (including

image-plane and aperture-plane coding-based ones).

The paper is organized as follows. Section 2 introduces dif-

ferent spe ctral SCI systems. The proposed PnP method is de-

rived in Section 3. Extensive results are shown in Section 4, and

Section 5 concludes the entire paper.

2. SPECTRAL SCI

The basic idea of SCI is to encode 3D or multidimensional

visual information onto 2D sensor measurement. For spectral

SCI, a 3D spatio-spectral data cube is encoded to form a snap-

shot 2D measurement on the charge coupled device (CCD) or

complementary metal oxide semiconductor (CMOS) sensor, as

shown in Fig. 1.

A. SCI Forward Model

The forward model of SCI is linear. For spectral SCI, the spec-

tral data cube of the scene X ∈ R

W ×H×B

, where W , H , and B

denote the width, height, and the number of spectral bands,

respectively, is encoded onto a single 2D measurement

Y ∈ R

W ×H

(or similar size) via spectrally variant coding. By

vectorizing the scene’s spectral cube and measurement, that

is, x  vecX  ∈ R

WHB

and y  vecY  ∈ R

, we can

form a linear system for spectral SCI,

y  Ax  ε, (1)

Research Article

Vol. 9, No. 2 / February 2021 / Photonics Research B19

剩余11页未读，继续阅读

评论收藏

内容反馈

weixin_38732315

粉丝: 7
资源: 963

Deep plug-and-play priors for spectral snapshot compressive imag...

最新资源

Deep plug-and-play priors for spectral snapshot compressive imag...

Plug-and-Play-Priors

去噪代码matlab-Plug-and-Play-Priors:即插即用

[论文分享]Deep Plug-and-Play Super-Resolution for Arbitrary Blur Kernels.pptx

离散控制Matlab代码-ShearLet_for_Snapshot_Compressive_Imaging:利用Slicelet变换中的稀疏

去噪代码matlab-Dimensionality-Reduced-Plug-and-Play-Priors-for-Improving-Li

Face-Super-Resolution-Guided-by-3D-Facial-Priors:3D Facial Priors指导的存储库Face Super-Resolution的基本实现

Group-based image decomposition using 3-D cartoon and texture priors

Deep-Hough-Transform-Line-Priors:深度转换线优先级的正式实施（ECCV 2020）

3D-RCNN: Instance-level 3D Object Reconstruction via Render-and-Compare

matlab非参数代码-Non-parametric-priors-for-GAN:它包含在题为“GAN的非参数先验”的论文中使用的生成非参数

傅里叶反变换matlab代码-LRTV-MRFResnet-for-MRFingerprinting:用于磁共振指纹的医学成像工具箱

differentiable-particle-filters:论文“可区分的粒子过滤器”的源代码和数据-Source code learning

Representation Learning A Review and New Perspectives

On the Choice of Priors in Bayesian Deep Learning

Non-Convex Analysis and Synthesis Priors：非凸稀疏性促进分析和综合先验的算法-matlab开发

2009_NIPS_Fast Image Deconvolution using Hyper-Laplacian Priors(ori)

latent-gaussian-process-priors:论文“具有潜在高斯过程先验的Raven渐进矩阵完成”的代码

uncertainty in deep learning

Hough-based Tracking

Bayesian Compressive Sensing Using Laplace Priors . matlab code

Learning multi-denoising autoencoding priors for image super-resolution

Fast Image Deconvolution using Hyper-Laplacian Priors

细节增强的matlab代码-cd_Prototypical_Priors_BMVC15:BMVC15提交的代码-利用原型先验分类到零射学习

Bayesian Reinforcement Learning A Survey

CVPR 2023 RIDCP源码

STAmizctharmation.zip_By Heart_image processing_level set

Nonlocal_PBM.rar_Nonlocal-PBM

Structural Priors in Deep Neural Networks.pdf

(Translate)2009_NIPS_Fast Image Deconvolution using Hyper-Laplacian Priors.pdf

最新资源