使用编码掩码的快照HDR视频构造_Snapshot HDR Video Construction Using Coded Mas

SNAPSHOT HDR VIDEO CONSTRUCTION USING CODED MASK

A PREPRINT

Masheal M. Alghamdi

National Center for Data Analytics and Artificial Intelligence

KACST

Qiang Fu

Visual Computing Center

KAUST

Wolfgang Heidrich

Visual Computing Center

video. Constructing an HDR video requires restoring the HDR values for each frame and maintaining

the consistency between successive frames. HDR image acquisition from single image capture, also

known as snapshot HDR imaging, can be achieved in several ways. For example, the reconfigurable

snapshot HDR camera is realized by introducing an optical element into the optical stack of the

camera; by placing a coded mask at a small standoff distance in front of the sensor. High-quality HDR

image can be recovered from the captured coded image using deep learning methods. This study

utilizes 3D-CNNs to perform a joint demosaicking, denoising, and HDR video reconstruction from

coded LDR video. We enforce more temporally consistent HDR video reconstruction by introducing

a temporal loss function that considers the short-term and long-term consistency. The obtained results

are promising and could lead to affordable HDR video capture using conventional cameras.

bit depths that modern camera sensors can offer.

Modern films are often shot using cameras with a higher dynamic range, which mainly require both HDR shooting and

also required in all applications that require high accuracy in capturing temporal aspects of the changes in a scene. HDR

video capture is essential for specific industrial tracking processes, such as melting, machine vision such as autonomous

driving, and monitoring systems.

In Alghamdi et al. [2019, 2021] we proposed a computational imaging solution to single shot HDR imaging by minimal

modifications of the camera to implement per-pixel exposure, as well as a deep learning algorithm based on the inception

network to reconstruct HDR images. Specifically we explored a variant of the spatially modulated HDR camera design

Imaging lens

Binary pattern closer to the imaging sensor

Figure 1: Illustration of the effect of the binary pattern distance from the sensor plane on the resulting optical mask.

When the binary pattern is exactly on the sensor plane, we obtain the exact binary pattern mask. As the binary pattern is

moved away from the sensor by various distances

D

, the mask becomes blurred versions of the binary pattern. The

blurriness depends on the distance.

into an HDR mode with the introduction of an optical element into the optical stack of the camera. To realize these

desired properties, we propose a mask that is not attached directly to the surface of the image sensor as in the case

primary focus was capturing and reconstructing a single HDR image from a single coded LDR image, and we worked

mainly on the spatial domain of the image.

Our main focus in this paper is to work with coded LDR videos, where we expect to benefit greatly from working

jointly on both the spatial and temporal domains of video images. In this article, we will concentrate on constructing

HDR video from coded LDR images obtained using the reconfigurable snapshot HDR camera. Reconstructing HDR

video from coded LDR video requires restoration of the HDR values for each frame and maintenance of the consistency

between successive frames. Spatio-temporal coherence between, and among, the frames must be exploited appropriately

for accurate prediction of HDR values. Although 2D-CNNs are powerful for modeling images, 3D-CNNs are more

appropriate for spatio-temporal feature extraction as they can maintain the temporal information. For this reason, the

reconstruction would fail if we directly used our proposed networks in Alghamdi et al. [2019, 2021] to video frames

with standard sensors. Some HDR cameras have been presented to the scientific society, but are still not accessible for

few alternative commercial HDR sensors, such as the Red Epic camera Red Company, Thomson Viper Grass Valley,

Arri Alexa Arri Alexa, Sony PXW-Z90 Sony, and Phantom HD Vision Research, though these devices still have a

HDR image acquisition Reinhard et al. [2010]. These methods can be categorized into three distinct approaches. The

most common way is to capture a sequence of low dynamic range (LDR) images with different exposures and fuse

them into an HDR image Debevec and Malik [1997], Mann et al. [1995]. Modern cameras and mobile devices can

easily afford successive image capture, making this method capable of producing decent HDR images for static scenes.

However, when either the scene is dynamic or the camera shakes during capture, the resulting images can suffer from

ghosting artifacts. The second approach is to utilize multiple sensors to simultaneously capture differently exposed LDR

images by, for example, splitting the light to multiple sensors with a beam-splitter McGuire et al. [2007], Tocci et al.

[2011], Kronander et al. [2013]. This sophisticated approach is expensive and needs additional rigorous calibration.

The third approach is to capture a single LDR image with a per-pixel or per-scanline coded exposure. Reconstruction

algorithms are applied later to create HDR images Nayar and Mitsunaga [2000], Nayar and Branzoi [2003], Serrano

we introduced an algorithm built upon an inception network that decodes reliable HDR images from the raw noisy

coded Bayer data. We demonstrate both in simulation and using a prototype that the combination of hardware encoding

and software decoding leads to a simple, yet efficient, HDR image acquisition system.

In Alghamdi et al. [2021] we present a transfer learning framework for solving the HDR reconstruction part. Our

training deep neural networks. In Alghamdi et al. [2019] we solved this issue by pre-training on a large simulated HDR

dataset. This pre-training is expensive in both in memory and time; experimenting with different network structures will

need weeks of pre-training. In tIn Alghamdi et al. [2021] we incorporate architectures pre-trained on a different large

scale task, and transfer them to our HDR reconstruction. This new approach reduces our processing time substantially.

classification Ji et al. [2012], Tran et al. [2015]. The spatio-temporal feature extraction capability of 3D-CNNs was

adequate video descriptor, and a homogeneous architecture with small 3

×

the best-performing architecture for 3D-CNNs. Moreover, the capabilities of 3 D-CNN in video enhancement, inpainting

and super-resolution have been proven Lv et al. [2018], Kappeler et al. [2016], Wang et al. [2017], Wan [2019]. This

LDR video. As far as the author knows, there is no published work on the construction of HDR video from coded LDR

images that utilizes temporal information in the reconstruction process.

3 Methods

3.1 Imaging Model