单目视频的实时相干3D重建
103 浏览量
2024-04-12
16:32:45
上传
评论
收藏 7.74MB PDF 举报
NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video
Jiaming Sun
1,2∗
Yiming Xie
1∗
Linghao Chen
1
Xiaowei Zhou
1
Hujun Bao
1†
1
Zhejiang University
2
SenseTime Research
Abstract
We present a novel framework named NeuralRecon for
real-time 3D scene reconstruction from a monocular video.
Unlike previous methods that estimate single-view depth
maps separately on each key-frame and fuse them later, we
propose to directly reconstruct local surfaces represented
as sparse TSDF volumes for each video fragment sequen-
tially by a neural network. A learning-based TSDF fusion
module based on gated recurrent units is used to guide the
network to fuse features from previous fragments. This de-
sign allows the network to capture local smoothness prior
and global shape prior of 3D surfaces when sequentially
reconstructing the surfaces, resulting in accurate, coher-
ent, and real-time surface reconstruction. The experiments
on ScanNet and 7-Scenes datasets show that our system
outperforms state-of-the-art methods in terms of both ac-
curacy and speed. To the best of our knowledge, this is
the first learning-based system that is able to reconstruct
dense coherent 3D geometry in real-time. Code is avail-
able at the project page: https://zju3dv.github.io/
neuralrecon/.
1. Introduction
3D scene reconstruction is one of the central tasks in 3D
computer vision with many applications. In augmented re-
ality (AR) for example, to enable realistic and immersive
interactions between AR effects and the surrounding phys-
ical scene, 3D reconstruction needs to be accurate, coher-
ent and performed in real-time. While camera motion can
be tracked accurately with state-of-the-art visual-inertial
SLAM systems [3, 35, 1], real-time image-based dense re-
construction remains to be a challenging problem due to low
reconstruction quality and high computation demands.
Most image-based real-time 3D reconstruction pipelines
[38, 52] adopt the depth map fusion approach, which re-
semble RGB-D reconstruction methods like KinectFusion
∗
The first two authors contributed equally. The authors are affiliated
with the State Key Lab of CAD&CG and ZJU-SenseTime Joint Lab of 3D
Vision.
†
Corresponding author: Hujun Bao.
Depth-based (38.78 s)
1
2
3
Ours (5.68 s)
…
1 2 3
15
…
Reference View
Source View
Figure 1. Comparison between depth-based 3D reconstruction
methods and the proposed method. In depth-based methods,
key-frame depths are estimated separately from each key frame,
and later fused into a TSDF volume. In the proposed method, the
TSDF volume is directly predicted with all the key frames in a
local window. This design leads to a much more coherent recon-
struction and real-time speed.
[31]. Single-view depth maps from each key frame are first
estimated with real-time multi-view depth estimation meth-
ods like [48, 24, 13, 46]. The estimated depth maps are later
filtered with criteria like multi-view consistency and tempo-
ral smoothness, and fused into a Truncated Signed Distance
Function (TSDF) volume. The reconstructed mesh can be
extracted from the fused TSDF volume with the Marching
Cubes algorithm [27]. This depth-based pipeline has two
major drawbacks. First, since single-view depth maps are
estimated individually on each key frame, each depth esti-
mation is from scratch instead of conditioned on the pre-
vious estimations even the view-overlapping is substantial.
As a result, the scale-factor may vary even with the correct
camera ego-motion. Due to depth inconsistencies between
different views, the reconstruction result is prone to be ei-
ther layered or scattered. One example is shown in the red
boxes in Fig. 1, where the depth-based method struggles to
produce coherent depth estimations on the chairs and wall.
Second, since key-frame depth maps need to be estimated
separately in overlapped local windows, geometry of the
same 3D surface is estimated multiple times in different key
1
arXiv:2104.00681v1 [cs.CV] 1 Apr 2021
资源评论
