1
Superpixel Soup: Monocular Dense 3D
Reconstruction of a Complex Dynamic Scene
Suryansh Kumar, Member, IEEE, Yuchao Dai, Member, IEEE, Hongdong Li, Senior Member, IEEE
Abstract—This work addresses the task of dense 3D reconstruction of a complex dynamic scene from images. The prevailing idea to
solve this task is composed of a sequence of steps and is dependent on the success of several pipelines in its execution [1]. To overcome
such limitations with the existing algorithm, we propose a unified approach to solve this problem. We assume that a dynamic scene can
be approximated by numerous piecewise planar surfaces, where each planar surface enjoys its own rigid motion, and the global change
in the scene between two frames is as-rigid-as-possible (ARAP). Consequently, our model of a dynamic scene reduces to a soup of
planar structures and rigid motion of these local planar structures. Using planar over-segmentation of the scene, we reduce this task to
solving a “3D jigsaw puzzle” problem. Hence, the task boils down to correctly assemble each rigid piece to construct a 3D shape that
complies with the geometry of the scene under the ARAP assumption. Further, we show that our approach provides an effective solution
to the inherent scale-ambiguity in structure-from-motion under perspective projection. We provide extensive experimental results and
evaluation on several benchmark datasets. Quantitative comparison with competing approaches shows state-of-the-art performance.
Index Terms—Dense 3D reconstruction, perspective camera, as-rigid-as-possible, relative scale ambiguity, structure from motion.
F
1 INTRODUCTION
T
HE task of reconstructing 3D geometry of the scene from
images —popularly known as structure-from-motion
(SfM), is a fundamental problem in computer vision. An
initial introduction and working solution to this problem
can be found as early as 1970’s and 1980’s [2] [3] [4], which
Blake et al. discussed comprehensively in their seminal
work [5]. While this field of study in the past was largely
dominated by sparse feature based reconstruction of a rigid
scene [6] [7] [8] [9] [10] and a non-rigid object [11] [12] [13]
[14] [15], in recent years, with the surge in computational
resources, dense 3D reconstruction of the scene have been
introduced and successfully demonstrated [16] [17] [1].
A dense solution to this inverse problem is essential due
to its increasing demand in many real-world applications
–from animation and entertainment industry to robotics
industry (VSLAM). In particular, with the proliferation of
monocular camera in almost all modern mobile devices has
elevated the demand for sophisticated dense reconstruc-
tion algorithm. When the scene is static and the camera
is moving, 3D reconstruction of such scenes from images
can be achieved by using conventional rigid structure from
motion techniques [8] [18] [19] [20]. In contrast, to model
arbitrary dynamic scene can be very challenging. When the
camera is moving and the scene is static under such settings,
the elegant geometrical constraint can help explain the
camera’s [7] [21], which are later used to realize the dense
3D reconstruction of the scene [19] [20] [17] [22]. However,
such geometrical constraint may fail when multiple rigidly
moving objects are observed by a moving camera. Although
• Suryansh Kumar is with ETH Z¨urich and Australian National Univer-
sity. E-mail: sukumar@vision.ee.ethz.ch, suryansh.kumar@anu.edu.au.
• Yuchao Dai is with Northwestern Polytechnical University. E-mail:
daiyuchao@gmail.com.
• Hongdong Li is with Australian National University and ARC Centre of
Excellent for Robotic Vision. E-mail: hongdong.li@anu.edu.au.
Dense 3D
reconstruction
Input Images
You-Tube Object Dataset(Messi) MPI Sintel (alley_1)
Fig. 1: Dense 3D reconstruction of a complex dynamic scene, where both the
camera and the objects are moving with respect to each other. The top left shows
a sample reconstruction on messi sequence from Youtube Object dataset [23].
The top right shows the reconstruction on alley 1 sequence from the MPI Sintel
dataset [24].
each of the individual rigid objects can be reconstructed
up to an arbitrary scale (assuming motion segmentation is
provided), the reconstruction of the whole dynamic scene
is generally impossible, simply because the relative scales
among all the moving shapes cannot be determined in a
globally consistent way. Furthermore, since all the estimated
motions are relative to each other, one cannot distinguish
camera motion from the object motion. Therefore, prior
information about the objects, or the scene, and their relation
to the frame of reference are used to fix the placement of
these objects relative to each other.
Hence, from the above discussion, it can be argued that
the solution to 3D reconstruction of a general dynamic
scene is non-trivial. Nevertheless, it is an important problem
to solve as many real-world applications need a reliable
solution to this problem. For example, understanding of a
traffic scene, a typical outdoor traffic scene consists of both
multiple rigid motions of vehicles, and non-rigid motion of
the pedestrians. To model such scenarios, it is important to
have an algorithm that can provide dense 3D information
from images.
arXiv:1911.09092v1 [cs.CV] 19 Nov 2019
剩余11页未读，继续阅读
资源评论
