SGM-Nets: Semi-global matching with neural networks
需积分: 0 5 浏览量
2022-12-07
10:42:05
上传
评论
收藏 4.86MB PDF 举报
SGM-Nets: Semi-global matching with neural networks
Akihito Seki
1∗
Marc Pollefeys
2,3
1
Toshiba Corporation
2
ETH Z
¨
urich
3
Microsoft
akihito.seki@toshiba.co.jp, marc.pollefeys@inf.ethz.ch
Abstract
This paper deals with deep neural networks for predict-
ing accurate dense disparity map with Semi-global match-
ing (SGM). SGM is a widely used regularization method for
real scenes because of its high accuracy and fast compu-
tation speed. Even though SGM can obtain accurate re-
sults, tuning of SGM’s penalty-parameters, which control a
smoothness and discontinuity of a disparity map, is uneasy
and empirical methods have been proposed. We propose a
learning based penalties estimation method, which we call
SGM-Nets that consist of Convolutional Neural Networks.
A small image patch and its position are input into SGM-
Nets to predict the penalties for the 3D object structures.
In order to train the networks, we introduce a novel loss
function which is able to use sparsely annotated disparity
maps such as captured by a LiDAR sensor in real environ-
ments. Moreover, we propose a novel SGM parameteriza-
tion, which deploys different penalties depending on either
positive or negative disparity changes in order to represent
the object structures more discriminatively.
Our SGM-Nets outperformed state of the art accuracy
on KITTI benchmark datasets.
1. Introduction
Stereo disparity estimation is one of the most important
problems in computer vision. The disparity map is widely
used, for example in object detection [
13], surveillance [29],
autonomous driving for cars [
27], and unmanned air vehi-
cles [
24].
Many disparity estimation methods have been proposed
for many years [
32]. A standard pipeline for the dense dis-
parity estimation starts by finding local correspondences
between stereo images. Incorrect correspondences occur
due to various reasons such as occlusion and pixel intensity
noise. In order to refine the disparity map, regularization
methods [
15, 31, 33, 35] and some filters [36, 40, 38] are
applied, and the fine dense disparity is finally obtained. In
the KITTI website [1], many state of the art researches focus
on accurate local correspondence methods with deep learn-
∗
This work has been done while the first author was visiting at ETH
Z
¨
urich.
(a) (b)
(c) (d)
Figure 1. (a) Left image. (b) Ground truth disparity map. Occlu-
sion in black. Disparity map by using SGM with (c) Hand tuned
penalties and (d) SGM-Net. The difference of inputs is only SGM
penalties.
ing [
38, 21, 3] and apply Semi-global matching (SGM) [15]
as regularization. Recently, deep learning methods such as
FlowNet [
6] and DispNet [22] which play end-to-end of
the pipeline have been proposed. However, the methods
haven’t achieved sufficient accuracy compared to the stan-
dard pipeline so far. We guess one of the reason for the
lower accuracy comes from the differences between train-
ing and testing datasets as mentioned in [
9, 26].
In this paper, we focus on the regularization part of the
standard pipeline since many sophisticated local correspon-
dence methods have been proposed. SGM is a widely used
regularization method due to its high accuracy while keep-
ing low computation cost. Some papers have reported its
real time computation even on mobile devices [
16, 14].
SGM has penalty-parameters, we call them “penalties” in
this paper, and they control the smoothness and discontinu-
ity of the disparity map. So far, the penalties are designed
empirically and are uneasy to be tuned.
We consider the penalties should be different depending
on 3D object structures. For instance, the penalties should
capture the fact that road is smooth. We propose a learn-
ing based penalties prediction method which uses CNNs.
CNNs provide high performances from primitive level pro-
cessing such as stereo correspondences to high level ones
such as scene classification [
2, 20] and object detection
[
11, 39]. Deep learning using a CNN offers a promising
way for our purpose. However, it isn’t straightforward to
involve the CNN for the task, i.e. How to train and con-
struct the CNNs for SGM?
The contributions of this paper are the following: (1) A
231
资源评论
