LIFT: Learned Invariant Feature Transform
Kwang Moo Yi
∗,1
, Eduard Trulls
∗,1
, Vincent Lepetit
2
, Pascal Fua
1
1
Computer Vision Laboratory, Ecole Polytechnique F´ed´erale de Lausanne (EPFL)
2
Institute for Computer Graphics and Vision, Graz University of Technology
{kwang.yi, eduard.trulls, pascal.fua}@epfl.ch, lepetit@icg.tugraz.at
Abstract. We introduce a novel Deep Network architecture that imple-
ments the full feature point handling pipeline, that is, detection, orienta-
tion estimation, and feature description. While previous works have suc-
cessfully tackled each one of these problems individually, we show how to
learn to do all three in a unified manner while preserving end-to-end dif-
ferentiability. We then demonstrate that our Deep pipeline outperforms
state-of-the-art methods on a number of benchmark datasets, without
the need of retraining.
Keywords: Local Features, Feature Descriptors, Deep Learning
1 Introduction
Local features play a key role in many Computer Vision applications. Find-
ing and matching them across images has been the subject of vast amounts
of research. Until recently, the best techniques relied on carefully hand-crafted
features [1–5]. Over the past few years, as in many areas of Computer Vision,
methods based in Machine Learning, and more specifically Deep Learning, have
started to outperform these traditional methods [6–10].
These new algorithms, however, address only a single step in the complete
processing chain, which includes detecting the features, computing their orienta-
tion, and extracting robust representations that allow us to match them across
images. In this paper we introduce a novel Deep architecture that performs all
three steps together. We demonstrate that it achieves better overall performance
than the state-of-the-art methods, in large part because it allows these individual
steps to be optimized to perform well in conjunction with each other.
Our architecture, which we refer to as LIFT for Learned Invariant Feature
Transform, is depicted by Fig. 1. It consists of three components that feed into
each other: the Detector, the Orientation Estimator, and the Descriptor. Each
one is based on Convolutional Neural Networks (CNNs), and patterned after
recent ones [6, 9, 10] that have been shown to perform these individual functions
well. To mesh them together we use Spatial Transformers [11] to rectify the
∗ First two authors contributed equally.
This work was supported in part by the EU FP7 project MAGELLAN under grant
number ICT-FP7-611526.
