Representation Learning: A Review and New Perspectives
需积分: 0 108 浏览量
2017-06-13
18:03:43
上传
评论
收藏 1.8MB PDF 举报
1
Representation Learning: A Review and New
Perspectives
Yoshua Bengio
†
, Aaron Courville, and Pascal Vincent
†
Department of computer science and operations research, U. Montreal
† also, Canadian Institute for Advanced Research (CIFAR)
F
Abstract—
The success of machine learning algorithms generally depends on
data representation, and we hypothesize that this is because different
representations can entangle and hide more or less the different ex-
planatory factors of variation behind the data. Although specific domain
knowledge can be used to help design representations, learning with
generic priors can also be used, and the quest for AI is motivating
the design of more powerful representation-learning algorithms imple-
menting such priors. This paper reviews recent work in the area of
unsupervised feature learning and deep learning, covering advances
in probabilistic models, auto-encoders, manifold learning, and deep
networks. This motivates longer-term unanswered questions about the
appropriate objectives for learning good representations, for computing
representations (i.e., inference), and the geometrical connections be-
tween representation learning, density estimation and manifold learning.
Index Terms—Deep learning, representation learning, feature learning,
unsupervised learning, Boltzmann Machine, autoencoder, neural nets
1 INTRODUCTION
The performance of machine learning methods is heavily
dependent on the choice of data representation (or features)
on which they are applied. For that reason, much of the actual
effort in deploying machine learning algorithms goes into the
design of preprocessing pipelines and data transformations that
result in a representation of the data that can support effective
machine learning. Such feature engineering is important but
labor-intensive and highlights the weakness of current learning
algorithms: their inability to extract and organize the discrimi-
native information from the data. Feature engineering is a way
to take advantage of human ingenuity and prior knowledge to
compensate for that weakness. In order to expand the scope
and ease of applicability of machine learning, it would be
highly desirable to make learning algorithms less dependent
on feature engineering, so that novel applications could be
constructed faster, and more importantly, to make progress
towards Artificial Intelligence (AI). An AI must fundamentally
understand the world around us, and we argue that this can
only be achieved if it can learn to identify and disentangle the
underlying explanatory factors hidden in the observed milieu
of low-level sensory data.
This paper is about representation learning, i.e., learning
representations of the data that make it easier to extract useful
information when building classifiers or other predictors. In
the case of probabilistic models, a good representation is often
one that captures the posterior distribution of the underlying
explanatory factors for the observed input. A good representa-
tion is also one that is useful as input to a supervised predictor.
Among the various ways of learning representations, this paper
focuses on deep learning methods: those that are formed by
the composition of multiple non-linear transformations, with
the goal of yielding more abstract – and ultimately more useful
– representations. Here we survey this rapidly developing area
with special emphasis on recent progress. We consider some
of the fundamental questions that have been driving research
in this area. Specifically, what makes one representation better
than another? Given an example, how should we compute its
representation, i.e. perform feature extraction? Also, what are
appropriate objectives for learning good representations?
2 WHY SHOULD WE CARE ABOUT LEARNING
REPRESENTATIONS?
Representation learning has become a field in itself in the
machine learning community, with regular workshops at the
leading conferences such as NIPS and ICML, and a new
conference dedicated to it, ICLR
1
, sometimes under the header
of Deep Learning or Feature Learning. Although depth is an
important part of the story, many other priors are interesting
and can be conveniently captured when the problem is cast as
one of learning a representation, as discussed in the next sec-
tion. The rapid increase in scientific activity on representation
learning has been accompanied and nourished by a remarkable
string of empirical successes both in academia and in industry.
Below, we briefly highlight some of these high points.
Speech Recognition and Signal Processing
Speech was one of the early applications of neural networks,
in particular convolutional (or time-delay) neural networks
2
.
The recent revival of interest in neural networks, deep learning,
and representation learning has had a strong impact in the
area of speech recognition, with breakthrough results (Dahl
et al., 2010; Deng et al., 2010; Seide et al., 2011a; Mohamed
et al., 2012; Dahl et al., 2012; Hinton et al., 2012) obtained
by several academics as well as researchers at industrial labs
bringing these algorithms to a larger scale and into products.
For example, Microsoft has released in 2012 a new version
of their MAVIS (Microsoft Audio Video Indexing Service)
1. International Conference on Learning Representations
2. See Bengio (1993) for a review of early work in this area.
arXiv:1206.5538v3 [cs.LG] 23 Apr 2014
剩余29页未读，继续阅读
资源评论
