2.1. Multilayer architectures 4
Multilayer architectures with learning bring about a different outlook on the
problem by proposing to learn, not only the classifier, but also learn the required
transformation operations directly from the data. This form of learning is commonly
referred to as representation learning [7,90], which when used in the context of deep
multilayer architectures is called deep learning.
Multilayer architectures can be defined as computational models that allow for
extracting useful information from the input data multiple levels of abstraction.
Generally, multilayer architectures are designed to amplify important aspects of the
input at higher layers, while becoming more and more robust to less significant
variations. Most multilayer architectures stack simple building block modules with
alternating linear and nonlinear functions. Over the years, a plethora of various mul-
tilayer architectures were proposed and this section will cover the most prominent
such architectures adopted for computer vision applications. In particular, artificial
neural network architectures will be the focus due to their prominence. For the sake
of succinctness, such networks will be referred to more simply as neural networks in
the following.
2.1.1 Neural networks
A typical neural network architecture is made of an input layer, x, an output layer,
y, and a stack of multiple hidden layers, h, where each layer consists of multiple
cells or units, as depicted in Figure 2.1. Usually, each hidden unit, h
j
, receives input
from all units at the previous layer and is defined as a weighted combination of the
inputs followed by a nonlinearity according to
h
j
= F (b
j
+
X
i
w
ij
x
i
) (2.1)
where, w
ij
, are the weights controlling the strength of the connections between the
input units and the hidden unit, b
j
is a small bias of the hidden unit and F (.) is
some saturating nonlinearity such as the sigmoid.
Deep neural networks can be seen as a modern day instantiation of Rosenblatt’s
perceptron [122] and multilayer perceptron [123]. Although, neural network models
