Unsupervised Feature Learning and Deep Learning

UFLDL Tutorial

Description: This tutorial will teach you the main ideas of Unsupervised

Feature Learning and Deep Learning. By working through it, you will

also get to implement several feature learning/deep learning algorithms,

get to see them work for yourself, and learn how to apply/adapt these

ideas to new problems.

Supervised Learning and Optimization ......................................................................... 5

Linear Regression ................................................................................................... 5

Problem Formulation ....................................................................................... 5

Function Minimization..................................................................................... 6

Exercise 1A: Linear Regression ....................................................................... 7

Logistic Regression ................................................................................................. 9

Exercise 1B .................................................................................................... 10

Vectorization ......................................................................................................... 12

Example: Many matrix-vector products ........................................................ 12

Introduction .................................................................................................... 18

Cost Function ................................................................................................. 19

Properties of softmax regression parameterization ........................................ 20

Relationship to Logistic Regression .............................................................. 22

Exercise 1C .................................................................................................... 22

Debugging: Bias and Variance .............................................................................. 24

Debugging: Optimizers and objectives ................................................................. 24

Supervised Learning and Optimization ....................................................................... 25

Multi-Layer Neural Networks ............................................................................... 25

Neural Network model ................................................................................... 27

Locally Connected Networks ......................................................................... 36

Convolutions .................................................................................................. 37

Pooling .................................................................................................................. 38

Pooling: Overview ......................................................................................... 38

Pooling for Invariance.................................................................................... 39

Formal description ......................................................................................... 39

Exercise: Convolution and Pooling ...................................................................... 40

Convolution and Pooling ............................................................................... 40

Dependencies ................................................................................................. 40

Optimization: Stochastic Gradient Descent .......................................................... 43

Back Propagation ........................................................................................... 47

Exercise: Convolutional Neural Network ............................................................. 48

Overview ........................................................................................................ 48

Dependencies ................................................................................................. 48

Implement Convolutional Neural Network ................................................... 49

Unsupervised Learning ................................................................................................ 53

Autoencoders ........................................................................................................ 53

Visualizing a Trained Autoencoder ................................................................ 57

PCA Whitening ..................................................................................................... 59

Introduction .................................................................................................... 59

References ...................................................................................................... 68

Whitening .............................................................................................................. 68

Introduction .................................................................................................... 68

2D example .................................................................................................... 68

ZCA Whitening ..................................................................................................... 70

Regularizaton ........................................................................................................ 71

Implementing PCA Whitening .............................................................................. 72

Exercise: PCA Whitening ..................................................................................... 74

PCA and Whitening on natural images .......................................................... 74

ICA ........................................................................................................................ 78

Exercise: RICA ..................................................................................................... 81

Step 0: Prerequisites ....................................................................................... 81

Step 1: RICA cost and gradient ...................................................................... 82

Self-Taught Learning ................................................................................................... 84

Self-Taught Learning ............................................................................................ 84

Overview ........................................................................................................ 84

Step 1: Generate the input and test data sets .................................................. 89

Step 2: Train RICA ........................................................................................ 89

Step 3: Extracting features ............................................................................. 89

Step 4: Training and testing the softmax regression model ........................... 90

Step 5: Classifying on the test set .................................................................. 90

Building Deep Networks for Classification ................................................................. 91

From Self-Taught Learning to Deep Networks .................................................... 91

Deep Networks: Overview .................................................................................... 93

Overview ........................................................................................................ 93

Advantages of deep networks ........................................................................ 93

Better local optima ......................................................................................... 96

Stacked Autoencoders ........................................................................................... 97

Overview ........................................................................................................ 97

Training .......................................................................................................... 97

Concrete example........................................................................................... 98

Discussion ...................................................................................................... 98

Fine-tuning Stacked AEs ....................................................................................... 99

Introduction .................................................................................................... 99

General Strategy ............................................................................................. 99

Finetuning with Backpropagation .................................................................. 99

input values . For example, we might want to make predictions about the

price of a house so that y represents the price of the house in dollars and the

elements x

of x represent "features" that describe the house (such as its size and the

number of bedrooms). Suppose that we are given many examples of houses where the

features for the i'th house are denoted x

the

Our goal is to find a function y = h(x) so that we have for each

training example. If we succeed in finding a function h(x) like this, and we have seen

Here, h

(We call this space of functions a "hypothesis class".) With this representation for h,

our task is to find a choice of θ so that h

) is as close as possible to y

. In

particular, we will search for a choice ofθ that minimizes: