Machine Learning (Winter 2018) Assignment #3
Machine Learning: Assignment #3
Winter 2018
Due: Jan 10th, 23:59:59 CST (UTC +8).
1. Neural Networks
The codes of this section are in the neural_networks folder.
In this problem, we will implement the feedforward and backpropagation process of
the neural networks. We will use digital.mat as our experiment data. Finish fullycon-
nect_feedforward, fullyconnect_backprop, relu_feedforward, relu_backprop and the testing
part in run.m/run.ipynb. Then we can train three layer (data, hidden-relu, loss) neural
networks and report test accuracy.
Supplementary Knowledges:
i) Instead of using MSE loss function, we adopt softmax loss function here.
Recall that in Assignment #2, we used logistic regression to classify two
classes. And softmax regression model is a model that extends logistic re-
gression to classify more clases than two
1
. In this problem, we have 10
classes. The softmax loss part codes are done.
ii) We can use gradient_check.m/gradient_check.py to check the correctness
your computation. If
d
dθ
J (θ) = lim
ϵ→0
J (θ+ϵ)−J (θ)
ϵ
holds, then we are in the
right way.
2
iii) Weight decay and momentum are used to update weight paramters in
get_new_weight_inc.m/get_new_weight_inc.py.
2. K-Nearest Neighbor
The codes of this section are in the knn folder.
In this problem, we will play with K-Nearest Neighbor (KNN) algorithm and try it on
real-world data. Implement KNN algorithm (in knn.m/knn.py), then answer the following
questions.
(a) In knn_exp.m/knn_exp.ipynb, try KNN with dierent K (you should at least experi-
ment K = 1, 10 and 100) and plot the decision boundary.
You are encouraged to vectorize
34
your code, otherwise the experiment time might be
extremely long. You may nd the MATLAB build-in functions pdist2, sort, max and
hist useful. Also, you can use the function eudist2
5
written by Prof. Deng Cai
6
.
1
http://ufldl.stanford.edu/wiki/index.php/Softmax_Regression
2
http://ufldl.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization
3
http://www.mathworks.cn/cn/help/matlab/matlab_prog/vectorization.html
4
https://stackoverflow.com/questions/47755442/what-is-vectorization
5
http://www.cad.zju.edu.cn/home/dengcai/Data/code/EuDist2.m
6
Prof. Deng Cai is an expert on MATLAB, you can nd all his code at http://www.cad.zju.edu.cn/
home/dengcai/Data/data.html. You can learn how to write fast MATLAB code by reading his code.
1
