Machine Learning Programming Exercise 4

Due 30th November, 2015. Please upload to LMS before midnight.

1 Neural-Networks [100 points]

You will implement forward and back propagation algorithm using stochastic gradient descent method
on CIFAR-10 data (you can reach the details of the data from the site:
http://www.cs.toronto.edu/~kriz/cifar.html). Use data_batch_1 as training set and test_batch as test set.
With this dataset, you will have 32x32 color images in 10 classes. However, in this exercise, you will
try to recognize whether a picture is a car or truck and then whether the picture is a car or a bird. In this
data set, labels are from 0 to 9. Consider a 3 layer neural network with one input, one hidden, and one
output layer. Since you have 3072 fetures, input layer should have 3072 units. Make hidden layer have
25 units.
Note: After loading the data, dont forget to convert the type of it to double.
i.

ii.
iii.

iv.
v.

vi.
vii.

[10 points] Implement the feedforward part of the neural network that returns the cost

only. Write the cost function that computes the cost of the neural network. Using
first 1000 training samples and all labels (where you change label 0 with label 10),
load initialThetas.mat to verify your function. With this data you should get a cost
of 6.952386.
[10 points] Now add regularization to your cost function. Again test your function
using initialThetas.mat when lambda=1. Then, you should get 7.136791 cost.
[15 points] Implement the backpropagation algorithm to compute the gradients. Add to your
cost function code to return the partial derivatives of the parameters. For this, you should
also write a function that calculates sigmoid gradient.
[20 points] Write stochastic gradient descent function that calculates optimal thetas.
[20 points] Initialize your thetas randomly. Find optimal thetas using your stochastic
gradient descent function when learning rate=0.3, regularization parameter=0.1, number of
epoch=30 and mini batch size=10. Do this for both distinguishing whether a picture is a car
or truck and then whether the picture is a car or a bird. Dont forget to discard all the remaing
training and test samples.
[15 points] Predict the labels of the training set using the thetas you get. Then, by comparing
your predicted labels and the real labels, calculate the training set accuracy.
[10 points] Do the same thing for test_batch data, calculate the test set accuracy.