HELLENIC MEDITERRANEAN UNIVERSITY

M.SC. INFORMATICS ENGINEERING

Image Classification using

Backpropagation Algorithm

Fashion Products by MNIST Dataset

PROJECT 2:

NEURAL NETWORKS

Course:
Computational Intelligence

Evangelia Anastasaki
Mtp 231
 Image Classification using Backpropagation Algorithm – Fashion Products by MNIST
Dataset | Evangelia Anastasaki

Contents

Abstract ....................................................................................................................... 3

Introduction ................................................................................................................. 4

Methodology ............................................................................................................... 5

Fashion-MNIST Dataset ......................................................................................... 5

Machine Learning Set-Up ....................................................................................... 6

Hardware ............................................................................................................. 6

Libraries ............................................................................................................... 7

Python & Spyder by Anaconda Distribution ....................................................... 8

Methods and Code ............................................................................................... 8

Results ....................................................................................................................... 11

Conclusion ................................................................................................................ 16

References ................................................................................................................. 17

2
 Image Classification using Backpropagation Algorithm – Fashion Products by MNIST
Dataset | Evangelia Anastasaki

Abstract

In this paper, I utilize an Artificial Neural Network (ANN) - Backpropagation

Algorithm to solve image classification with Fashion-MNIST dataset. Firstly, image
classification mentions the task of entering an image into a neural network and having
it output a few kinds of the label for that image. Moreover, there are two sorts of
classification: directed and unsupervised. Furthermore, Fashion-MNIST Dataset is a
28 x 28 grayscale image, related to a label from 10 classes. Finally, I purpose to
achieve the best accuracy from the training of the model.

Keywords: Artificial Neural Network (ANN), Backpropagation Algorithm, Fashion-

MNIST Dataset.

3
 Image Classification using Backpropagation Algorithm – Fashion Products by MNIST
Dataset | Evangelia Anastasaki

Introduction
In this paper, I utilize an Artificial Neural Network (ANN) - Backpropagation
Algorithm to solve image classification using Fashion-MNIST dataset on purpose the
best accuracy from the training of the model. Image classification refers to the task
of entering an image into a neural network and having it output some kind of label
for that image. The label that the network outputs will correspond to a pre-defined
class. There can be multiple classes that the image can be labelled as, or just one. A
subsection of image classification is object detection, where particular occurrences of
objects are recognized as belonging to a certain class like clothes. There are two sorts
of classification: supervised and unsupervised. Supervised learning is based on
training a data sample from the information establishment with the accurate
classification as of now apportioned. Such techniques are utilized in feedforward or
MultiLayer Perceptron (MLP) models. Learning through training in a supervised
Artificial Neural Network (ANN) model too called an error backpropagation
algorithm. The error correction-learning calculation trains the network based on the
input-output samples and finds error signal, which is the difference of the output
calculated and the specified output and adjusts the synaptic weights of the neurons
that are relative to the product of the error signal and the input occurrence of the
synaptic weight Self-Organizing neural networks learn to utilize an unsupervised
learning algorithm to distinguish hidden patterns in the unlabeled input information.
This unsupervised mentions the ability to memorize and organize data without
providing an error signal to evaluate the potential arrangement. The lack of direction
for the learning algorithm in unsupervised learning can in some cases be beneficial
since it lets the algorithm to look back for patterns that have not been previously
considered. The computational layer is additionally called a competitive layer since
the neurons within the layer compete with each other to end up dynamic. Thus, this
learning algorithm is called a competitive algorithm.

4
 Image Classification using Backpropagation Algorithm – Fashion Products by MNIST
Dataset | Evangelia Anastasaki

Figure 1: Learning Rules of ANN.

Methodology
Fashion-MNIST Dataset

Image Data that I utilized is Fashion-MNIST Dataset. This is a dataset of Zalando's

article images, which contain a training set of 60,000 examples and a test set of 10,000
examples. Each example is a 28 x 28 grayscale image, related to a label from 10
classes. Zalando aims Fashion-MNIST to serve as a straight drop-in replacement for
the first MNIST dataset for benchmarking machine learning algorithms. It offers the
same image size and structure of the training and testing parts. The original MNIST
dataset contains a part of handwritten digits. Members of the Artificial Intelligence /
Machine Learning / Data Science community choice this dataset and utilize it as a
benchmark to approve their algorithms. In fact, MNIST is frequently the first dataset
researchers attempt. Furthermore, each image is 28 pixels in height and 28 pixels in
width, for a total of 784 pixels in add up to. Each pixel incorporates a single pixel-
value related with it, demonstrating the lightness or darkness of that pixel, with higher
numbers meaning darker. This pixel-value is an integer between 0 and 255. The
training and test information sets have 785 columns. The primary column comprises

5
 Image Classification using Backpropagation Algorithm – Fashion Products by MNIST
Dataset | Evangelia Anastasaki

of the class labels and represents the article of clothing. The rest of the columns
contain the pixel-values of the related image.

Figure 2: Classes and Examples in Fashion-MNIST Dataset.

Machine Learning Set-Up

Hardware

First of all, to perform machine learning the software required a computer system
powerful enough to handle the computing power necessary. Therefore, I have used:
➢ Central Processing Unit (CPU): Intel Core i7 8th Generation processor.
➢ Random Access Memory (RAM): 16 GB.
➢ Graphics Processing Unit (GPU): NVIDIA GeForce GT 640M LE.

6
 Image Classification using Backpropagation Algorithm – Fashion Products by MNIST
Dataset | Evangelia Anastasaki

Libraries

Firstly, for the implementation of the software, I have used specifics machine learning
libraries:
➢ T-SNE (T-distributed Stochastic Neighbor Embedding) by scikit-learn is
a tool to visualize high-dimensional information. It converts similarities
between information points to joint probabilities and tries to minimize the
Kullback-Leibler dissimilarity between the joint probabilities of the low-
dimensional inserting and the high-dimensional information. Τ-SNE contains
a cost function that is not convex, i.e. with diverse initializations we will get
diverse results.
➢ NumPy is the basic package for scientific computing with Python. It contains
among other things: an effective N-dimensional cluster object, advanced
(broadcasting) functions, tools for joining C/C++ and Fortran code, useful
linear algebra, Fourier change, and irregular number capabilities.
➢ Matplotlib is a Python 2D plotting library which produces publication quality
figures in an assortment of hardcopy designs and interactive environments.
Matplotlib can be utilized in Python scripts, the Python and IPython shells (a
wealthy interactive interface, letting rapidly process information and test
thoughts.), the Jupyter notebook, web application servers, and four graphical
user interface toolkits. Moreover, it produces plots, histograms, control
spectra, bar charts, error charts, scatterplots, etc., with just a few lines of code.
Besides, it gives a MATLAB-like interface, especially when combined with
IPython.
➢ SciPy is one of the fundamental packages that make up the SciPy stack. It
gives numerous user-friendly and effective numerical routines, such as
routines for numerical integration, interpolation, optimization, linear algebra,
and statistics.

7
 Image Classification using Backpropagation Algorithm – Fashion Products by MNIST
Dataset | Evangelia Anastasaki

Python & Spyder by Anaconda Distribution

The programming language that I utilized is Python. Python is an interpreted, high-

level, general-purpose programming language. Its language builds and object-
oriented approach point to assist software engineers to compose clear, consistent code
for little and large-scale projects. Moreover, it is powerfully and provides numerous
programming standards, counting procedural, object-oriented and functional
programming. The program that I utilized is Spyder by Anaconda Distribution, which
is open-source. Besides, it is the most straightforward way to perform Python data
science and machine learning. Also, it is the industry standard for developing, testing,
and training on a single machine, manages libraries, dependencies, and environments
with Conda, develops and trains machine learning and deep learning models, analyzes
data with scalability and visualizes results.

Methods and Code

Firstly, I utilized the Backpropagation algorithm. Backpropagation comprises

3 subtasks: Make a forward pass, Calculate the error and make a backward
pass (backpropagation). Also, I utilized 1 hidden layer. Particularly, for
building Neural Network (NN):
➢ I utilized the sigmoid function. The sigmoid function is very
commonly utilized enactment function, at least it was until recently.
It encompasses a particular S shape, it is a differentiable real function
for any real input value and output values between 0 and 1. Also, it
incorporates a positive derivative at each point. More importantly, I
utilized it as an enactment function for the hidden layer of the model.
The definition of the sigmoid is:

8
 Image Classification using Backpropagation Algorithm – Fashion Products by MNIST
Dataset | Evangelia Anastasaki

Its first derivative, which I utilized during the backpropagation step of the training
algorithm, has the following equation:

Figure 3: Visualization of sigmoid and prime-sigmoid function

➢ Also, I utilized the Softmax function. The Softmax function can be effectively
separated, output depends only on input and the elements of the resulting
vector sum to 1. The definition is:

➢ Initializing the weights, I sampled a uniform distribution with values

between -1 and 1 for the initial weights.

9
 Image Classification using Backpropagation Algorithm – Fashion Products by MNIST
Dataset | Evangelia Anastasaki

➢ Training: for each epoch, I applied the backpropagation algorithm, evaluate

the error and the gradient with regard to the weights. At that point, I utilized
the learning rate and gradients to upgrade the weights. Moreover, I did an
extra step before returning the gradients and I connected L1 and L2
Regularization. Regularization is utilized to direct the training towards less
complex methods by penalizing expansive values for the parameters W.

➢ Measuring the error: I utilized the Cross-Entropy loss (known as log loss)
work to evaluate the error. This function measures the execution of a
classification model whose output is a likelihood. It penalizes (cruelly)
predictions that are wrong and certain. Here is the definition:

where C is the number of classes, y is a binary indicator in case a class label

is the proper classification for the observation and P is the predicted
probability that o is of class c. After computing the Cross-Entropy loss, I
include the regularization terms and calculate the mean error.

➢ Making predictions: the model can learn from data and form predictions on
data it has not seen before. I am aiming to implement two strategies for
prediction - predict and predict_proba. By and large talking, the predictions in
NN includes applying a forward step on the data. But the result of it is a vector
of values representing how solid the belief for each class is for the data and I
utilize Maximum likelihood estimation (MLE) to get the ultimate predictions.
MLE is a strategy of assessing the parameters of likelihood distribution by
maximizing a probability function so that beneath the expected statistical
model the observed information is most probable. The point within the
parameter space that maximizes the probability function is called the maximum
likelihood estimate.

10
 Image Classification using Backpropagation Algorithm – Fashion Products by MNIST
Dataset | Evangelia Anastasaki

➢ Evaluation: the NN model to the test

Furthermore, I created a Neural Network (NN) that can properly predict values from
the XOR function.

Figure 4: The truth table of XOR Function

Then, I defined 500 epochs parameters:

The epochs parameter controls how numerous times the algorithm will “see” the
information during training. At that point, I set the number of neurons within the
input, hidden and output layers - I have 2 numbers as input and 1 number as the output
size. The learning rate parameter controls how rapidly the NN will learn from new
information and forget what already knows.

Results

For the exploration we have defined 10 classes of possible fashion products:

11
 Image Classification using Backpropagation Algorithm – Fashion Products by MNIST
Dataset | Evangelia Anastasaki

As it is reported above that the product images are grayscale, 28x28 pixels, like
figure 4 shows:

Figure 4: Product Image grayscale 28 x 28 from Fashion-MNIST Dataset

Then, the program appears a plot which is a lower dimensional representation of some
of the products using a T-SNE. It transforms the data into 2-dimensional using the
implementation from scikit-learn:

12
 Image Classification using Backpropagation Algorithm – Fashion Products by MNIST
Dataset | Evangelia Anastasaki

Figure 5: 2-dimensional representation with fashion products

The plot appears a clear separation between a few classes and significant overlap
between others. Therefore, I have built a Neural Network that can attempt to isolated
between diverse fashion products. Then, from the training, it is observed that the sum
error seems to be decreasing. But as the plot appears (figure 7), the model cannot
proceed to decrease the error from the 60 epochs and so on.

Figure 6: Log loss

13
 Image Classification using Backpropagation Algorithm – Fashion Products by MNIST
Dataset | Evangelia Anastasaki

Figure 7: Visualization of log loss

Then, the program appears a single prediction (figure 8), which seems correct. After
it is evaluated a few more predictions (figure 9), the most are wrong.

Figure 8: Single Prediction

14
 Image Classification using Backpropagation Algorithm – Fashion Products by MNIST
Dataset | Evangelia Anastasaki

Figure 9: More Predictions

Figure 10: Accuracy of Training and Test

Well, those do not look that good. While a random classifier will return ~20%
accuracy, ~61% accuracy on the test dataset will not make a practical classifier either.
Consequently, I improved the evaluation utilizing scikit-learn`s scale to normalize
the data and the final results seem much better.

15
 Image Classification using Backpropagation Algorithm – Fashion Products by MNIST
Dataset | Evangelia Anastasaki

Figure 11: Improvement of Predictions

Figure 12: Improvement of Training and Test Accuracy

Finally, the accuracy ~86% (vs ~61%) on the training set is a vast improvement over
the unscaled method.

Conclusion
In conclusion, after training and testing the fashion-MNIST dataset using the
backpropagation algorithm to solve image classification the model can fit the dataset
whose best accuracy is 86.55%.

16
 Image Classification using Backpropagation Algorithm – Fashion Products by MNIST
Dataset | Evangelia Anastasaki

References

[1] C. Hill and C. Hill, “SciPy,” in Learning Scientific Programming with

Python, 2016.
[2] W. Lee, Python® Machine Learning. 2019.
[3] S. Nissen, “Neural Networks Made Simple,” Softw. 2.0, 2005.
[4] Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid, “Label-Embedding for
Image Classification,” IEEE Trans. Pattern Anal. Mach. Intell., 2016, doi:
10.1109/TPAMI.2015.2487986.
[5] A. C. Muller and S. Guido, Introduction to machine learning with scikit-
learn. 2017.
[6] N. Ketkar, Deep Learning with Python. 2017.
[7] S. Raschka and V. Mirjalili, Python Maching Learning: Machine Learning
and Deep Learning with Python, scikit-learn, and TensorFlow. 2017.
[8] R. Hecht-Nielsen, “Theory of the backpropagation neural network,” 1989,
doi: 10.1109/ijcnn.1989.118638.
[9] C. Kamusoko, “Image classification,” in Springer Geography, 2019.
[10] S. Tosi, Matplotlib for Python Developers. 2009.
[11] Anaconda Inc., “Anaconda Distribution,” Anaconda, 2019.
[12] G. Varoquaux, L. Buitinck, G. Louppe, O. Grisel, F. Pedregosa, and A.
Mueller, “Scikit-learn,” GetMobile Mob. Comput. Commun., 2015, doi:
10.1145/2786984.2786995.
[13] F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” J. Mach.
Learn. Res., 2011.
[14] D. Lu and Q. Weng, “A survey of image classification methods and
techniques for improving classification performance,” International Journal
of Remote Sensing. 2007, doi: 10.1080/01431160600746456.
[15] B. Klein and B. Klein, “NumPy,” in Einführung in Python 3, 2014.

[16] H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-MNIST: A Novel Image Dataset

for Benchmarking Machine Learning Algorithms,” pp. 1–6, 2017.

17