A

Major Project Report

ON
“MNIST (DIGIT RECOGNISATION)”

SUBMITTED TO
RAJIV GANDHI PROUDYOGIKI VISHWAVIDYALAYA
(M.P.)

In partial fulfillment for the requirement of VIII Sem of

BACHELOR OF ENGINEERING
IN
INFORMATION TECHNOLOGY
YEAR 2016-2020

SUBMITTED BY
Dishakshi Jadon(0905IT161024)
Prasuk Jain (0905IT161040)
Sandesh Singhal (0905IT161047)
Tanu Singhal (0905IT161060)

GUIDED BY
CP Bhargav
Assistant professor
Deptt. of CSE &
IT ITM, Gwalior
(M.P)

DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING /DEPARTMENT OF

INFORMATION TECHNOLOGY

INSTITUTE OF TECHONOLOGY & MANAGAMENT GWALIOR-474001

Sandesh Singhal
 INSTITUTE OF TECHONOLOGY &
MANAGAMENT GWALIOR

DEPARTMENT OF COMPUTER SCIENCE &

ENGINEERING / DEPARTMENT OF INFORMATION
TECHNOLOGY

CERTIFICATE

This is to certify that “ Dishakshi Jadon (0905IT161024), Prasuk Jain (0905IT16140), Sandesh
Singhal (0905IT161047), Tanu Singhal (0905IT161060” students of Bachelor of Engineering
(Computer Science & Engineering) VIII semester has developed a project on MNIST under the
guidance of CP BHARGAV using Machine learning for the partial fulfillment of Bachelor of
Engineering Computer Science & Engineering and submitted a satisfactory report of the project.
This work has not been submitted in part or full to this or any other university for the award of any
degree or diploma to the best of my knowledge.
We wish him/ her success in the future.

Prof. CP Bhargav
Deptt. of CSE &
IT ITM, Gwalior
(M.P.)

Sandesh Singhal
 Candidate Declaration

I/We Dishakshi Jadon (0905IT161024), Prasuk Jain (0905IT161040), Sandesh Singhal

(0905IT161047), Tanu Singhal (0905IT161060) students of Bachelor of Engineering
Information & technology VIII semester, hereby declare that I/we have developed this project
on MNIST. This report is developed by us under the guidance of Prof. CP BHARGAV using
MACHINE LEARNING. We submit this report for the partial fulfillment of the VIII semester of
Bachelor of Engineering Information & Technology. This work has not been submitted in part
or full to this or any other university for the award of any degree or diploma to the best of our
knowledge.

Dishakshi Jadon (0905IT161024)

Prasuk Jain (0905IT161040)

Sandesh Singhal (0905IT161047)

Tanu Singhal (0905IT161060)

Sandesh Singhal
 INSTITUTE OFTECHNOLOGY & MANAGMENT
GWALIOR (M.P.)

DEPARTMENT OF COMPUTER SCIENCE

& ENGINEERING / DEPARTMENT OF
INFORMATION TECHNOLOGY

ACKNOWLEDGEMENT

At the outset, I would like to thank my guide Prof CP Bhargav, Department of

Computer Science & Engg., Gwalior for supporting to work on this project and
providing us sample and valuable guidance through the course of this project.

My sincere thanks go to Prof Manuj Mishra, Department of Computer Science & Engg.,
Gwalior
for helping me in various ways.

I wish to thank Dr. Rishi Soni, HOD, Department of Computer Science & Engg.,
Gwalior for his continuous moral support.

I am equally grateful to Dr. Meenakshi Mazumdar, DIRECTOR, ITM Gwalior for providing
me all the necessary resources to carry out this project work. I would like to thank all the
staff members in the Department of Computer Science & Engg., Gwalior for their
support.

Dishakshi Jadon (0905IT161024)

Prasuk Jain (0905IT161040)

Sandesh Singhal (0905IT161047)

Tanu Singhal (0905IT161060)

Sandesh Singhal
INTODUCTION

MNIST is a simple computer vision dataset. It consists of images of handwritten digits like these:

It also includes labels for each image, telling us which digit it is. For example, the labels for the
above images are 5, 0, 4, and 1.

We're going to train a model to look at images and predict what digits they are. Our goal isn't to
train a really elaborate model that achieves state-of-the-art performance.

The MNIST database (Modified National Institute of Standards and Technology database) is a large
database of handwritten digits that is commonly used for training various image processing systems.
The database is also widely used for training and testing in the field of machine learning.

Handwritten digits recognition problem has been studied by researchers since 1998 with almost all
the algorithms designed by then and even until now. The test error rate decreased from 12% in 1988
by linear classifier to 0,23% in 2012 by convolutional nets, and these days more and more data
scientists and machine learning experts are trying to develop and validate unsupervised learning
methods such as auto-encoder and deep learning model.

The recognition of handwritten characters individually, we create a pipeline that allows any image of
handwritten text passed in and segmented into separate characters. Our final project will allow
essentially any imagine of a document or note to be segmented and translate to a digitized version.

Sandesh Singhal
AIM

Train a computer system to first understand what arc handwritten images of digit a then test the system
to predict new handwritten images correctly. We will feed our system with MNIST dataset and test it
with random images to check the accuracy of its predictions.

Today, we will sec TensorFlow MNIST Dataset. This TensorFlow MNIST tutorial will teach us the
meaning of TensorFlow MNIST, Moreover, we will discuss SoftMax regression and
implementation of MNIST dataset in TensorFlow. Also, we will see the training and accuracy of

TensorFlow MNIST dataset.

Here, we will learn how to create a function that is a model for recognizing handwritten digits by
looking at each pixel in the image, then using TensorFlow to train the model to predict the image
by making it look at thousands of examples which are already labeled (supervised learning). You
will then check the model's accuracy with a test dataset.

Sandesh Singhal
PROBLEM SELECTION

The MNIST problem is a dataset developed by Yann LeCun, Corinna Cortes and Christopher
Burges for evaluating machine learning models on the handwritten digit classification problem.
The dataset was constructed from a number of scanned document dataset available from
the National Institute of Standards and Technology (NISI.). This is where the name for the dataset
comes from, as the Modified NIST or MNIST dataset.
Images of digits were taken from a variety of scanned documents, normalized in size and centered.
This makes it an excellent dataset for evaluating models, allowing the developer to focus on the
machine learning with very little data cleaning or preparation required.

Each image is a 28 by 28-pixel square (784 pixels total). A standard split of the dataset is used to
evaluate and compare models, where 60,000 images are used to train a model and a separate set
of 10,000 images are used to test it.

It is a digit recognition task. As such there are 10 digits (0 and 9) or 10 classes to predict. Results
are reported using prediction error, which is nothing more than inverted classification accuracy.

Excellent results achieve a prediction error of less than1% State of art prediction error of
approximately 0.2% can achieve with large convolutional neural network. There is a listing of the
state-of-the-art results and link to the relevant papers on the MNIST.

Sandesh Singhal
GANTT CHART/PERT CHART

Sandesh Singhal
SYSTEM STUDY

Hand writing recognition of character has been around since the 1980s. The task of handwritten
digit recognition, using a classifier, has great important and use such as- Online handwriting
recognition on computer tablets, recognize zip codes on mail for postal mail sorting, processing
bank check amount, numeric entries in forms filled up by hand (for example- tax forms) and so
on. There are different challenges faced while attempting to solve this problem. The handwritten
digits are not always of the same size, thickness, or orientation and position relative to the margin.
Our goal was to implement a pattern classification method to recognize the handwritten digits
provided in the MNIST dataset of images of handwritten digits (0-9). The dataset used for
application is composed of 300 training images and 300 testing images, and is a subset of MNIST
dataset [1] (originally composed of 60,000 training images ad 10,000 testing images). Each image
is a 28*28 grayscale (0-255) labeled representation of an individual digit. The general problem
we predicted we would face in this digit classification problem was the similarity between the
digits like 1 and 7, 5 and 6, 3 and 8, 9 etc. Also, people write the same digit in many different ways
– the digit ‘1’ is written as ‘1’,’1’, ’1’ or ‘1’. Similarly, 7 may be written as ‘7’,’7’, or ‘7’. Finally,
the uniqueness and variety in the handwritten of different individuals also influences the formation
and appearance of the digits.

Sandesh Singhal
PROPOSED SYSTEM

In MNIST dataset, the images were centered in a 28*28 image by computing the center of the
pixels, and translating the image so as to position this point at the center of the 28*28 field. The
training set is composed of 30,000 patterns from SD-3 and 30,000 patterns from SD-1. The test set
was composed of 5,000 patterns from SD-3 and 5,000 patterns from SD-1. Because the dataset is
too huge, a better idea is beginning with an available subset, while could be found of Kaggle
webpage. Although the dataset has been labeled, it’s still my task to design proper method to
preprocessing the data including read binary files with proper programming language and choose
acceptance data structure to avoid overflow, Moreover, a series of machine learning algorithms
will be applied to larger dataset for evaluation. If time permits, I’d like to make more attempts to
improve the algorithms or find some interesting application of such models.

This study focuses on features extraction and classification. The performance of a classifier can
read as much on the quality of the features as on the classifier itself. Moreover, our study presents
efficient offline handwritten character recognition system based on diagonal features and
transitions features using the K-NN classifier. Diagonal and transition features of a character have
been computed based on the distribution of points on the bitmap image of the character. In this
study, we compare the performance of five different machine learning classifiers for recognition
of digits. The five classifiers namely neural network, K-Nearest Neighbor, Random Forest,
Decision Tree and Bagging with gradient boost. The combination of different classifiers, Such as
ANN, and K-NN implant in a recognition system is also a suitable method of increasing
recognition performance, because different classifier combinations have different merits in dealing
with discriminant problems. Then, a digit recognition module is employed to classify each
segmented digit completing the handwritten digit recognition task. In this study, a novel method,
which uses the K-Nearest Neighbors classifier proposed to achieve high performance than Neural
Net to discover that the result of our study is different than other studies. K-Nearest Neighbor (K-
NN) classifier performed slightly better than the neural network.

Sandesh Singhal
DESCRIPTION OF DATASET
The MNIST database of handwritten digits, available from this page, has a training set of 60,000
examples, and a test set of 10,000 examples.

The original black and white (bilevel) images from NIST were size normalized to fit in a 20x20
pixel box while preserving their aspect ratio. The resulting images contain grey levels as a result
of the anti-aliasing technique used by the normalization algorithm. the images were centered in a
28x28 image by computing the center of mass of the pixels, and translating the image so as to
position this point at the center of the 28x28 field.

With some classification methods (particularly template-based methods, such as SVM and K-
nearest neighbors), the error rate improves when the digits are centered by bounding box rather
than center of mass. If you do this kind of pre-processing, you should report it in your publications.

The MNIST database was constructed from NIST's Special Database 3 and Special Database 1
which contain binary images of handwritten digits. NIST originally designated SD-3 as their
training set and SD-1 as their test set. However, SD-3 is much cleaner and easier to recognize than
SD-1. The reason for this can be found on the fact that SD-3 was collected among Census Bureau
employees, while SD-1 was collected among high-school students. Drawing sensible conclusions
from learning experiments requires that the result be independent of the choice of training set and
test among the complete set of samples. Therefore, it was necessary to build a new database by
mixing NIST's datasets.

The MNIST training set is composed of 30,000 patterns from SD-3 and 30,000 patterns from SD-
1. Our test set was composed of 5,000 patterns from SD-3 and 5,000 patterns from SD-1. The
60,000-pattern training set contained examples from approximately 250 writers. We made sure
that the sets of writers of the training set and test set were disjoint.

SD-1 contains 58,527-digit images written by 500 different writers. In contrast to SD-3, where
blocks of data from each writer appeared in sequence, the data in SD-1 is scrambled. Writer
identities for SD-1 is available and we used this information to unscramble the writers. We then

Sandesh Singhal
split SD-1 in two: characters written by the first 250 writers went into our new training set. The
remaining 250 writers were placed in our test set. Thus, we had two sets with nearly 30,000
examples each. The new training set was completed with enough examples from SD-3, starting at
pattern # 0, to make a full set of 60,000 training patterns. Similarly, the new test set was completed
with SD-3 examples starting at pattern # 35,000 to make a full set with 60,000 test patterns. Only
a subset of 10,000 test images (5,000 from SD-1 and 5,000 from SD-3) is available on this site.
The full 60,000 sample training set is available.

DATA TRAINING TESTING SUBTOTAL

9 5949 1009 6958
8 5851 974 6825
7 6265 1028 7293
6 5918 958 6876
5 5421 892 6313
4 5842 982 6824
3 6131 1010 7141
2 5958 1032 6990
1 6742 1135 7877
0 5923 980 6903
TOTAL 60,000 10,000 70,000

Sandesh Singhal
FEASIBILITY STUDY

CHARACTER RECOGNITION

For training on the MNIST dataset, we will implement different neural networks and convolution
neural networks architectures and compare the accuracy of the different architectures. The large
collection of examples and potentially complex characters due to variability in handwriting using
multilayer networks trained with gradient descent an obvious candidate for recognition tasks.
LeCun et al. (1998) implemented a seven-layer convolution neural network not including the input
layer that had an error of 0.9% on the MNIST dataset. We will try to at least minimize this error
rate and possibly improve on the result by implementing brewer concept like dropout in per
network. Like LeCun et. al.., we will also implement a basic linear classifier and k-nearest
neighbors’ classifier to establish a minimum baseline for our neural networks. We expect the digit
classifiers to outperform the characters classifiers due to only 10 different classifications instead
of 62. Moving on to the character dataset, we plan on implementing the supervised classifier
models used in de Campos et. al. (2009) and attain a similar accuracy as well as incorporate the
techniques used in LeCun et al. with respect to shallow and deep convolutional neural network. In
particular, we can implement the convnet architecture used in the Lecun et al. paper, use transfer
learning with a larger network, as well as implement and train a common architecture discusses
class.

Sandesh Singhal
Image Processing –

Slope and slant correction of handwritten words are necessary to reduce the variations in
handwriting styles. Careful estimation of the slope and slant can make the following segmentation
process much simpler. In the ideal scenario, a word is written horizontally with ascenders and
descenders aligned in the vertical direction. However, this is rarely the case. Slope is defined as
the angle between the horizontal direction and the direction of the line the word is aligned. Slant
is defined as the angle between vertical direction and the direction of strokes supposed to be
vertical. Before segmentation occurs, we should work to eliminate both of these angles. To handle
the slant estimation, we referred to the work done by Papandreou et. al. First, the word’s core
region is found using black run profiles. From there, we can divide the core region into vertical
strips and determine the centroid for each region and fit a straight line through the centroids. In
addition, this calculation also provides an estimation of the average thickness of the stroke, which
will be used in slant correction.

Sandesh Singhal
SYSTEM ANALYSIS-

This project as a warm-up aims to explore feature extractions using existing networks, such as
pretrained deep neural networks and scattering nets, in image classifications with traditional
machine learning methods. 1. Pick up ONE (or more if you like) favorite dataset below to work.
If you would like to work on a different problem outside the candidates we proposed, please email
course instructor about your proposal.

2. Team work: we encourage you to form small team, up to FOUR persons per group, to work on
the same problem. Each team just submit ONE report, with a clear remark on each person’s
contribution. The report can be in the format of either a technical report within 8 pages, e.g. NIPS
conference style (preferred format).

3. MNIST Dataset

Sandesh Singhal
 SYSTEM ANALYSIS

Training Pre- Feature Recognize

test Processi Extractio Classificatio d Digit
ng n n

DATA FLOW DIAGRAM

Data Flow Diagram 1

Sandesh Singhal
Data Flow Diagram 2

Sandesh Singhal
\

Data Flow Diagram 3

Sandesh Singhal
SOURCE OF DATA-

Source of data for MNIST project is from Spyder interface. There is in built dataset for MNIST
knows as MNIST original. The MNIST data set is still very popular and important for evolution
and validation of new algorithms. In MNIST dataset, the data is already well prepared: the images
were centered in a 28*28 image by computing the center of mass of the pixels, and translating the
image so as to position this point at the center of the 28*28 field.

Sandesh Singhal
SCOPE OF THE PROJECT-

• Extend the model to work on MNIST dataset.

• Increase the accuracy further by implementing more number of hidden layers and/or
epochs.
• Detect custom Hand Written digits.
• Use CNN with less layers to get better accuracy.

Sandesh Singhal
REFERENCES-

• http://neuralnetworksanddeeplearning.com/
• https://en.wikipedia.org/wiki/MNIST_database
• https://machinelearningmastery.com/how-to-develop-a-convolutional-neural-network-
from- scratch-for-mnist-handwritten-digit-classification/
• https://towardsdatascience.com/image-classification-in-10-minutes-with-mnist-
dataset- 54c35b77a38d
• https://medium.com/@mjbhobe/mnist-digits-classification-with-keras-ed6c2374bd0e
• http://yann.lecun.com/exdb/mnist/
• https://www.python-course.eu/neural_network_mnist.php

• https://towardsdatascience.com/mnist-cnn-python-c61a5bce7a19
• https://machinelearningmastery.com/handwritten-digit-recognition-using-convolutional-
neural- networks-python-keras/

Sandesh Singhal