Neural Networks for Image

Classification
 Overview
! Inspired by biological nervous system


! Consist of neurons that fire electrical signals


http://physicsworld.com/cws/article/news/2011/sep/13/physicists-in-tune-with-neurons

https://sites.google.com/site/secretsofthebrainsz/neurons
 Overview

! Neurons can be used as a powerful computational model


! Flexible system for machine learning

http://singularityhub.com/wp-content/uploads/2015/08/autistic-neural-network-3.jpg
 History
! Warren McCulloch and Walter Pitts (1943)
! Paul Werbos (1975)
! Best MNIST Performance (2012)
! Swiftkey (2015)
! AlphaGo (2016)

http://goo.gl/7iPTK0

https://blog.swiftkey.com/neural-networks-a-meaningful-leap-for-mobile-typing/
https://goo.gl/djpFVt
 Motivation
! Neural Networks is an emerging field in machine-learning


! Combines biological and computational fields


! Wide range of implementation complexities


http://www.frontiersin.org/files/Articles/7025/fninf-04-00112-r1/image_m/fninf-04-000112-g001.jpg
 Approach
! Learn image features using adaptable filters


! Sigmoidal Activation Function


! Feed-Forward Back-Propagation

https://upload.wikimedia.org/wikipedia/commons/thumb/6/60/ArtificialNeuronModel_english.png/600px-ArtificialNeuronModel_english.png
 Layer Architecture
! Each pixel value is an input neuron
! Input-to-Hidden and Hidden-to-Output Weights
! Each output neuron corresponds to a categorical value

http://engineering.flipboard.com/assets/convnets/Convolution_schematic.gif

http://www.extremetech.com/wp-content/uploads/2015/07/NeuralNetwork.png
 Sigmoidal Activation

http://i.stack.imgur.com/iIcbq.gif http://www.saedsayad.com/images/ANN_Sigmoid.png
 Feed-Forward Back-Propagation

https://idea2bank.files.wordpress.com/2011/04/bpp.jpg
 Software
! Modularity 


! User Interface


! MATLAB for generating training data

https://docs.toradex.com/102686-modularity-icon.jpg?v=1
 Procedure
! Trained on set of 28,000 images
○ 10 digits and 4 symbols
■ Generated symbol images


! Parameter Optimization
○ Number of Hidden Nodes
○ Momentum
○ Learning Rate Annealing


https://www2.warwick.ac.uk/fac/cross_fac/complexity/study/
msc_and_phd/co902/2013_2014/resources/mnisttrain.png
 Results
Digit Marginal Accuracy

! 95.43% accuracy on MNIST test database 0 98%

1 99%

2 92%
Classifier Accuracy
3 91%
Linear 88.0%
4 93%
K-Nearest-Neighbors 95.0%
5 90%

2-layer NN, 1000 hidden units 95.5%

6 96%

40 PCA + quadratic classifier 96.7% 7 93%

3-layer NN, 500+300 HU 98.47% 8 91%

http://yann.lecun.com/exdb/mnist/ 9 90%
 Results
! Tested network on our own handwritten digits and symbols


! Inconsistent accuracy across digits

 Challenges
! Back-Propagation algorithm


! Proper parameter selection

○ Achieve >90% accuracy
○ Efficient training


! Sufficient training data


! LCDK USB speeds

http://www.ebuyer.com/blog/wp-content/uploads/2015/07/neural-map.jpg
 Discussion
! Network performed better on the MNIST testing 


! Slow USB speeds were not solved


! Add more hidden layers


! Testing design flexibility


http://www.theneweconomy.com/wp-content/uploads/2015/10/neural-networks.jpg
 Conclusion
! Neural networks do work!


! Number of layers modeling complexity 


! Networks have a small parameter solution space 


http://www.learningmachines101.com/wp-content/uploads/2015/09/LM101-035image.jpg