Neural Networks

14th Nov 2022

Recent Success stories !!
NN in Computer Vision
SR - GANs
NLP - Captioning
Saliency
Medical Diagnostics
Speech
 NN - Wiki

• Neural networks are non-linear statistical Data modeling or

Decision making tools.

• Model complex relationships between inputs and outputs; Used

to find patterns in Data.
NN- Attempt to emulate brain
NN- Attempt to emulate brain
Neuron => Perceptron
Different Activation functions
What can a Single Perceptron
do ?
 Single Perceptron - Limitation
• Fails when Data is arranged as XOR
Single Perceptron : Limitation
Single Perceptron : Limitation
Re-examine XOR
 Layers of Perceptrons (MLP)

With careful choices of Weights and Biases, the XOR can

be realized using MLP
Getting to the Decision
Boundary

Linear separator
from Perceptron 1

AND-ing of the
Regions from the
two Perceptrons

Linear separator
from Perceptron 2
MLPs - Complex functions
Building General Boundaries
Composing the Boundary
More-complicated Boundaries
Generalizing Boundaries
 From Boolean to Real I/Ps

• What does MLP do with Real I/Ps ???

 Weights to compute

Note : The impact of Dimensions “D”

More sophisticated boundary will have HIGH “N”

What is the Hypothesis class ?
 What is the Hypothesis class ?

• The function realized by the Trained Neural

Network with “L” Layers is of the form,
f = fL (f(L -1)(…..f2(f1(x))……))
It is a composition of functions

How..???
 Composition of Functions

• f = fL (f(L -1)(…..f2(f1(x))……))
At each Layer “l” , each node “j”, the output,
x j(l) = θ (W (l) x (l-1))
At next layer “l + 1 ” , each node “j”, the output,

x j(l + 1 ) = θ (W (l+1) x (l)) = θ (W (l+1) θ (W (l) x (l-1)) )

And so on…..
 Universal Approximation
Theorem
Neural networks can realize, to any arbitrary
approximation, any desired continuous function.
Appropriate Weights and Biases
 Limitation of the Neural
Network
• This is a curve-fitting technique

• It cannot explain causality of a predictor.

• Eg. F = ma, {(f1,a1), (f2,a2),……(f100,a100)}

With Data points pertaining to this phenomenon
NN cant answer if Force caused the Acceleration, or the
Other-way.
 Capacity of the Network

• Not ALL Architectures can realize Approximations

of all Continuous functions

• The Number of functions that can be realized by

the Network is determined by the Number of
Layers, Number of Hidden Neurons, the range of
values the weights can take, Number of inputs ???
 VC Dimension

• Given “H” hidden neurons, each with “K” weights,

The VC Dimension is of the order (HK)

Implication on the Number of training samples needed ?

 Connectionist Architecture

• The function of the Neural network depends on the

Architecture and Interconnections between the
Neurons.
• What the Neural Network is capable of doing, what
information it holds – are all embodied in the
architecture, Weights and Biases
• In contrast "von Neumann architecture" refers to
any stored-program computer Eg. Mobile Phones,
CPUs
Learning Algorithm….
3-Layer NN
Non-linear Boundaries
Generic Neural Network
NN- Constituents
Learning Process
Gradient Descent
Stochastic Gradient Descent
SGD - Benefits
Applying Chain Rule
Recursion in Delta computation
Back-Propagation Algo