Neural

Networks
Neural Networks
1. History

2. Threshold Neuron

3. Perceptron with Hebbian Learning

4. Linear Neuron as classifier

5. Linear Neuron as predictor

History
1943 − It has been assumed that the concept of neural network started with the work of physiologist,
Warren McCulloch, and mathematician, Walter Pitts, when in 1943 they modeled a simple neural
network using electrical circuits in order to describe how neurons in the brain might work.
1949 − Donald Hebb’s book, The Organization of Behavior, put forth the fact that repeated activation of
one neuron by another increases its strength each time they are used.
1956 − An associative memory network was introduced by Taylor.
1958 − A learning method for McCulloch and Pitts neuron model named Perceptron was invented by
Rosenblatt.
1960 − Bernard Widrow and Marcian Hoff developed models called "ADALINE" and “MADALINE.”
History continued..
• 1961 − Rosenblatt made an unsuccessful attempt but proposed the “backpropagation” scheme for
multilayer networks.
• 1964 − Taylor constructed a winner-take-all circuit with inhibitions among output units.
• 1969 − Multilayer perceptron MLP was invented by Minsky and Papert.
• 1971 − Kohonen developed Associative memories.
• 1976 − Stephen Grossberg and Gail Carpenter developed Adaptive resonance theory.
• 1982 − The major development was Hopfield’s Energy approach.
• 1985 − Boltzmann machine was developed by Ackley, Hinton, and Sejnowski.
• 1986 − Rumelhart, Hinton, and Williams introduced Generalised Delta Rule.
• 1988 − Kosko developed Binary Associative Memory BAM and also gave the concept of Fuzzy
Logic in ANN.
Biological neuron
What’s in a neuron
• Dendrites − They are tree-like branches, responsible for receiving the information from
other neurons it is connected to. In other sense, we can say that they are like the ears of
neuron.
• Soma − It is the cell body of the neuron and is responsible for processing of information,
they have received from dendrites.
• Axon − It is just like a cable through which neurons send the information.
• Synapses − It is the connection between the axon and other neuron dendrites.
ANN vs. BNN
• Artificial Neural Network ANN and Biological Neural Network BNN
similarities.

Biological Neural Network Artificial Neural Network

(BNN) (ANN)
Soma Node
Dendrites Input
Synapse Weights or Interconnections
Axon Output
Communication between neurons
Neural
Network
types
• Circles(neurons) are
individual processing
elements
• The arrows are links
between them from inputs to
output(s)
NN types explanation
• 2.2a Linear classifier and is functionally similar to simple- and multiple-discriminant function
analysis in statistics.
• 2.2b Linear classifier and predictor whose predictive capabilities are equivalent to simple and multiple
linear regression models
• 2.2c Neural network for the nonlinear prediction and classification

• 2.2d Unsupervised networks that can find clusters in the data

•2.2e Self-organizing feature map(SOM) finds unknown clusters in the data but also preserves the
topological structure (spatial relations) of the data and clusters.
•2.2f Networks contain feedback links that help to capture temporal effects
Threshold function(McCulloch-Pitts)
x1

u y
𝛴

x2 Threshold function
Product purchase decision
Category Price OK Colour OK Combined decision Final decision
X1 X2 u=x1 + x2 y
1 0 0 0 0
2 0 1 1 1
3 1 0 1 1
4 1 1 2 1

y = u = f(x1,x2) = u = f(x1,x2) = x1 + x2
Network with weights
x1
w1

u y
𝛴

x2 w2
Threshold function

u = f(x1,x2) = w1x1 + w2x2

 Threshold neuron
x1 x2 t
0.2 0.3 0
0.2 0.8 0
0.8 0.2 0
1.0 0.8 1

Σ=𝑢=𝑤 1 𝑥 1+𝑤 2 𝑥 2
Input(x1, x2) u y
(0.2, 0.3) 0.5 0
{
𝑓 ( Σ )= 𝑦= 0 𝑢<1.3
1 𝑢 ≥ 1.3 } (0.2, 0.8) 1.0 0
(0.8, 0.2) 1.0 0
(1.0, 0.8) 1.8 1
Hebbian learning main idea
Hebbian learning variations
Appendix
Biological to Artificial

a. Biological working
of a neuron

b. Neuron model

c. Detailed working of
single neuron
Detailed working of Single Neuron