Unit 1

Lecture 5
15-07-2020
Topics
• ANN vs Biological Neuron
• Building blocks
• Architecture
• Types of learning
• Activation functions
• Important terminologies of ANN
ANN
Execution time is in few nanoseconds
Processing is parallel (much faster than biological
neuron)
Size and complexity depends on the application
Information is stored in contiguous memory locations,
can be overloaded and some information can get
overwritten, whatever is stored can be retrieved
Not fault tolerant, if the interconnections fail the
network becomes corrupt. Information loss can occur
Control mechanism is much simpler. CPU controls
everything. Control passed from unit to unit by the
CPU
Biological Neuron
Few milliseconds
Processing is parallel
Size and complexity much higher(1011 neurons with
1015 interconnections)
Information is stored in interconnections and synapse
strength, any amount of information can be stored by
making new interconnections, brain memory can fail
to recollect stored information.
Fault tolerant. Even if the interconnections fails it
works fine. Distributed nature enables to store and
retrieve information. Even if some cells die performs
pretty well.
Controlled by chemical reactions and the signal
strength. Complex control mechanism
Building blocks of ANN
• Neurons
• Synaptic interconnections
• Weights
• Learning rules adapted for updating the weights
• Activation functions
• Inputs
• Bias
Architecture
Arrangement of neurons to form layers and the connection patterns formed
within and between layers
5 different architectures
1) Single layer feed forward
2) Multilayer feed forward
3) Single node with its own feedback
4) Single layer recurrent
5) Multi layer recurrent

Basically divided into single layer and multilayer

• Layers of processing nodes
formed
• Inputs connected to outputs
using weights
• Wij i is the source node
j is the destination node
• Input layer : receives the
signal and buffers, no
processing done
• Output layer: generate
output of the whole network
• Hidden layer: between input
and output
• No direct contact with
the external
environment
• More no of hidden layers
more complex the n/w is
Feed forward
• No neuron in the output layer is an input to a node in the same layer or preceding
layer
Feed backward:
• When outputs are directed back as inputs to the nodes in same layer or preceding
layers
Lateral feedback
• When outputs are directed back as inputs to the nodes in same layer
Recurrent networks
• Neuron feedbacks to itself
Competitive nets
• Interconnection links assigned a fixed weight of -epsilon
Two types of Learning
• Parameter learning
• Updates and adjusts the connection weights
• Structure learning
• Focus of change in the network structure
• No. of processing elements
• Connection types used between layers
Also Supervised, Unsupervised, Reinforcement
Activation functions
• Some function applied on the net input to get the output of the total network
• To make the output more efficient
• To get a non-linear output to solve complex problems
Commonly used activation functions
• Identity function
• Output same as input

• Binary step function

• Theta is a chosen threshold

• Bipolar step function

• Converts input to a bipolar output (-1 and +1), used in single nets

•Sigmoid function (widely used)

•Binary sigmoid (output between 0 and 1)

•Bipolar sigmoid (output between -1 and +1)

• Ramp function

• tanh (tangent hyperbolic function)

• Variation of sigmoid
• f(x)= 2*sigmoid(2x)-1
• Widely used better than sigmoid
• RELU(Rectified Linear Unit)
• f(x)= max(0,x)
• If x>0 then x else 0
• Computationally less expensive than sigmoid and tanh
• Widely used
• If output needed is binary use sigmoid, if no idea which one to use, use RELU
Important Terminologies in ANN
• Weights
• Assigned to interconnection links
• Denoted as wij where i is the source node and j is the destination node
• Weights of a network normally denoted using a weight matrix
• Bias
• Included by adding an input component x0
• X=(x0,x1,x2….xn)
• x0 = 1
• X=(1,x1,x2….xn)
• Bias is considered like another weight w0j = bj
• Y=mx+c c is the bias
• Plays a major rule in determining the output of the n/w
• Positive bias increases the net input of the network
• Negative bias decreases the net input of the network
• Threshold
• Set or fixed value based on which the output is calculated
• Net calculated input compared with the threshold to obtain the net output of the n/w
• Learning rate
• Denoted as alpha
• Used to control the amount of weight adjustment at each step of training
• Ranges from 0 to 1
• Momentum factor
• To make the convergence faster
• Added to the weight updation process
• Used in back propagation network
• Based on the error rate fine tuning of the weights is done
• Vigilance parameter
• Denoted by rho
• Used in adaptive resonance network
• Degree of similarity required for pattern to be assigned to the same cluster unit