Artificial Neural Network

Introduction-Basics
Introduction
• Neural networks are information processing systems that are constructed
and implemented to model human brain.
• An artificial neural network (ANN) is an efficient information system
which resembles in characteristics with biological neural network.
• ANNs perform various tasks such as pattern matching and classification,
optimization function, approximation, vector quantization and data
clustering .
Biological nervous system
• Biological nervous system is the most important part of many living
things, in particular, human beings.
• There is a part called brain at the center of human nervous system.
• In fact, any biological nervous system consists of a large
number of interconnected processing units called neurons.

• Each neuron is approximately 10µm long and they can operate in

parallel.

• Typically, a human brain consists of approximately 1011 neurons

communicating with each other with the help of electrical impulses.
Biological neural network
The building block of a human brain is the biological neuron

❖ It consists of three primary parts, viz., the dendrites,

soma, and the axon.
❖ The dendrites collect stimuli from the neighbouring
neurons and pass it on to soma which is the main body of
the cell.
❖ The soma accumulates the stimuli received through the
dendrites. It ‘fires’ when sufficient stimuli is obtained.
❖ When a neuron fires it transmits its own stimulus through
the axon.
❖ Eventually, this stimulus passes on to the neighboring
neurons through the axon terminals.
• There is a small gap between the end of an axon terminal and the adjacent dendrite of the neighbouring
neuron. This gap is called the synapse.
• A nervous stimulus is an electric impulse. It is transmitted across a synaptic gap by means of
electrochemical process.
• The synaptic gap has an important role to play in the activities of the nervous system. It scales the input
signal by a weight.
 Terminology relationships between biological and artificial
Neurons
Biological Neuorn Artificial Neuron
Cell Neuron
Dendrites Weights or interconnections
Soma Net Input
Axon Output
Mathematical model of artificial neuron
Where i represents the i th processing element.
The activation function is applied over it to calculate output. The weights represents the strength of synapse connecting the
input and the output neurons.
A positive weight corresponds to an excitatory synapse and negative weight correspond to an inhibitory synapse
• ANNs possess large number of highly interconnected processing elements
called neurons, which usually operate in parallel and are configured in regular
architectures.

• Each neuron is connected with other by a connection link. Each connection

link is associated with weights which contain information about the input
signal.

• This information is used by the neuron net to solve a particular problem.

• ANNs collective behavior is characterized by their ability to learn, recall and

generalize training patterns or data similar to that of a human brain
 Basic Models of ANN
The models of ANN are specified by the three basic entities namely
1. The models synaptic interconnections
2. The training or learning rule adopted for updating and adjusting the connection weights
3. Their activation function

The arrangement of neurons to form layers and connection pattern formed within and between layers is called
the network architecture. There exists five basic types of neuron connection architectures

1. Single layer feed forward network

2. Multilayer feed forward network
3. Single node with its own feedback
4. Single layer recurrent network
5. Multilayer recurrent network
 2. Multilayer feed forward network
1. Single layer feed forward network

The competitive interconnections having fixed weights.

3. (A) Single node with its own feedback (B) Competitive nets
 4. Single layer recurrent network 5. Multilayer recurrent network

Recurrent networks are feedback networks with closed loop.

If the feedback of the output of the processing elements is directed back as input to the processing elements in the same
layer then it is called lateral feedback
 Learning
The main property of ANN is its capability to learn. There are two kinds of
learning in ANNs

1. Parameter learning : It updates the connecting weights in a neural net

2. Structure learning : It focuses on the change in network structure

Apart from these two the learning of ANNs generally categorized as

❖ Supervised Learning
❖ Unsupervised Learning
❖ Reinforcement learning
 Supervised Learning
❖ The learning here is performed with the help of a teacher.
❖ Each input vector requires a corresponding target vector,
which represents the desired output
❖ The input vector along with the target vector is called
training pair.
❖ Error signal is used for adjustment of weights until the
actual output matches the desired output.
❖ In this type of training a supervisor or teacher is required for
error minimization.
Unsupervised Learning
❖ The learning here is performed without the help
of a teacher.
❖ The input vectors of similar type are grouped
with out the use of training data to specify how a
member of each group looks or to which group
member belongs
❖ In the training process, the network receives the
input patterns and organizes these patterns to
form clusters.
❖ When a new input pattern is applied the neural
network gives an output response indicating the
class to which the input pattern belongs.
Reinforcement Learning

In supervised learning, the correct target output values are known for each input pattern

But, in some cases less information might be available. For example, the network might be told that its actual output is
only 50% correct or so. Thus, here only critic information is available not the exact information.

The learning based on this critic information is called reinforcement learning and the feedback sent is called
reinforcement signal .
Activation Functions
The activation function is applied over the net input to calculate the output of an ANN
Activation Functions (cont..)
Activation Functions (cont..)
Activation Functions (cont..)
Important terminologies of ANN
Important terminologies of ANN (cont…)

Threshold is a set value based upon which the final output of the network may be calculated.
The threshold value is used in the activation function
 McCulloch-Pitts Neuron
It is usually called as M-P neuron. The M-P neurons are connected
by directed weighted paths. The activation of M-P neuron is Binary

There is a fixed threshold for each neuron, and if the net input to
the neuron is greater than the threshold then the neuron fires.

It is excitatory with weight (w>0) or inhibitory with weight –

p(p<0).

The output will fire if it receives say k or more excitatory inputs but no inhibitory inputs where
 Linear Separability

An ANN does not give an exact solution for a nonlinear

problems. However it provides possible approximate
solutions to nonlinear problems

Linear separability is the concept wherein the separation of

the input space into regions is based on whether the
network response is positive or negative
Linearly vs nonlinearly separable patterns in 2D space
• So, far a 2‐classification problem, if there is a straight line, which acts as a
decision boundary then we can say such problem as linearly separable;
otherwise, it is non‐ linearly separable.
• The same concept can be extended to n‐classification problem. Such a
problem can be represented by an n‐dimensional space and a boundary
would be with n —1 dimensions that separates a given sets.
• In fact, any linearly separable problem can be solved with a single layer
feed forward neural network. For example, the AND problem.
• On the other hand, if the problem is non‐linearly separable, then a single
layer neural network can not solves such a problem. To solve such a
problem, multilayer feed forward neural network is required.
 HEBB Network
• Hebb explained as “ when an axon of cell A is near enough to excite cell B and
repeatedly or permanently takes place in firing it, some growth process or metabolic
change takes place in one or both the cells such that A’s efficiency as one of cells
firing B is increased.
• According to Hebb rule, the weight vector is found to increase proportionately to
the product of the input and the learning signal. Here the learning signal is equal to
the neurons output.
• The weight update in Hebb rule is given by

Note : The Hebb rule is more suited for bipolar data than binary data.
Training Algorithm-Hebb Network
Hebb Network-Flow Chart