Professional Documents
Culture Documents
Neural Networks-I
(Introduction & Architecture)
Soft Computing (KCS- 056)
MR. Saurabh Singh Tomar (Asst. Professor)
Department of Computer Science &
Engineering
United College of Engineering & Research,
Prayagraj
Syllabus
Lecture Detail
Soft computing provides insights into real-world problems and is just not
limited to theory.
Some important fields like Biology, Medicine and humanities, etc are still
intractable using Convention mathematical and Analytical models.
It is possible to map the human mind with the help of Soft computing but it is
not possible with Convention mathematical and Analytical models.
GOALS OF SOFT COMPUTING
• To better understand what fuzzy logic is, take a look at the simple diagram below.
Paradigm/Techniques of soft computing
Artificial Neural Networks
• An artificial neural network is a computer program that emulates a particular
biological counterpart. A machine, therefore, designed to work as a human brain
is an artificial neural network. It uses trial and error to get to the desired output.
Genetic Algorithms
• Genetic algorithms refer to a group of search methods that are inspired by the
theory of evolution. As such, they create sets of solutions that evolve to get the
lowest or highest value of an objective function or a linear expression (in math,
that’s the formula you are given, such as f = c1x1 + … + cnxn).
Paradigm/Techniques of soft computing
• They help obtain all the values that would possibly result given a specific objective function.
Application of genetic algorithm Traveling salesman problem and its applications, DNA Analysis,
Scheduling applications e.t.c
Hybrid systems: A Hybrid system is an intelligent system that is framed by combining at least two
intelligent technologies like Fuzzy Logic, Neural networks, Genetic algorithms, reinforcement
learning, etc. The combination of different techniques in one computational model makes these
systems possess an extended range of capabilities.
1. Nerve Structure
•The concept of neurons as the fundamental constituents of the
brain.
•Brain contains about basic units called neurons. Each unit in turn,
is connected to other neurons. A neuron is small cell that receives
electro-chemical signal from its various sources and in term
respond by transmitting electrical impulses to the other neurons.
•Some of neurons of neurons perform input output operations
referred afferent and efferent cells respectively. Remaining neurons
are part of interconnected networks responsible for information
storage and signal transmission.
2. Structure of Neuron
A neuron is composed of:
• Input: Inputs are the set of values for which we need to predict a output value.
They can be viewed as features or attributes in a dataset.
• Weights: weights are the real values that are attached with each input/feature
and they convey the importance of that corresponding feature in predicting the
final output.
• Transfer function - The job of the transfer function is to combine multiple inputs
into one output value so that the activation function can be applied. It is done by
a simple summation of all the inputs to the transfer function.
• Activation Function—It introduces non-linearity in the working of neural
netwoork to consider varying linearity with the inputs. Without this, the output
would just be a linear combination of input values and would not be able to
introduce non-linearity in the network.
• Bias - The role of bias is to shift the value produced by the activation function. Its
role is similar to the role of a constant in a linear function.
Architecture of simple artificial neural network:
neural network has neurons that work in correspondence of weight, bias and their respective
activation function.
There are several activation functions. Some of them are :
A neural network without an activation function is essentially just a linear regression model.
The activation function does the non-linear transformation to the input making it capable to
learn and perform more complex tasks.
1. Identity Function :It is also called linear function and can be defined as
It means output of neuron will be equal to its net input. This function
is basically used to find activation of input layer neurons
7. Tangent Function: It is given by y = tanh(yin ) and used to produce negative
output values
Neural Network Architecture
• An, ANN architecture is represented using directed graph. A graph G
(V, E) is 2-tuple where V represents set of vertices and E represents set
off edges. It assumes significance in neuron because signals in NN
systems are restricted to flow in specific directions.
• The vertices of the graph may represent neuron and the edges the
synaptic links. There are several classes of NN according to their
learning mechanism. There are three fundamental classes off
networks
Single Layer Feed forward
The input layer neurons receive input signals and output neurons receive output
signals. The synaptic links carry weights from every input neuron to every
output neuron but not vice versa. This network is called single layer feedf
orward network and acyclic in nature.
Inputs are connected to the processing nodes with various weights, resulting is
series of outputs one per node.
#Multilayer Feedforward Network:
•As its name indicates is made up of multiple layers.
•Beside input layer and output layer this architecture has several intermediary layers
called hidden layers.
•Computation of hidden unit is called hidden neurons.
•The hidden layer aids in performing useful intermediary computations before
directing input to the output layer.
•The input layer neurons are linked to hidden layer neurons and the synaptic weights
are called input-hidden weights.
• Again hidden layer neurons are linked to output neurons and corresponding weights
are called hidden-output weights.
•The figure given below is called l – m – n architecture because there are l input
neurons, m hidden neurons and n output neurons
Where is input neuron, is hidden neuron is output neuron weight of
interconnection between ith input neuron and jth hidden neuron and weight of
interconnection between jth hidden neuron and kth output neuron.
Recurrent network:
These networks differ from feedforward network architecture in the sense that
there is at least one feedback loop. There could also be neurons with self-feedback
link as shown figure. If neurons feedback in same layer it is called lateral feedback.
McCulloch-pits neuron model:
• It is also called M-P neuron model.
• It is the first computational(mathematical) model of neuron proposed by Warron McCulloch and Walter Pitts
in 1943.
• This model allows binary 0 or 1 states only.
• these binary neuron are connected by directed weighted path.
• the connected path can have positive weights (excitatory) or negative weights (inhibitory).
• There will be same weights for positive or negative.
• The neuron is associated with the threshold value.
• The neuron activates(fires), if the total input to the neuron is >= to the threshold.
• The M-P neuron model has no particular training algorithm.
• The M-P neurons are used as basic building blocks on which we can model any function or phenomenon, which
can be represented as a logic function.
McCulloch-pits neuron model
• Since the firing of output neuron is based upon the threshold,
activation function here is defined as
1 𝑖𝑓 𝑦𝑖𝑛 ≥ 𝜃
Y=ቊ
0 𝑖𝑓 𝑦𝑖𝑛 < 𝜃
What is Learning in ANN
• The main property of an ANN is its capability to learn. Learning or training is a
process by means of which a neural network adapts itself to a stimulus by
resulting in the production of desired response. Broadly, there are two kind of
learning in ANN.
• Learning, in artificial neural network, is the method of modifying the
weights of connections between the neurons of a specified network.
1. Parameter learning: it updates the connecting weights in a neural net.
2. Structure learning: It focuses on the change in network structure (which
includes the number of processing elements as well as their connection types).
• Learning can be categorized in three category
1. Supervised learning
2. Unsupervised learning
3. Reinforcement learning
#Various Learning Methods
1. Supervised Learning
Supervised learning, also known as supervised machine learning, is a
subcategory of machine learning and artificial intelligence. It is defined by
its use of labeled datasets to train algorithms that to classify data or predict
outcomes accurately.
In supervised learning, models are trained using labelled dataset, where the
model learns about each type of data. Once the training process is completed,
the model is tested on the basis of test data (a subset of the training set), and
then it predicts the output.
The working of Supervised learning can be easily understood by the below
example and diagram:
In this learning method, every input pattern that is used to train the network is
associated with an output pattern, which is a target or desired pattern. A teacher is
assumed to be present during the learning process, where a comparison is made
between networks computed output and desired output, to determine error. The
error is used to change network parameters. It results in an improvement in
performance.
Advantages of Supervised learning:
With the help of supervised learning, the model can predict the output on the basis of
prior experiences.
In supervised learning, we can have an exact idea about the classes of objects.
Supervised learning model helps us to solve various real-world problems such as fraud
detection, spam filtering, etc.
In this method, a teacher through available does not present the expected
answer but only indicate if the computed output is correct or not. The
information is provided to the network in learning process, a reward is given to
the correct answer computed and penalty for the wrong answer computed. It is
not popular form of learning. Supervised and unsupervised learning methods
which are most popular form of learning have formed expression through
various rules.
Comparison Table
Regression and
Type of problems Association and Clustering Exploitation or Exploration
classification
• Where, 𝑊𝑖 is weight of link between Ith input neuron to output neuron Y. Xi is Ith input,
and y is associated output.
• The rule is well suited for bipolar data than binary data. If binary data is used, then above
weight updating formula cannot distinguish, namely two conditions
• 1. A training pair in which an input unit is “on” and target value is “off”.
• 2. A training pair in which an input unit is “on” and target value is “off”.
• Hebb rule is widely used in Pattern classification, Pattern Association, etc
Training Algorithm For Hebbian Learning Rule
• The training steps of the algorithm are as follows:
• Initially, the weights are set to zero, i.e. w =0 for all inputs i =1 to n and n is the
total number of input neurons.
• Let s be the output. The activation function for inputs is generally set as an
identity function.
• The activation function for output is also set to y= t.
• The weight adjustments and bias are adjusted to:
•The steps 2 to 4 are repeated for each input vector and output.
Flowchart of Hebbian network
Associative Memories:
• These type of neural networks work on the basis of pattern
association, which means they can store different patterns and at the
time of giving an output they can produce one of the stored patterns
by matching them with the given input patterns.
• These types of memories are also called content addressable memory
CAM.
• There are two types of associative memory.
1. Auto associative memory.
2. Hetro associative memory.
Auto Associative Memory
• This is a single layer neural network in which the input vector and the
output target vectors are the same.
• An auto-associative memory recovers a previously stored pattern that
most closely relates to the current pattern. It is also known as
an auto-associative correlator.