Soft Computing

Chapter 1:-Introduction to Neural Network

For BTI, Computer Engineering, MPSTME
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models
of neuron, Models of ANN, Neural processing, Neural network learning rules, Single
layer perceptron classifier, Linearly separable classification

• Lotfi Aliasker Zadeh

• Hard Computing:-
• Traditional mathematical model/ algorithm, deterministic and precise, well defined solution,
binary logic and crisp result, sequential computation
• If the Industry /plant is strictly linear, or if PID loop control does an adequate job, If high
speed of operation is required
• Soft Computing:-
• Genetic algorithm, fuzzy logic, neural network, approximation, formal logic and probabilistic
reasoning, approximate result parallel computations.
• practical where the process/plant is complex nonlinear and operator-dependent
Chapter 1 Soft Computing 10/31/2023 2
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models
of neuron, Models of ANN, Neural processing, Neural network learning rules, Single
layer perceptron classifier, Linearly separable classification

• Soft Computing:-

Artificial Neural Network Fuzzy Logic

Genetic Algorithm Probabilistic computing

Chapter 1 Soft Computing 10/31/2023 3
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models
of neuron, Models of ANN, Neural processing, Neural network learning rules, Single
layer perceptron classifier, Linearly separable classification

Characteristics of soft computing :

• It may not yield a precise solution.

• Algorithms are adaptive.

• evolution changes for a specific species like the human nervous system and behavior of an Ant’s,
etc.

• Learning from experimental data.

Chapter 1 Soft Computing 10/31/2023 4

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models
of neuron, Models of ANN, Neural processing, Neural network learning rules, Single
layer perceptron classifier, Linearly separable classification

• Biological Neuron:-
• Axons connect to dendrites via synapses.
• Electro-chemical signals are propagated from
• the dendritic input, through the cell body, and down
the axon to other neurons
• A neuron only fires if its input signal exceeds a
certain amount (the threshold) in a short time period.
• Synapses vary in strength
• Good connections allowing a large signal
• Slight connections allow only a weak signal.
Chapter 1 Soft Computing 10/31/2023 5
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models
of neuron, Models of ANN, Neural processing, Neural network learning rules, Single
layer perceptron classifier, Linearly separable classification

• Artificial Neural Network:-

• parallel computing devices.
• computer model of the brain Biological Neural Artificial Neural
Network Network
• the basic units are neurons
Soma Node
• organized into layers
• Every neuron is connected with other neuron Dendrites Input
through a connection link
• Each connection link is associated with a weight Synapse Weights or
Interconnections
• delay leads and lag-free connections are allowed
Axon Output

Chapter 1 Soft Computing 10/31/2023 6

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models
of neuron, Models of ANN, Neural processing, Neural network learning rules, Single
layer perceptron classifier, Linearly separable classification
• Each neuron consists of three major components:-

• A signal xi forms the input to the i-th synapse

• The i-th synapse has the weight wi
• A summation junction is a linear combiner or
adder it adds weighted input signals

• A threshold activation function results in an

output signal only when an input signal exceeding a
specific threshold value comes as an input

Chapter 1 Soft Computing 10/31/2023 7

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models
of neuron, Models of ANN, Neural processing, Neural network learning rules, Single
layer perceptron classifier, Linearly separable classification

• Basic Models of ANN

Basic Models of
ANN

Activation
Interconnections Learning rules
function

Chapter 1 Soft Computing 10/31/2023 8

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models
of neuron, Models of ANN, Neural processing, Neural network learning rules, Single
layer perceptron classifier, Linearly separable classification

• Types of Activation Function :-

• Identity Function:
• Threshold/step Function:
• ReLU (Rectified Linear Unit) Function:
• Sigmoid Function:
• Binary Sigmoid Function
• Bipolar Sigmoid Function
• Hyperbolic Tangent Function

Chapter 1 Soft Computing 10/31/2023 9

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models
of neuron, Models of ANN, Neural processing, Neural network learning rules, Single
layer perceptron classifier, Linearly separable classification

• five basic types of neuron connection architectures :-

• Single layer feed forward network.

• Multilayer feedforward network

• Single node with its own feedback

• Single layer recurrent network

• Multilayer recurrent network

Chapter 1 Soft Computing 10/31/2023 10

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models
of neuron, Models of ANN, Neural processing, Neural network learning rules, Single
layer perceptron classifier, Linearly separable classification

• Types of learning :-
• Supervised Learning:-

• Unsupervised Learning:-

• Reinforcement Learning:-

Chapter 1 Soft Computing 10/31/2023 11

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

• Supervised: • Unsupervised
• Adaline, Madaline • Competitive Learning
• Perceptron • Kohenen self organizing map
• Back Propagation • Learning vector quantization
• multilayer perceptron • Hebbian learning
• Radial Basis Function Networks • Reinforced
• Positive
• Negative

Chapter 1 Soft Computing 10/31/2023 12

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models
of neuron, Models of ANN, Neural processing, Neural network learning rules, Single
layer perceptron classifier, Linearly separable classification

Chapter 1 Soft Computing 10/31/2023 13

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models
of neuron, Models of ANN, Neural processing, Neural network learning rules, Single
layer perceptron classifier, Linearly separable classification

McCulloch-Pitts Model of Neuron

• earliest ANN model

• has only two types of inputs —
• Excitatory (+ve Magnitude) and
• Inhibitory (-ve mag)
• The inputs of the McCulloch-Pitts neuron could be
either 0 or 1.
• the output signal yout is 1 if the input ysum is greater than
or equal to a given threshold value
Chapter 1 Soft Computing 10/31/2023 14
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models
of neuron, Models of ANN, Neural processing, Neural network learning rules, Single
layer perceptron classifier, Linearly separable classification

McCulloch-Pitts Model of Neuron

x1
X1

y = f ( yin )
w1

Y y

x2
w2
X2 yin = x1w1 + x2 w2

Chapter 1 Soft Computing 10/31/2023 15

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models
of neuron, Models of ANN, Neural processing, Neural network learning rules, Single
layer perceptron classifier, Linearly separable classification

McCulloch-Pitts Model of Neuron

Chapter 1 Soft Computing 10/31/2023 16

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models
of neuron, Models of ANN, Neural processing, Neural network learning rules, Single
layer perceptron classifier, Linearly separable classification

McCulloch-Pitts Model of Neuron

Chapter 1 Soft Computing 10/31/2023 17

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models
of neuron, Models of ANN, Neural processing, Neural network learning rules, Single
layer perceptron classifier, Linearly separable classification

McCulloch-Pitts Model of Neuron

Characteristics
• Allows binary 0,1 states only
• Operates under a discrete-time assumption
• Weights and the neurons’ thresholds are fixed and no interaction among network
neurons
• Just a primitive model

Advantages Disadvantages
Simplistic Weights and thresholds are fixed
Substantial computing power Not very flexible
Chapter 1 Soft Computing 10/31/2023 18
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

• Supervised:
• Adaline, Madaline
• Perceptron
• Back Propagation
• multilayer perceptron
• Radial Basis Function Networks
• Unsupervised
• Competitive Learning
• Kohenen self organizing map
• Learning vector quantization
• Hebbian learning

Chapter 1 Soft Computing 10/31/2023 19

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Neural processing
• refers to the way the brain works
• the ability to adapt to changing situations and
• to improve its function as more information becomes available.
• Recall , retrieval, associate and relearn are basic process
• Auto association
• Hetero association
• Classification

Chapter 1 Soft Computing 10/31/2023 20

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Auto association

• Set of patterns can be stored in the network

• If a pattern similar to a member of the stored
set is presented, an association with the input
of closest stored pattern is made
• a single layer neural network.
• Input training vector and the output target
vectors are the same

Chapter 1 Soft Computing 10/31/2023 21

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Auto association

• has ‘n’ number of input training

vectors and output target vectors

Chapter 1 Soft Computing 10/31/2023 22

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Hetero association

• Associations between pairs of

patterns are stored
• Distorted input pattern may cause
correct heteroassociation at the
output

Chapter 1 Soft Computing 10/31/2023 23

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Auto association

• has ‘n’ number of input training

vectors and ‘m’ number output
target vectors

Chapter 1 Soft Computing 10/31/2023 24

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Auto association

• has ‘n’ number of input training

vectors and ‘m’ number output
target vectors

Chapter 1 Soft Computing 10/31/2023 25

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Important terminologies of ANNs

• Weights
• Bias
• Threshold
• Learning rate
• Momentum factor
• Vigilance parameter
• Notations used in ANN
Chapter 1 Soft Computing 10/31/2023 26
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Important terminologies of ANNs

• Weights • Each neuron is connected to every other neuron by

• Bias means of directed links

• Threshold • Links are associated with weights

• Learning rate
• Momentum factor
• Vigilance parameter
• Notations used in ANN
Chapter 1 Soft Computing
• Weights contain information about the input signal
and is represented as a matrix
• Weight matrix also called connection matrix
10/31/2023 27
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Important terminologies of ANNs

 T
• Weights  w1   w11w12 w13...w1m 
 wT 
• Bias  2  
 wT 
 3
 w21w22 w23...w 2 m 
• Threshold .  .................. 
. 
• Learning rate W=   =  
. 
 
................... 
• Momentum factor .   
• Vigilance parameter
. 
 T  wn1wn 2 wn 3...wnm 
 wn 
• Notations used in ANN  

Chapter 1 Soft Computing 10/31/2023 28

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Important terminologies of ANNs

n
• Weights
1
y =x w
inj i ij
i =0
• Bias X1
bj = x 0 w0 j + x1w1 j + x 2 w2 j + .... + x n wnj
• Threshold w n
Xi 1j Yj = w0 j +  x i wij
• Learning rate w i =1
n
• Momentum factor y =b +x w
ij
Xn w inj j i ij
i =1
• Vigilance parameter nj

• Notations used in ANN

Chapter 1 Soft Computing 10/31/2023 29
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Important terminologies of ANNs

• Weights • Bias is like another weight. Its included by adding a

• Bias component x0=1 to the input vector X.
• Threshold • X=(1,X1,X2…Xi,…Xn)
• Learning rate • Bias is of two types
• Momentum factor • Positive bias: increase the net input
• Vigilance parameter • Negative bias: decrease the net input
• Notations used in ANN
Chapter 1 Soft Computing 10/31/2023 30
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Important terminologies of ANNs

Why Bias is required?

• Weights The relationship between input and output given by
• Bias the equation of straight line y=mx+c

• Threshold C(bias)
• Learning rate
• Momentum factor
Input
• Vigilance parameter X Y y=mx+C

• Notations used in ANN

Chapter 1 Soft Computing 10/31/2023 31
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Important terminologies of ANNs

• Weights
• One of the transfer function /activation in ANN
• Set value based upon which the final output of
• Bias
the network may be calculated
• Threshold • The activation function using threshold can be
• Learning rate defined as
• Momentum factor
1ifnet   
• Vigilance parameter f (net ) =  
• Notations used in ANN − 1ifnet   
Chapter 1 Soft Computing 10/31/2023 32
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Important terminologies of ANNs

• Weights • Denoted by α.
• Bias
• Used to control the amount of weight adjustment at each
• Threshold step of training
• Learning rate
• Learning rate ranging from 0 to 1 determines the rate of
• Momentum factor
learning in each time step
• Vigilance parameter
• Notations used in ANN
Chapter 1 Soft Computing 10/31/2023 33
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Important terminologies of ANNs

• Weights • Momentum factor:

• Bias • used for convergence when momentum factor is added to weight
• Threshold updation process.
• Learning rate • Vigilance parameter:
• Momentum factor • Denoted by ρ
• Vigilance parameter • Used to control the degree of similarity required for patterns to be
• Notations used in ANN assigned to the same cluster
Chapter 1 Soft Computing 10/31/2023 34
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Single layer perceptron classifier

• Frank Rosenblatt first proposed in 1958
• is used to classify its input into one or two categories.
• is a single processing unit of any neural network.
• is used in supervised learning
• also known as Linear Binary Classifier.
• uses the step function.

Chapter 1 Soft Computing 10/31/2023 35

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Single layer perceptron classifier – Training

• weights are initialized with random values.
• For each element of the training set,
• the error is calculated with the difference between desired output and the
actual output.
• error calculated is used to adjust the weights.

• The process is repeated

• until the error made on the entire training set is not less than the specified
threshold,
• until the maximum number of iterations is reached.
• Δw = η * d * x
• d: Predicted Output – Desired Output

• η: Learning Rate, Usually Less than 1.

• x: Input Data.
Chapter 1 Soft Computing 10/31/2023 36
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Linearly separable classification

• applies to binary classification problems.
• A decision boundary separates the positive objects from the
negative ones
• such a decision boundary is a linear function of the features
• If linear straight line separate then it is linearly separable 2 D data
• There could be many such lines.

Chapter 1 Soft Computing 10/31/2023 37

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Linearly separable classification

• One dimension
• If you choose two different numbers, you can always
find another number between them. This number
"separates" the two numbers you chose
• if both numbers are the same, you simply cannot
separate them.

• Two dimension
• to separate the point (1,1) from the other points. You can
see that there exists a line that does this. In fact, there
exist infinite such lines. So, these two "classes" of points
are linearly separable.
• In this case, you just cannot use one single line to separate
the two classes . So, they are linearly inseparable.

. Chapter 1 Soft Computing 10/31/2023 38

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Linearly separable classification

• Three dimension
• The dashed plane separates the red point from the other blue points.
So its linearly separable. If bottom right point on the opposite side was
red too, it would become linearly inseparable ..

• n dimension
• Things go up to a lot of dimensions in neural networks. So to separate classes in
n-dimensions, you need an n-1 dimensional "hyperplane".

Chapter 1 Soft Computing 10/31/2023 39

Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Questions for Practice

.
• 1. What are different basic models of Neural Network (5) Co 1So1 BL1
• 2. Discuss different types of activation functions in neural network
(5) Co2 So6 BL2
• 3. Implement AND NOT using MCP neural model (use binary data)
(10) Co 2So 2BL.4
• 4. Classify the two dimensional pattern shown below using
Perceptron network symbol @ indicate +1 and # indicate -110/31/2023
Chapter 1 Soft Computing
ie I and40
F (10) Co 2So 2BL3
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Questions for Practice

.
• 1. State different Learning Rules in neural networks (3) Co 1 SL 1 BL
3
• 2. Explain any two activation functions of Neural Network (7) Co 2
So 1 BL 3
• 3. Differentiate ANN with BNN (7) Co 1 SO 1 BL 2
• 4. Explain auto associative network with diagram (7) Co 2 So 1 BL3
Chapter 1 Soft Computing 10/31/2023 41
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Questions for Practice

.
• 1. What are the characteristics of Neural Network (3)
• 2. Compare between ANN and BNN (4)
• 3. Draw an ADALINE network and give its training algorithm (7)
• 4. Draw architecture of Auto Associative network and give it's training algorithm
(7)
• 5. Explain Rosenblatts Perceptron and explain (7)
• 6. Write short notes on Kohenen's self organising map (4)
• 7. Implement NAND using 2 input and 1 output neuron (4)
• 8. Hamming network and its testing algorithm (3)
Chapter 1 Soft Computing 10/31/2023 42
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Questions for Practice

.
• 1. Explain biological neuron with diagram
• 2. Explain Supervised and unsupervised learning
• 3. Explain winner take all and Out Star rule
• 4. Explain Hebbs training algorithm with flowchart
• 5. Explain learning factors for back propagation network
• 6. Explain architecture of discrete BAM and give algorithm
• 7. Implement AND using Mc-Culloth Pitts model(All are of 6 marks )
Chapter 1 Soft Computing 10/31/2023 43
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Questions for Practice

.
• 1. State True or false, Ex-Or is a linearly separable logic (3) CO2 BL 2
• 2. Describe Hamming net (3) CO3BL2.
• 3. Describe Hopfield Neural Network with its architecture and training
algorithm (7) CO 3 BL 2
• 4. What is the activation function, explain different activation function with
corresponding graph and equation (7)CO 1 BL 3
• 5. State the Backpropation Training algorithm (7) CO 2 BL 1
• 6. Discuss the Hebbian learning rule along with its limitations (7) CO 2 BL 2
• 7. Realize AND function using Mc-Culloth Pitts model (7) CO 1 BL10/31/2023
Chapter 1 Soft Computing
6 44
Introduction to Neural Network- Biological Neurons, McCullah and Pitts models of
neuron, Models of ANN, Neural processing, Neural network learning rules, Single layer
perceptron classifier, Linearly separable classification

Thank You Chapter 1

.

Chapter 1 Soft Computing 10/31/2023 45