UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

Concept of computing :
According to the concept of computing, the input is called an antecedent and the output is called
the consequent. For example, Adding information in DataBase, Compute the sum of two
numbers using a C program, etc.
 Hard computing is best suited for solving mathematical problems which give some
precise answers.
 Some important fields like Biology, Medicine and humanities, etc are still intractable
using Convention mathematical and Analytical models.

There are two types of computing as following :  It is possible to map the human mind with the help of Soft computing but it is not
possible with Convention mathematical and Analytical models.

1. Hard computing Examples –

Consider a problem where a string w1 is “abc” and string w2 is “abd”.
2. soft computing

Characteristics of hard computing :  Problem-1 :

Tell that whether w1 is the same as w2 or not?

 The precise result is guaranteed. Solution –

The answer is simply No, it means there is an algorithm by which we can analyze it.
 The control action is unambiguous.

 The control action is formally defined (i.e. with a mathematical model)

 Problem-2 :
Now, the question arises that if we have hard computing then why do we require the need for Tell how much these two strings are similar?
soft computing.
Solution –
Characteristics of soft computing : The answer from conventional computing is either YES or NO. But these maybe 80%
similar, this can be answered only by Soft Computing.

 It may not yield a precise solution.

Recent development in Soft Computing :
 Algorithms are adaptive.

 In soft computing, you can consider an example where you can see the evolution changes 1. In the field of Big Data, soft computing working for data analyzing models, data behavior
for a specific species like the human nervous system and behavior of an Ant’s, etc. models, data decision, etc.

 Learning from experimental data. 2. In case of Recommender system, soft computing plays an important role for analyzing
the problem on the based of algorithm and works for precise results.
Need For Soft Computing :
3. In Behavior and decision science, soft computing used in this for analyzing the behavior,
and model of soft computing works accordingly.
 Many analytical models are valid for ideal cases. Real-world problems exist in a non-
ideal environment. 4. In the fields of Mechanical Engineering, soft computing is a role model for computing
problems such that how a machine will works and how it will make the decision for a
 Soft computing provides insights into real-world problems and is just not limited to specific problem or input given.
theory.
5. In this field of Computer Engineering, you can say it is core part of soft computing and
computing working on advanced level like Machine learning, Artificial intelligence, etc.
 UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

Let's consider a problem that actually does not have any solution via traditional computing, but
What is soft computing soft computing gives the approximate solution.

Soft computing is the reverse of hard (conventional) computing. It refers to a group of string1 = "xyz" and string2 = "xyw"
computational techniques that are based on artificial intelligence (AI) and natural selection. It
provides cost-effective solutions to the complex real-life problems for which hard computing 1. Problem 1
solution does not exist. 2. Are string1 and string2 same?
3. Solution
Zadeh coined the term of soft computing in 1992. The objective of soft computing is to provide 4. No, the solution is simply No. It does not require any algorithm to analyze this.
precise approximation and quick solutions for complex real-life problems.
Let's modify the problem a bit.

1. Problem 2
2. How much string1 and string2 are same?
3. Solution
4. Through conventional programming, either the answer is Yes or No. But these strings mi
ght be 80% similar according to soft computing.

You have noticed that soft computing gave us the approximate solution.

Applications of soft computing

There are several applications of soft computing where it is used. Some of them are listed below:

In simple terms, you can understand soft computing - an emerging approach that gives the
amazing ability of the human mind. It can map a human mind and the human mind is a role
model for soft computing.
Note: Basically, soft computing is different from traditional/conventional computing and it
deals with approximation models.
Some characteristics of Soft computing
 Soft computing provides an approximate but precise solution for real-life problems.
 The algorithms of soft computing are adaptive, so the current process is not affected by
any kind of change in the environment.
 The concept of soft computing is based on learning from experimental data. It means
that soft computing does not require any mathematical model to solve the problem.
 Soft computing helps users to solve real-world problems by providing approximate
results that conventional and analytical models cannot solve.
 It is based on Fuzzy logic, genetic algorithms, machine learning, ANN, and expert
systems.
Example
Soft computing deals with the approximation model. Yoi will understand with the help of
examples of how it deals with the approximation model.
 It is widely used in gaming products like Poker and Checker.
 In kitchen appliances, such as Microwave and Rice cooker.
 In most used home appliances - Washing Machine, Heater, Refrigerator, and AC as
well.
 Apart from all these usages, it is also used in Robotics work (Emotional per Robot
form).
 Image processing and Data compression are also popular applications of soft
computing.
 Used for handwriting recognition.
As we already said that, soft computing provides the solution to real-time problems and here you
can see that. Besides these applications, there are many other applications of soft computing.
Need of soft computing
Sometimes, conventional computing or analytical models does not provide a solution to some
real-world problems. In that case, we require other technique like soft computing to obtain an
approximate solution.
 Hard computing is used for solving mathematical problems that need a precise answer. It
fails to provide solutions for some real-life problems. Thereby for real-life problems
whose precise solution does not exist, soft computing helps.
 UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

 When conventional mathematical and analytical models fail, soft computing helps, e.g.,
You can map even the human mind using soft computing.
 Analytical models can be used for solving mathematical problems and valid for ideal
cases. But the real-world problems do not have an ideal case; these exist in a non-ideal
environment.
 Soft computing is not only limited to theory; it also gives insights into real-life problems.
 Like all the above reasons, Soft computing helps to map the human mind, which cannot
be possible with conventional mathematical and analytical models.
Elements of soft computing
Soft computing is viewed as a foundation component for an emerging field of conceptual
intelligence. Fuzzy Logic (FL), Machine Learning (ML), Neural Network (NN), Probabilistic
Reasoning (PR), and Evolutionary Computation (EC) are the supplements of soft computing.
Also, these are techniques used by soft computing to resolve any complex problem.
Neural Network (ANN)
Neural networks were developed in the 1950s, which helped soft computing to solve real-world
problems, which a computer cannot do itself. We all know that a human brain can easily describe
real-world conditions, but a computer cannot.
An artificial neural network (ANN) emulates a network of neurons that makes a human brain
(means a machine that can think like a human mind). Thereby the computer or a machine can
learn things so that they can take decisions like the human brain.
Artificial Neural Networks (ANN) are mutually connected with brain cells and created using
regular computing programming. It is like as the human neural system.
Genetic Algorithms (GA)
Genetic algorithm is almost based on nature and take all inspirations from it. There is no genetic
algorithm that is based on search-based algorithms, which find its roots in natural selection and
the concept of genetics.

In addition, a genetic algorithm is a subset of a large branch of computation.

Soft computing vs hard computing

Hard computing uses existing mathematical algorithms to solve certain problems. It provides a
precise and exact solution of the problem. Any numerical problem is an example of hard
computing.

On the other hand, soft computing is a different approach than hard computing. In soft
computing, we compute solutions to the existing complex problems. The result calculated or

Any problems can be resolved effectively using these components. Following are three types of Parameters Soft Computing Hard Computing
techniques used by soft computing: Computation
Takes less computation time. Takes more computation time.
time
 Fuzzy Logic
 Artificial Neural Network (ANN) It depends on approximation and It is mainly based on binary logic and
Dependency
 Genetic Algorithms dispositional. numerical systems.
Computation
Parallel computation Sequential computation
Fuzzy Logic (FL) type
Result/Output Approximate result Exact and precise result
Fuzzy logic is nothing but mathematical logic which tries to solve problems with an open and Any numerical problem or traditional
imprecise spectrum of data. It makes it easy to obtain an array of precise conclusions. Neural Networks, such as
Example methods of solving using personal
Madaline, Adaline, Art Networks.
computers.
Fuzzy logic is basically designed to achieve the best possible solution to complex problems from provided by soft computing are also not precise. They are imprecise and fuzzy in nature.
all the available information and input data. Fuzzy logics are considered as the best solution
finders.
 UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

 1961 − Rosenblatt made an unsuccessful attempt but proposed the “backpropagation”

scheme for multilayer networks.
 1964 − Taylor constructed a winner-take-all circuit with inhibitions among output units.
 1969 − Multilayer perceptron (MLP) was invented by Minsky and Papert.
 1971 − Kohonen developed Associative memories.
 1976 − Stephen Grossberg and Gail Carpenter developed Adaptive resonance theory.

ANN from 1980s till Present

Some key developments of this era are as follows −

 1982 − The major development was Hopfield’s Energy approach.

 1985 − Boltzmann machine was developed by Ackley, Hinton, and Sejnowski.
 1986 − Rumelhart, Hinton, and Williams introduced Generalised Delta Rule.
 1988 − Kosko developed Binary Associative Memory (BAM) and also gave the concept
of Fuzzy Logic in ANN.

The historical review shows that significant progress has been made in this field. Neural network
based chips are emerging and applications to complex problems are being developed. Surely,
today is a period of transition for neural network technology.

Neural Network Biological Neuron

A Brief History of ANN A nerve cell (neuron) is a special biological cell that processes information. According to an
estimation, there are huge number of neurons, approximately 10 11 with numerous
The history of ANN can be divided into the following three eras − interconnections, approximately 1015.

ANN during 1940s to 1960s Schematic Diagram

Some key developments of this era are as follows −

 1943 − It has been assumed that the concept of neural network started with the work of
physiologist, Warren McCulloch, and mathematician, Walter Pitts, when in 1943 they
modeled a simple neural network using electrical circuits in order to describe how
neurons in the brain might work.
 1949 − Donald Hebb’s book, The Organization of Behavior, put forth the fact that
repeated activation of one neuron by another increases its strength each time they are
used.
 1956 − An associative memory network was introduced by Taylor.
 1958 − A learning method for McCulloch and Pitts neuron model named Perceptron was
invented by Rosenblatt.
 1960 − Bernard Widrow and Marcian Hoff developed models called "ADALINE" and
“MADALINE.”
Working of a Biological Neuron
ANN during 1960s to 1980s
As shown in the above diagram, a typical neuron consists of the following four parts with the
Some key developments of this era are as follows − help of which we can explain its working −
 UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

 Dendrites − They are tree-like branches, responsible for receiving the information from
other neurons it is connected to. In other sense, we can say that they are like the ears of
neuron.
 Soma − It is the cell body of the neuron and is responsible for processing of information,
they have received from dendrites.
 Axon − It is just like a cable through which neurons send the information.
 Synapses − It is the connection between the axon and other neuron dendrites.

ANN versus BNN

Before taking a look at the differences between Artificial Neural Network (ANN) and Biological
Neural Network (BNN), let us take a look at the similarities based on the terminology between
these two. Artificial Neural Network Tutorial provides basic and advanced concepts of ANNs. Our
Artificial Neural Network tutorial is developed for beginners as well as professions.
Biological Neural Network (BNN) Artificial Neural Network (ANN)
The term "Artificial neural network" refers to a biologically inspired sub-field of artificial
Soma Node intelligence modeled after the brain. An Artificial neural network is usually a computational
Dendrites Input network based on biological neural networks that construct the structure of the human brain.
Synapse Weights or Interconnections Similar to a human brain has neurons interconnected to each other, artificial neural networks also
Axon Output have neurons that are linked to each other in various layers of the networks. These neurons are
known as nodes.
The following table shows the comparison between ANN and BNN based on some criteria
Artificial neural network tutorial covers all the aspects related to the artificial neural network. In
mentioned.
this tutorial, we will discuss ANNs, Adaptive resonance theory, Kohonen self-organizing map,
Building blocks, unsupervised learning, Genetic algorithm, etc.
Criteria BNN ANN
Massively parallel, slow but
Processing
superior than ANN
Massively parallel, fast but inferior than BNN What is Artificial Neural Network?
1011 neurons and 1015 102 to 104 nodes (mainly depends on the type of
Size The term "Artificial Neural Network" is derived from Biological neural networks that develop
interconnections application and network designer)
the structure of a human brain. Similar to the human brain that has neurons interconnected to one
Very precise, structured and formatted data is another, artificial neural networks also have neurons that are interconnected to one another in
Learning They can tolerate ambiguity
required to tolerate ambiguity various layers of the networks. These neurons are known as nodes.
Fault Performance degrades with even It is capable of robust performance, hence has
tolerance partial damage the potential to be fault tolerant Video Player is loading.
Storage Stores the information in the Stores the information in continuous memory Current Time 0:03
capacity synapse locations /
Duration 4:57
Â
 UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

An Artificial Neural Network in the field of Artificial intelligence where it attempts to mimic
the network of neurons makes up a human brain so that computers will have an option to
understand things and make decisions in a human-like manner. The artificial neural network is
designed by programming computers to behave simply like interconnected brain cells.

There are around 1000 billion neurons in the human brain. Each neuron has an association point
somewhere in the range of 1,000 and 100,000. In the human brain, data is stored in such a
manner as to be distributed, and we can extract more than one piece of this data when necessary
from our memory parallelly. We can say that the human brain is made up of incredibly amazing
parallel processors.

We can understand the artificial neural network with an example, consider an example of a
digital logic gate that takes an input and gives an output. "OR" gate, which takes two inputs. If
one or both the inputs are "On," then we get "On" in output. If both the inputs are "Off," then we
get "Off" in output. Here the output depends upon input. Our brain does not perform the same
task. The outputs to inputs relationship keep changing because of the neurons in our brain, which
are "learning."
The given figure illustrates the typical diagram of Biological Neural Network.
The architecture of an artificial neural network:
The typical Artificial Neural Network looks something like the given figure.
To understand the concept of the architecture of an artificial neural network, we have to
understand what a neural network consists of. In order to define a neural network that consists of
a large number of artificial neurons, which are termed units arranged in a sequence of layers.
Lets us look at various types of layers available in an artificial neural network.

Artificial Neural Network primarily consists of three layers:

Dendrites from Biological Neural Network represent inputs in Artificial Neural Networks, cell
nucleus represents Nodes, synapse represents Weights, and Axon represents Output.

Relationship between Biological neural network and artificial neural network:

Biological Neural Network Artificial Neural Network

Dendrites Inputs
Cell nucleus Nodes
Synapse Weights
Axon Output Input Layer:

As the name suggests, it accepts inputs in several different formats provided by the programmer.
 UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

Hidden Layer: Having fault tolerance:

The hidden layer presents in-between input and output layers. It performs all the calculations to Extortion of one or more cells of ANN does not prohibit it from generating output, and this
find hidden features and patterns. feature makes the network fault-tolerance.

Output Layer:

The input goes through a series of transformations using the hidden layer, which finally results in
output that is conveyed using this layer.
The artificial neural network takes input and computes the weighted sum of the inputs and
includes a bias. This computation is represented in the form of a transfer function.
Difference between ANN and BNN
Do you ever think of what it’s like to build anything like a brain, how these things work, or what
do they do? Let us look at how nodes communicate with neurons and what are some differences
between artificial and biological neural networks.

It determines weighted total is passed as an input to an activation function to produce the output.
Activation functions choose whether a node should fire or not. Only those who are fired make it
to the output layer. There are distinctive activation functions available that can be applied upon
the sort of task we are performing.
Advantages of Artificial Neural Network (ANN)
Parallel processing capability:
1. Artificial Neural Network : Artificial Neural Network (ANN) is a type of neural network
which is based on a Feed-Forward strategy. It is called this because they pass information
through the nodes continuously till it reaches the output node. This is also known as the simplest
type of neural network. Some advantages of ANN :
 Ability to learn irrespective of the type of data (Linear or Non-Linear).
 ANN is highly volatile and serves best in financial time series forecasting.
Some disadvantages of ANN :
 The simplest architecture makes it difficult to explain the behavior of the network.
 This network is dependent on hardware.

Artificial neural networks have a numerical value that can perform more than one task 2. Biological Neural Network : Biological Neural Network (BNN) is a structure that consists of
simultaneously. Synapse, dendrites, cell body, and axon. In this neural network, the processing is carried out by
neurons. Dendrites receive signals from other neurons, Soma sums all the incoming signals and
Storing data on the entire network: axon transmits the signals to other cells. Some advantages of BNN :

Data that is used in traditional programming is stored on the whole network, not on a database.  The synapses are the input processing element.
The disappearance of a couple of pieces of data in one place doesn't prevent the network from  It is able to process highly complex parallel inputs.
working.
Some disadvantages of BNN :
Capability to work with incomplete knowledge:
 There is no controlling mechanism.
After ANN training, the information may produce output even with inadequate data. The loss of  Speed of processing is slow being it complex.
performance here relies upon the significance of missing data.
Differences between ANN and BNN :
Having a memory distribution:
Parameters ANN BNN
For ANN is to be able to adapt, it is important to determine the examples and to encourage the
input dendrites
network according to the desired output by demonstrating these examples to the network. The
succession of the network is directly proportional to the chosen instances, and if the event can't Structure
weight synapse
appear to the network in all its aspects, it can produce false output.
 UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

output axon Difficulty of showing the issue to the network:

hidden layer cell body ANNs can work with numerical data. Problems must be converted into numerical values before
very precise structures and being introduced to ANN. The presentation mechanism to be resolved here will directly impact
Learning they can tolerate ambiguity the performance of the network. It relies on the user's abilities.
formatted data
complex simple
The duration of the network is unknown:
Processor high speed low speed
The network is reduced to a specific value of the error, and this value does not give us optimum
results.
one or a few large number
separate from a processor integrated into processor Science artificial neural networks that have steeped into the world in the mid-20th century are
exponentially developing. In the present time, we have investigated the pros of artificial neural
Memory localized distributed networks and the issues encountered in the course of their utilization. It should not be overlooked
that the cons of ANN networks, which are a flourishing science branch, are eliminated
non-content addressable content-addressable individually, and their pros are increasing day by day. It means that artificial neural networks
centralized distributed will turn into an irreplaceable part of our lives progressively important.

Computing sequential parallel How do artificial neural networks work?

stored programs self-learning
Artificial Neural Network can be best represented as a weighted directed graph, where the
Reliability very vulnerable robust artificial neurons form the nodes. The association between the neurons outputs and neuron inputs
numerical and symbolic perceptual can be viewed as the directed edges with weights. The Artificial Neural Network receives the
Expertise input signal from the external source in the form of a pattern and image in the form of a vector.
manipulations problems These inputs are then mathematically assigned by the notations x(n) for every n number of
well-defined poorly defined inputs.
Operating
Environment
well-constrained un-constrained
performance degraded even on partial
Fault Tolerance the potential of fault tolerance
damage

Disadvantages of Artificial Neural Network:

Assurance of proper network structure:

There is no particular guideline for determining the structure of artificial neural networks. The
appropriate network structure is accomplished through experience, trial, and error.

Unrecognized behavior of the network:

It is the most significant issue of ANN. When ANN produces a testing solution, it does not
provide insight concerning why and how. It decreases trust in the network.

Hardware dependence:
Afterward, each of the input is multiplied by its corresponding weights ( these weights are the
Artificial neural networks need processors with parallel processing power, as per their structure. details utilized by the artificial neural networks to solve a specific problem ). In general terms,
Therefore, the realization of the equipment is dependent.
 UNIT-1 SOFT COMPUTING
these weights normally represent the strength of the interconnection between neurons inside the
artificial neural network. All the weighted inputs are summarized inside the computing unit.
If the weighted sum is equal to zero, then bias is added to make the output non-zero or something
else to scale up to the system's response. Bias has the same input, and weight equals to 1. Here
the total of weighted inputs can be in the range of 0 to positive infinity. Here, to keep the
response in the limits of the desired value, a certain maximum value is benchmarked, and the
total of weighted inputs is passed through the activation function.
The activation function refers to the set of transfer functions used to achieve the desired output.
There is a different kind of the activation function, but primarily either linear or non-linear sets
of functions. Some of the commonly used sets of activation functions are the Binary, linear, and
Tan hyperbolic sigmoidal activation functions. Let us take a look at each of them in details:
Binary:
In binary activation function, the output is either a one or a 0. Here, to accomplish this, there is a
threshold value set up. If the net weighted input of neurons is more than 1, then the final output
of the activation function is returned as one or else the output is returned as 0.
Sigmoidal Hyperbolic:
The Sigmoidal Hyperbola function is generally seen as an "S" shaped curve. Here the tan
hyperbolic function is used to approximate output from the actual net input. The function is
defined as:
F(x) = (1/1 + exp(-????x))
Where ???? is considered the Steepness parameter.
Types of Artificial Neural Network:
There are various types of Artificial Neural Networks (ANN) depending upon the human brain
neuron and network functions, an artificial neural network similarly performs tasks. The majority
of the artificial neural networks will have some similarities with a more complex biological
partner and are very effective at their expected tasks. For example, segmentation or
classification.
Feedback ANN:
In this type of ANN, the output returns into the network to accomplish the best-evolved results
internally. As per the University of Massachusetts, Lowell Centre for Atmospheric Research.
The feedback networks feed information back into itself and are well suited to solve optimization
issues. The Internal system error corrections utilize feedback ANNs.
UNIT-1 SOFT COMPUTING
Feed-Forward ANN:
A feed-forward network is a basic neural network comprising of an input layer, an output layer,
and at least one layer of a neuron. Through assessment of its output by reviewing its input, the
intensity of the network can be noticed based on group behavior of the associated neurons, and
the output is decided. The primary advantage of this network is that it figures out how to evaluate
and recognize input patterns.
Prerequisite
No specific expertise is needed as a prerequisite before starting this tutorial.
Audience
Our Artificial Neural Network Tutorial is developed for beginners as well as professionals, to
help them understand the basic concept of ANNs.
Problems
We assure you that you will not find any problem in this Artificial Neural Network tutorial. But
if there is any problem or mistake, please post the problem in the contact form so that we can
further improve it.
Characteristics of Artificial Neural Network
 It is neurally implemented mathematical model
 It contains huge number of interconnected processing elements called neurons to do all
operations
 Information stored in the neurons are basically the weighted linkage of neurons
 The input signals arrive at the processing elements through connections and connecting
weights.
 It has the ability to learn , recall and generalize from the given data by suitable
assignment and adjustment of weights.
 The collective behavior of the neurons describes its computational power, and no single
neuron carries specific information .
How simple neuron works ?
Let there are two neurons X and Y which is transmitting signal to another neuron Z . Then , X
and Y are input neurons for transmitting signals and Z is output neuron for receiving signal . The
input neurons are connected to the output neuron , over a interconnection links ( A and B ) as
 UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

shown in figure . 6. By analyzing data and determining which of the data has any fault ( files diverging from peers
) called as Data mining, cleaning and validation can be achieved through neural network.

7. Neural Network can be used to predict targets with the help of echo patterns we get from
sonar, radar, seismic and magnetic instruments.

8. It can be used efficiently in Employee hiring so that any company can hire the right employee
depending upon the skills the employee has and what should be its productivity in future.

9. It has a large application in Medical Research .

10. It can be used to for Fraud Detection regarding credit cards , insurance or taxes by analyzing
the past records .

Artificial Neural Network - Building Blocks

Processing of ANN depends upon the following three building blocks −

For above neuron architecture, the net input has to be calculated in the way.
I = xA + yB
where x and y are the activations of the input neurons X and Y. The output z of the output
neuron Z can be obtained by applying activations over the net input.
O = f(I)
Output = Function ( net input calculated )
The function to be applied over the net input is called activation function . There are various
activation function possible for this.
Application of Neural Network
1. Every new technology need assistance from the previous one i.e. data from previous ones and
these data are analyzed so that every pros and cons should be studied correctly. All of these
things are possible only through the help of neural network.
 Network Topology
 Adjustments of Weights or Learning
 Activation Functions
In this chapter, we will discuss in detail about these three building blocks of ANN
Network Topology
A network topology is the arrangement of a network along with its nodes and connecting lines.
According to the topology, ANN can be classified as the following kinds −
Feedforward Network
It is a non-recurrent network having processing units/nodes in layers and all the nodes in a layer
are connected with the nodes of the previous layers. The connection has different weights upon
them. There is no feedback loop means the signal can only flow in one direction, from input to
output. It may be divided into the following two types −

2. Neural network is suitable for the research on Animal behavior, predator/prey relationships
and population cycles .
3. It would be easier to do proper valuation of property, buildings, automobiles, machinery etc.
with the help of neural network.
 Single layer feedforward network − The concept is of feedforward ANN having only
one weighted layer. In other words, we can say the input layer is fully connected to the
output layer.

4. Neural Network can be used in betting on horse races, sporting events, and most importantly
in stock market.

5. It can be used to predict the correct judgment for any crime by using a large data of crime
details as input and the resulting sentences as output.
 UNIT-1 SOFT COMPUTING
 Multilayer feedforward network − The concept is of feedforward ANN having more
than one weighted layer. As this network has one or more layers between the input and
the output layer, it is called hidden layers.
Feedback Network
As the name suggests, a feedback network has feedback paths, which means the signal can flow
in both directions using loops. This makes it a non-linear dynamic system, which changes
continuously until it reaches a state of equilibrium. It may be divided into the following types −
 Recurrent networks − They are feedback networks with closed loops. Following are the
two types of recurrent networks.
 Fully recurrent network − It is the simplest neural network architecture because all
nodes are connected to all other nodes and each node works as both input and output.
 Jordan network − It is a closed loop network in which the output will go to the input
again as feedback as shown in the following diagram.
UNIT-1 SOFT COMPUTING
Adjustments of Weights or Learning
Learning, in artificial neural network, is the method of modifying the weights of connections
between the neurons of a specified network. Learning in ANN can be classified into three
categories namely supervised learning, unsupervised learning, and reinforcement learning.
Supervised Learning
As the name suggests, this type of learning is done under the supervision of a teacher. This
learning process is dependent.
During the training of ANN under supervised learning, the input vector is presented to the
network, which will give an output vector. This output vector is compared with the desired
output vector. An error signal is generated, if there is a difference between the actual output and
the desired output vector. On the basis of this error signal, the weights are adjusted until the
actual output is matched with the desired output.
 UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

Unsupervised Learning Training of ANN

As the name suggests, this type of learning is done without the supervision of a teacher. This The term “artificial neural network” (ANN) refers to a hardware or software system in
learning process is independent. information technology (IT) that copies the functioning of neurons in the human brain. A class of
deep learning technology, ANNs (also known as neural networks) are a subset of AI (artificial
During the training of ANN under unsupervised learning, the input vectors of similar type are intelligence). They were originally developed
combined to form clusters. When a new input pattern is applied, then the neural network gives an from the inspiration of human brains. They are basic units of human brains.
output response indicating the class to which the input pattern belongs.

There is no feedback from the environment as to what should be the desired output and if it is
correct or incorrect. Hence, in this type of learning, the network itself must discover the patterns
and features from the input data, and the relation for the input data over the output.

Reinforcement Learning

As the name suggests, this type of learning is used to reinforce or strengthen the network over
some critic information. This learning process is similar to supervised learning, however we
might have very less information. Data mining is the term used to describe the process of extracting value from a database. A data
warehouse is a location where information is stored.
During the training of network under reinforcement learning, the network receives some
feedback from the environment. This makes it somewhat similar to supervised learning. Training of ANN :
However, the feedback obtained here is evaluative not instructive, which means there is no
teacher as in supervised learning. After receiving the feedback, the network performs We can train the neural network by feeding it by teaching patterns and letting it change its
adjustments of the weights to get better critic information in future. weight according to some learning rule. We can categorize the learning situations as follows.

1. Supervised Learning: In which the network is trained by providing it with input and
matching output patterns. And these input-output pairs can be provided by an external
system that contains the neural network.
2. Unsupervised Learning: In which output is trained to respond to a cluster of patterns
within the input. Unsupervised learning uses a machine learning algorithm to analyze and
cluster unlabeled datasets.
3. Reinforcement Learning: This type of learning may be considered as an intermediate
form of the above two types of learning, which trains the model to return an optimum
solution for a problem by taking a sequence of decisions by itself.

Another method of teaching artificial neural networks is Backpropagation Algorithm. It is a

commonly used method for teaching artificial neural networks. The backpropagation algorithm is
used feed-forward ANNs.The motive of the backpropagation algorithm is to reduce this error
until the ANN learns the training data.
 UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

Steps of Backpropagation Algorithm:  A number of output nodes: The way to identify a number of output nodes is to identify
the number of output classes you want the neural network to process.
1. Present the training sample to the neural network. o Random Initialization of Weights: The weights are randomly initialized to a value
2. Compare the ANN’s Output to the wanted output from the data. between 0 and 1, or rather, very close to zero.
3. Calculate the error in each output neuron. o Implementation of forward propagation algorithm to calculate hypothesis function
4. For each neuron, calculate the scaling factor, output, and how much lower or higher the for a set of input vectors for any of the hidden layers.
output should be to match the desired output. This is a local error. o Implementation of the cost function for optimizing parameter values. One may
recall that the cost function would help determine how well the neural network
Algorithm: fits the training data.
o Implementation of a backpropagation algorithm to compute the error vector
1. Initialize the weights in the network. related to each of the nodes.
o Use the gradient checking method to compare the gradient calculated using partial
2. Repeat. derivatives of a cost function using backpropagation and using a numerical
estimate of the cost function gradient. The gradient checking method is used to
 1. O =neural-net-output(network, e) ; forward pass validate if the implementation of the backpropagation method is correct.
 T = teacher output for e o Use gradient descent or advanced optimization technique with backpropagation to
 Calculate the error (T – O) at the output units try and minimize the cost function as a function of parameters or weights.
 Compute delta_wi for all weights from the hidden layer to
output layer; backward pass The Iterative Learning Process:
 Compute delta_wi for all weights from the input layer to
hidden layer; backward pass continued During this literacy phase, the network learns by conforming the weights so as to be suitable to
 Update the weights in the network prognosticate the correct class marker of input samples. Neural network literacy is also
appertained to as” connectionist literacy,” due to the connections between the units. The
3. Until all examples are classified correctly or the stopping criterion is satisfied return(network) advantages of neural networks include their high forbearance to noisy data, as well as their
capability to classify patterns on which they’ve not been trained.

Activation Functions

Key Steps for Training a Neural Network:
Pick a neural network architecture. This implies that you shall be pondering primarily upon the
connectivity patterns of the neural network including some of the following aspects:
It may be defined as the extra force or effort applied over the input to obtain an exact output. In
ANN, we can also apply activation functions over the input to get the exact output. Followings
are some activation functions of interest −
Linear Activation Function
It is also called the identity function as it performs no input editing. It can be defined as −

 A number of input nodes: The way to identify a number of input nodes is to identify F(x)=x
the number of features.
o A number of hidden layers: The default is to use a single or one hidden layer. Sigmoid Activation Function
This is the most common practice.
o The number of nodes in each of the hidden layers: In the case of using multiple It is of two type as follows −
hidden layers, the best practice is to use the same number of nodes in each hidden
layer. In general practice, the number of hidden units is taken as a comparable Binary sigmoidal function − This activation function performs input editing between 0 and 1. It
number to that of a number of input nodes. That means one could take either the is positive in nature. It is always bounded, which means its output cannot be less than 0 and more
same number of hidden nodes as input nodes or maybe twice or thrice the number than 1. It is also strictly increasing in nature, which means more the input higher would be the
of input nodes. output. It can be defined as

F(x)=sigm(x)=11+exp(−x)
 UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

Bipolar sigmoidal function − This activation function performs input editing between -1 and 1. Mathematical proof
It can be positive or negative in nature. It is always bounded, which means its output cannot be
less than -1 and more than 1. It is also strictly increasing in nature like sigmoid function. It can Suppose we have a Neural net like this :-
be defined as

F(x)=sigm(x)=21+exp(−x)−1=1−exp(x)1+exp(x)

Activation functions in Neural Networks

In the process of building a neural network, one of the choices you get to make is what
Activation Function to use in the hidden layer as well as at the output layer of the network. This
article discusses some of the choices.

Elements of a Neural Network

Input Layer: This layer accepts input features. It provides information from the outside world to
the network, no computation is performed at this layer, nodes here just pass on the
information(features) to the hidden layer.

Hidden Layer: Nodes of this layer are not exposed to the outer world, they are part of the
abstraction provided by any neural network. The hidden layer performs all sorts of computation
on the features entered through the input layer and transfers the result to the output layer.
Elements of the diagram are as follows:
Output Layer: This layer bring up the information learned by the network to the outer world.
Hidden layer i.e. layer 1:
What is an activation function and why use them?
z(1) = W(1)X + b(1) a(1)
The activation function decides whether a neuron should be activated or not by calculating the
Here,
weighted sum and further adding bias to it. The purpose of the activation function is to introduce
non-linearity into the output of a neuron.
 z(1) is the vectorized output of layer 1
 W(1) be the vectorized weights assigned to neurons of hidden layer i.e. w1, w2, w3 and
Explanation: We know, the neural network has neurons that work in correspondence with
w4
weight, bias, and their respective activation function. In a neural network, we would update the
 X be the vectorized input features i.e. i1 and i2
weights and biases of the neurons on the basis of the error at the output. This process is known as
 b is the vectorized bias assigned to neurons in hidden layer i.e. b1 and b2
back-propagation. Activation functions make the back-propagation possible since the gradients
 a(1) is the vectorized form of any linear function.
are supplied along with the error to update the weights and biases.
(Note: We are not considering activation function here)

Layer 2 i.e. output layer :-

Why do we need Non-linear activation function?
Note : Input for layer 2 is output from layer 1
A neural network without an activation function is essentially just a linear regression model. The
activation function does the non-linear transformation to the input making it capable to learn and
z(2) = W(2)a(1) + b(2)
perform more complex tasks.
a(2) = z(2)
 UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

Calculation at Output layer  It is a function which is plotted as ‘S’ shaped graph.

 Equation : A = 1/(1 + e-x)
z(2) = (W(2) * [W(1)X + b(1)]) + b(2)  Nature : Non-linear. Notice that X values lies between -2 to 2, Y values are very steep.
This means, small changes in x would also bring about large changes in the value of Y.
z(2) = [W(2) * W(1)] * X + [W(2)*b(1) + b(2)]  Value Range : 0 to 1
 Uses : Usually used in output layer of a binary classification, where result is either 0 or 1,
Let, as value for sigmoid function lies between 0 and 1 only so, result can be predicted easily
to be 1 if value is greater than 0.5 and 0 otherwise.
[W(2) * W(1)] = W
Tanh Function
[W(2)*b(1) + b(2)] = b

Final output : z(2) = W*X + b

which is again a linear function

This observation results again in a linear function even after applying a hidden layer, hence we
can conclude that, doesn’t matter how many hidden layer we attach in neural net, all layers will
behave same way because the composition of two linear function is a linear function itself.
Neuron can not learn with just a linear function attached to it. A non-linear activation function
will let it learn as per the difference w.r.t error. Hence we need an activation function.

Variants of Activation Function

 The activation that works almost always better than sigmoid function is Tanh function
Linear Function also knows as Tangent Hyperbolic function. It’s actually mathematically shifted
version of the sigmoid function. Both are similar and can be derived from each other.
 Equation : Linear function has the equation similar to as of a straight line i.e. y = x  Equation :-
 No matter how many layers we have, if all are linear in nature, the final activation
function of last layer is nothing but just a linear function of the input of first layer.
 Range : -inf to +inf
 Uses : Linear activation function is used at just one place i.e. output layer.
 Issues : If we will differentiate linear function to bring non-linearity, result will no more  Value Range :- -1 to +1
depend on input “x” and function will become constant, it won’t introduce any ground-  Nature :- non-linear
breaking behavior to our algorithm.  Uses :- Usually used in hidden layers of a neural network as it’s values lies between -1 to
1 hence the mean for the hidden layer comes out be 0 or very close to it, hence helps in
For example : Calculation of price of a house is a regression problem. House price may have centering the data by bringing mean close to 0. This makes learning for the next layer
any big/small value, so we can apply linear activation at output layer. Even in this case neural net much easier.
must have any non-linear function at hidden layers.
RELU Function
Sigmoid Function
 UNIT-1 SOFT COMPUTING
 It Stands for Rectified linear unit. It is the most widely used activation function. Chiefly
implemented in hidden layers of Neural network.
 Equation :- A(x) = max(0,x). It gives an output x if x is positive and 0 otherwise.
 Value Range :- [0, inf)
 Nature :- non-linear, which means we can easily backpropagate the errors and have
multiple layers of neurons being activated by the ReLU function.
 Uses :- ReLu is less computationally expensive than tanh and sigmoid because it involves
simpler mathematical operations. At a time only a few neurons are activated making the
network sparse making it efficient and easy for computation.
In simple words, RELU learns much faster than sigmoid and Tanh function.
Softmax Function
The softmax function is also a type of sigmoid function but is handy when we are trying to
handle multi- class classification problems.
 Nature :- non-linear
 Uses :- Usually used when trying to handle multiple classes. the softmax function was
commonly found in the output layer of image classification problems.The softmax
function would squeeze the outputs for each class between 0 and 1 and would also divide
by the sum of the outputs.
 Output:- The softmax function is ideally used in the output layer of the classifier where
we are actually trying to attain the probabilities to define the class of each input.
 The basic rule of thumb is if you really don’t know what activation function to use, then
simply use RELU as it is a general activation function in hidden layers and is used in
most cases these days.
 If your output is for binary classification then, sigmoid function is very natural choice for
output layer.
 If your output is for multi-class classification then, Softmax is very useful to predict the
probabilities of each classes.
UNIT-1 SOFT COMPUTING
ANN Network Architectures
An Artificial Neural Network (ANN) is an information processing paradigm that is inspired by
the brain. ANNs, like people, learn by examples. An ANN is configured for a specific
application, such as pattern recognition or data classification, through a learning process.
Learning largely involves adjustments to the synaptic connections that exist between the
neurons.
Artificial Neural Networks (ANNs) are a type of machine learning model that are inspired by the
structure and function of the human brain. They consist of layers of interconnected “neurons”
that process and transmit information.
There are several different architectures for ANNs, each with their own strengths and
weaknesses. Some of the most common architectures include:
Feedforward Neural Networks: This is the simplest type of ANN architecture, where the
information flows in one direction from input to output. The layers are fully connected, meaning
each neuron in a layer is connected to all the neurons in the next layer.
Recurrent Neural Networks (RNNs): These networks have a “memory” component, where
information can flow in cycles through the network. This allows the network to process
sequences of data, such as time series or speech.
Convolutional Neural Networks (CNNs): These networks are designed to process data with a
grid-like topology, such as images. The layers consist of convolutional layers, which learn to
detect specific features in the data, and pooling layers, which reduce the spatial dimensions of
the data.
Autoencoders: These are neural networks that are used for unsupervised learning. They consist
of an encoder that maps the input data to a lower-dimensional representation and a decoder that
maps the representation back to the original data.
Generative Adversarial Networks (GANs): These are neural networks that are used for
generative modeling. They consist of two parts: a generator that learns to generate new data
samples, and a discriminator that learns to distinguish between real and generated data.
The model of an artificial neural network can be specified by three entities:
Interconnections:
Interconnection can be defined as the way processing elements (Neuron) in ANN are connected
to each other. Hence, the arrangements of these processing elements and geometry of
interconnections are very essential in ANN.
These arrangements always have two layers that are common to all network architectures, the
Input layer and output layer where the input layer buffers the input signal, and the output layer
generates the output of the network. The third layer is the Hidden layer, in which neurons are
 UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

neither kept in the input layer nor in the output layer. These neurons are hidden from the people This layer also has a hidden layer that is internal to the network and has no direct contact with
who are interfacing with the system and act as a black box to them. By increasing the hidden the external layer. The existence of one or more hidden layers enables the network to be
layers with neurons, the system’s computational and processing power can be increased but the computationally stronger, a feed-forward network because of information flow through the input
training phenomena of the system get more complex at the same time. function, and the intermediate computations used to determine the output Z. There are no
feedback connections in which outputs of the model are fed back into itself.
There exist five basic types of neuron connection architecture :
3. Single node with its own feedback
1. Single-layer feed-forward network
2. Multilayer feed-forward network
3. Single node with its own feedback
4. Single-layer recurrent network
5. Multilayer recurrent network

1. Single-layer feed-forward network

Single Node with own Feedback

In this type of network, we have only two layers input layer and the output layer but the input
layer does not count because no computation is performed in this layer. The output layer is
formed when different weights are applied to input nodes and the cumulative effect per node is
taken. After this, the neurons collectively give the output layer to compute the output signals.
2. Multilayer feed-forward network
When outputs can be directed back as inputs to the same layer or preceding layer nodes, then it
results in feedback networks. Recurrent networks are feedback networks with closed loops. The
above figure shows a single recurrent network having a single neuron with feedback to itself.
4. Single-layer recurrent network
 UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

The above network is a single-layer network with a feedback connection in which the processing
element’s output can be directed back to itself or to another processing element or both. A
recurrent neural network is a class of artificial neural networks where connections between nodes
form a directed graph along a sequence. This allows it to exhibit dynamic temporal behavior for
a time sequence. Unlike feedforward neural networks, RNNs can use their internal state
(memory) to process sequences of inputs.

5. Multilayer recurrent network

McCulloch-Pitts Model

Simple McCulloch-Pitts neurons can be used to design logical operations. For that
purpose, the connection weights need to be correctly decided along with the threshold
function (rather than the threshold value of the activation function). For better
understanding purpose, let me consider an example:
John carries an umbrella if it is sunny or if it is raining. There are four given situations. I
need to decide when John will carry the umbrella. The situations are as follows:
 First scenario: It is not raining, nor it is sunny
In this type of network, processing element output can be directed to the processing element in  Second scenario: It is not raining, but it is sunny
the same layer and in the preceding layer forming a multilayer recurrent network. They perform  Third scenario: It is raining, and it is not sunny
the same task for every element of a sequence, with the output being dependent on the previous
 Fourth scenario: It is raining as well as it is sunny
computations. Inputs are not needed at each time step. The main feature of a Recurrent Neural
Network is its hidden state, which captures some information about a sequence.
To analyse the situations using the McCulloch-Pitts neural model, I can consider the
input signals as follows:
 X1: Is it raining?
 X2 : Is it sunny?
Implementing Models of Artificial Neural So, the value of both scenarios can be either 0 or 1. We can use the value of both weights
X1 and X2 as 1 and a threshold function as 1. So, the neural network model will look like:
Network
1. McCulloch-Pitts Model of Neuron
The McCulloch-Pitts neural model, which was the earliest ANN model, has only two
types of inputs — Excitatory and Inhibitory. The excitatory inputs have weights of
positive magnitude and the inhibitory weights have weights of negative magnitude. The
inputs of the McCulloch-Pitts neuron could be either 0 or 1. It has a threshold function as
an activation function. So, the output signal yout is 1 if the input ysum is greater than or equal
to a given threshold value, else 0. The diagrammatic representation of the model is as
follows:
 UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

Truth Table for this case will be:

Rosenblatt’s Perceptron
Situation x1 x2 ysum yout
The perceptron receives a set of input x1, x2,….., xn. The linear combiner or the adder
1 0 0 0 0 mode computes the linear combination of the inputs applied to the synapses with synaptic
weights being w1, w2,……,wn. Then, the hard limiter checks whether the resulting sum is
positive or negative If the input of the hard limiter node is positive, the output is +1, and
2 0 1 1 1 if the input is negative, the output is -1. Mathematically the hard limiter input is:

3 1 0 1 1 However, perceptron includes an adjustable value or bias as an additional weight w0. This
additional weight is attached to a dummy input x0, which is assigned a value of 1. This
consideration modifies the above equation to:
4 1 1 2 1

So, I can say that, The output is decided by the expression:

The truth table built with respect to the problem is depicted above. From the truth table, I
can conclude that in the situations where the value of yout is 1, John needs to carry an
umbrella. Hence, he will need to carry an umbrella in scenarios 2, 3 and 4.
2. Rosenblatt’s Perceptron
Rosenblatt’s perceptron is built around the McCulloch-Pitts neural model. The
diagrammatic representation is as follows:
The objective of the perceptron is o classify a set of inputs into two classes c1 and c2. This
can be done using a very simple decision rule – assign the inputs to c1 if the output of the
perceptron i.e. yout is +1 and c2 if yout is -1. So for an n-dimensional signal space i.e. a
space for ‘n’ input signals, the simplest form of perceptron will have two decision
regions, resembling two classes, separated by a hyperplane defined by:

Therefore, the two input signals denoted by the variables x1 and x2, the decision boundary
is a straight line of the form:
or
So, for a perceptron having the values of synaptic weights w 0,w1 and w2 as -2, 1/2 and 1/4,
respectively. The linear decision boundary will be of the form:
 UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

So, any point (x,1x2) which lies above the decision boundary, as depicted by the graph,
will be assigned to class c1 and the points which lie below the boundary are assigned to
class c2.

Linear Vs Non-Linearly Separable Classes

Multi-layer perceptron: A basic perceptron works very successfully for data sets which
possess linearly separable patterns. However, in practical situations, that is an ideal
situation to have. This was exactly the point driven by Minsky and Papert in their work in
1969. They showed that a basic perceptron is not able to learn to compute even a simple 2
bit XOR. So, let us understand the reason.
Consider a truth table highlighting output of a 2 bit XOR function:

x1 x2 x1 XOR x2 Class
Thus, we see that for a data set with linearly separable classes, perceptrons can always be
employed to solve classification problems using decision lines (for 2-dimensional space),
decision planes (for 3-dimensional space) or decision hyperplanes (for n-dimensional 1 1 0
c2
space). Appropriate values of the synaptic weights can be obtained by training a
perceptron. However, one assumption for perceptron to work properly is that the two
classes should be linearly separable i.e. the classes should be sufficiently separated from 1 0 1
c1
each other. Otherwise, if the classes are non-linearly separable, then the classification
problem cannot be solved by perceptron.
0 1 1
c1

0 0 0
c2

The data is not linearly separable. Only a curved decision boundary can separate the
classes properly. To address this issue, the other option is to use two decision boundary
lines in place of one.
 UNIT-1 SOFT COMPUTING UNIT-1 SOFT COMPUTING

Classification with two decision lines in the XOR function output

This is the philosophy used to design the multi-layer perceptron model. The major
highlights of this model are as follows:
 The neural network contains one or more intermediate layers between the input and
output nodes, which are hidden from both input and output nodes
 Each neuron in the network includes a non-linear activation function that is
differentiable.
 The neurons in each layer are connected with some or all the neurons in the previous
layer.
3. ADALINE Network Model
Adaptive Linear Neural Element (ADALINE) is an early single-layer ANN developed by
Professor Bernard Widrow of Stanford University. As depicted in the below diagram, it
has only output neurons. The output value can be +1 or -1. A bias input x0 (where x0 =1)
having a weight w0 is added. The activation function is such that if weighted sum is
positive or 0, the output is 1, else it is -1. Formally I can say that,

The supervised learning algorithm adopted by ADALINE network is known as Least

Mean Square (LMS) or DELTA Rule. A network combining a number of ADALINE is
termed as MADALINE (many ADALINE). MEADALINE networks can be used to
solve problems related to non-linear separability.