Chapter: Five

Neural Network
 Outline
What is Neural network?
Why Neural Network?
Understanding the Brain
Neural networks as a paradigm
for parallel processing
The Perceptron
Training a Perceptron
Artificial neural network (ANN)
Multilayer Perceptron
Backpropagation algorithm
Nonlinear Regression
Two-Class Discrimination
Multiclass Discrimination
Multiple Hidden Layers
Training procedures
Improving convergence
Overtraining
Structuring the network
Tuning the network size
A case study in neural network
 What is a Neural Network (NN)?
A neural network is a method in artificial intelligence that teaches computers to process data
in a way that is inspired by the human brain.

It is a type of ML process, called deep learning, that uses interconnected nodes or neurons in a
layered structure that resembles the human brain.

And also, collaborate to tackle complicated problems.

NN-it creates an adaptive system that computers use to learn from their mistakes and improve
continuously.

Also referred to as artificial neural networks (ANNs) or deep neural networks.

 NN represents a type of deep learning technology that's classified under the broader field of
AI.
 Why Neural Networks?
Problem Before Neural Networks:
Unless the specific steps that the computer needs to follow are known, the computer
cannot solve a problem.
i.e. The computer follows a set of instructions to solve a problem.

This restricts the problem-solving capability of conventional computers to problems that

we already understand and know how to solve.
---that is where our traditional approach was a failure and why neural networks were
introduced.
What are the scenarios After Neural Networks:
 With neural Networks computers can do things that we don’t exactly know how to do
 Neural networks learn by example
 With experience it can learn how to solve the problem.
 They can not be programmed to perform a specific task.
 Motivation Behind Neural Networks
The human brain is the inspiration behind neural network architecture.

In living organisms, the brain is the control unit of the neural network, and it has different subunits
that take care of vision, senses, movement, and hearing.

The brain is connected with a dense network of nerves to the rest of the body’s sensors and actors.

There are approximately 10¹¹ neurons in the brain, and these are the building blocks of the
complete central nervous system of the living body.

The neuron is the fundamental building block of neural networks.

 In biological systems, a neuron is a cell just like any other cell of the body, which has a DNA
code and is generated in the same way as the other cells.

So, an artificial neuron works much the same way the biological one does.
 Motivation Behind Neural Networks…
A neuron comprises three major parts: the cell body (also called Soma), the dendrites,
and the axon.

Neuron
 Understanding the Brain

 How do our brains Work?

 The Brain is A massively parallel information processing system.

 Our brains are a huge network of processing elements.
 A typical brain contains a network of 10 billion neurons.
 How do our brains work? …
A processing element
 Dendrites: Input
 Cell body: Processor
 Synaptic: Link
 Axon: Output

 A neuron is connected to other neurons through about 10,000 synapses

 A neuron receives input from other neurons. Inputs are combined.

 Once input exceeds a critical level, the neuron discharges a spike ‐ an electrical pulse that travels
from the body, down the axon, to the next neuron(s)
 The axon endings almost touch the dendrites or cell body of the next neuron.
 Transmission of an electrical signal from one neuron to the next is affected by neurotransmitters.
 Neurotransmitters are chemicals that are released from the first neuron and which bind to the
Second.
 This link is called a synapse.
 The strength of the signal that reaches the next neuron depends on factors such as the amount of
neurotransmitter available.
 Why Understand Biological Neural Networks?
ANNs can be used to replicate and simulate components of the human (or animal) brain, thereby giving
us insight into natural information processing.
For creating mathematical models for ANNS, theoretical analysis of biological neural networks is
essential as they have a very close relationship.
 This understanding of the brain’s neural networks has opened horizons for the development of artificial
neural network systems and adaptive systems designed to learn and adapt to situations and inputs.

Other factors that need to be considered.

 There are many layers in ANNs and they are interconnected to solve classification problems.
 Biological neural networks tolerate a great deal of ambiguity in data.
 However, ANNs require somewhat precise, structured, and formatted data to tolerate
ambiguity.
 Biological neural networks are fault-tolerant to a certain level, and the minor failures will
not always result in memory loss.
 Why are Neural Networks important?
Neural networks can help computers make intelligent decisions with limited human
assistance.
This is because they can learn and model the relationships between input and output data that
are nonlinear and complex.
 For instance, they can do the following tasks.
Make generalizations and inferences
NN can comprehend unstructured data and make general observations without explicit
training.
For instance, they can recognize that two different input sentences have a similar
meaning:
 Can you tell me how to make the payment?
 How do I transfer money?
A neural network would know that both sentences mean the same thing.
Or it would be able to broadly recognize that Baxter Road is a place, but Baxter Smith is a
person’s name.
 The Perceptron

Perceptron is a single layer neural network, or we can say a neural network is a multi-
layer perceptron.

 Perceptron is a binary classifier, and it is used in supervised learning.

A simple model of a biological neuron in an ANN is known as Perceptron.

A function that can decide whether or not an input which is represented by a vector of
number belongs to some specific class is known as binary classifiers.

The binary classifier is a type of linear classifier.

 A linear classifier is a classification algorithm which makes its predictions based on a

linear predictor function combining a set of weight with the feature vector.
 The Perceptron…
The perceptron algorithm was designed to categorizing subjects into one of two types,
classify visual input and separating groups with a line.
 Classification is a key part of image processing and machine learning.
The perceptron algorithm classifies patterns, i.e., find and classify by many different means
using machine learning algorithm, and groups by finding the linear separation between
different objects and patterns which are received through numeric or visual input.

A normal neural network looks like the following.

 The Perceptron…

Input values or one input layer

The input layer of a perceptron is made of artificial input neurons and brings the initial
data into the system for further processing.

Weight:-represents the strength or dimension of the connection between units.

 If the weight from node 1 to node 2 has the greater quantity, then neuron 1 has greater
influence over neuron 2.
How much influence of the input will have on the output, is determined by weight.

Bias:-is similar to the intercept added in a linear equation.

 It is an additional parameter which task is to adjust the output along with the weighted
sum of the inputs to the neuron.
 Activation Function
 A neuron should be activated or not, is determined by an activation function.
 Activation function calculates a weighted sum and further adding bias with it to give the
result.

 Note: Neural Network is based on the Perceptron, so if we want to know the working of the
neural network, learn how perceptron work.
 The Perceptron works on three simple steps which are as
follows:

A. In the first step, all the input x are multiplied with their weights denoted as K. This step
is essential because the output of this step will be input for the next step.
 Step 2:
b) add all the multiplied value from K1 to Kn. It is known as the weighted sum.
This weighted sum will be treated as an input for the next step.
 Step 3:
c) In the next step, the weighted sum, which is calculated from the previous step, is
applied to the correct activation function.
For example: A unit step activation function

 Note 1:
 Weight shows the strength of the particular node.
 A bias value allows you to shift the activation
function curve up or down.
 The activation functions are used to map the input
between the required value like (0, 1) or (-1, 1)
 Perceptron is usually used to classify the data into
two parts. Therefore, it is also known as a Linear
Binary Classifier.
 Artificial Neural Network (ANN)
What is Artificial Neural Network (ANN)?

 The term "Artificial Neural Network" is derived from Biological neural networks that
develop the structure of a human brain.
 ANNs are computing systems inspired by the biological neural networks that constitute
human (animal) brains.
 Such systems learn (progressively improve performance) to do tasks by considering
examples, generally without task-specific programming
 Similar to the human brain which has neurons interconnected to one another,

 An ANN consists of a pool of simple processing units that communicate by sending

signals to each other over a large number of weighted connections.
 Artificial Neural Network (ANN)
 ANN also have neurons that are interconnected to one another in various layers of the
networks.
 These neurons are known as nodes.

 The typical ANN looks something like

the given figure.

 Dendrites from Biological Neural Network represent inputs in ANN , cell nucleus represents
Nodes, synapse represents Weights, and Axon represents Output.
 Relationship between Biological neural network and ANN

Biological Neural Network Artificial Neural Network

Dendrites Inputs

Cell nucleus Nodes

Synapse Weights

Axon Output
 Artificial Neural Network (ANN)
An ANN in the field of Artificial intelligence where it attempts to mimic the network
of neurons makes up a human brain so that computers will have an option to
understand things and make decisions in a human-like manner.

 The ANN is designed by programming computers to behave simply like

interconnected brain cells.
There are around 1000 billion neurons in the human brain. Each neuron has an
association point somewhere in the range of 1,000 and 100,000. In the human brain, data is
stored in such a manner as to be distributed, and we can extract more than one piece of this
data when necessary from our memory parallelly. We can say that the human brain is made
up of incredibly amazing parallel processors.
 Artificial Neural Network (ANN)
Let us understand the ANN with an example,

Consider an example of a digital logic gate that takes an input and gives an output.
"OR" gate, which takes two inputs.
 If one or both the inputs are "On," then we get "On" in output. If both the inputs are
"Off," then we get "Off" in output.

 Here the output depends upon input.

Our brain does not perform the same task.

The outputs-inputs relationship keeps changing because of the neurons in our brain, which
are "learning."
 Artificial Neural Network (ANN)
The “building blocks” of neural networks are the neurons.
• In technical systems, we also refer to them as units or nodes.

Basically, each neuron

 receives input from many other neurons.
 changes its internal state (activation) based on the current input.
 sends one output signal to many other neurons, possibly including its input neurons
(recurrent network).
Information is transmitted as a series of electric impulses, so-called spikes.
The frequency and phase of these spikes encodes the information.
In biological systems, one neuron can be connected to as many as 10,000 other
neurons.
Usually, a neuron receives its information from other neurons in a confined
area, its so-called receptive field.
 The architecture of an ANN:
Let us look at various types of layers available in an artificial neural network.
Artificial Neural Network primarily consists of three layers:
 The three layers of ANN
Input Layer:

 it accepts inputs in several different formats provided by the programmer.

Hidden Layer:
 presents in-between input and output layers.
 It performs all the calculations to find hidden features and patterns.

Output Layer:
 The input goes through a series of transformations using the hidden layer, which
finally results in output that is conveyed using this layer.
 The ANN takes input and computes the weighted sum of the inputs and includes a
bias.
 This computation is represented in the form of a transfer function.
 The three layers of ANN-Function

 It determines weighted total is passed as an input to an activation function to produce the

output.
 Activation functions choose whether a node should fire or not.
 Only those who are fired make it to the output layer.
 There are distinctive activation functions available that can be applied upon the sort of task
we are performing.
 How do ANNs work?
 An ANN is either a hardware implementation or a computer program which strives to

simulate the information processing capabilities of its biological exemplar.

 ANNs are typically composed of a great number of interconnected artificial neurons.

The artificial neurons are simplified models

of their biological counterparts.
ANN is a technique for solving problems by
constructing software that works like our
brains.
 How do ANNs work?
An ANN usually involves many processors operating in parallel and arranged in tiers or layers.
The first tier -- analogous to optic nerves in human visual processing -- receives the raw input
information.
 Each successive tier receives the output from the tier preceding it rather than the raw input --
the same way neurons further from the optic nerve receive signals from those closer to it. The
last tier produces the output of the system.
Each processing node has its own small sphere of knowledge, including what it has seen and
any rules it was originally programmed with or developed for itself.
The tiers are highly interconnected, which means each node in Tier N will be connected to
many nodes in Tier N-1 -- its inputs -- and in Tier N+1, which provides input data for those
nodes.
There could be one or more nodes in the output layer, from which the answer it produces can be
read.
 How do ANNs work?
 Let’s discuss an Artificial neuron called a perception.

Artificial Neuron
Biological neuron
 How do ANNs work?

 An artificial neuron is an imitation of a human neuron

 How do ANNs work?

Input

Processing

Output
 How do ANNs work?
Not all inputs are equal
Input

weights ∑= X1w1+X2w2 + ….+Xmwm =y

Processing

Output Y=
 How do ANNs work?
 The signal is not passed down to the next neuron verbatim

Input

weights

Processing

Output
The output is a function of the input, that is affected by the weights, and the transfer functions.
 Multilayer Perceptron-ANNs
 Single-Layer Perceptron – This is the simplest feedforward neural network and does not
contain any hidden layer.
As you know our brain is made up of millions of neurons, so a Neural network is just a
composition of perception, connected in different ways and operating on different activations.
MLP is a neural network similar to perceptron, but with more than one layer of neurons in
direct power.
 ANN- Example
Let’s see an example where an Artificial neural network is used for image recognition.
 Training A Neural Network

Let’s see how to train a neural network or a multilayer perceptron.

The most common deep learning algorithm for supervised training of the multi-layer
perceptions is known as backpropagation.

In it, after the weighted sum of inputs and passing through the activation function we
propagate backward and update the weights to reduce the error(desired output-model
output).

 Questions: Which among the following is not a layer of a neural network?

A. input Layer B. Output Layer C. Hidden Layer D. propagation Layer
 Backpropagation Algorithm
Basic topics:
o What is backpropagation?
o What is backpropagation in a neural network?
o How does Backpropagation algorithm work?
o Benefits of backpropagation
o Example of backpropagation

What is Backpropagation?

 Is an algorithm which is created to test errors that will travel back from input
nodes to output nodes.
 What is Backpropagation in Neural network?
Backpropagation is the essence of neural network training.

It is the method of fine-tuning the weights of a neural network based on the error rate

obtained in the previous epoch (i.e., iteration).

Is applied to reduce cost function and reduce errors.

Proper tuning of the weights allows you to reduce error rates and make the model

reliable by increasing its generalization.

Backpropagation in neural network is a short form for “backward propagation of
errors.”
 It is a standard method of training artificial neural networks.
This method helps calculate the gradient of a loss function with respect to all the
weights in the network.
 How Backpropagation Algorithm Works
In neural network computes the gradient of the loss function for a single weight by the chain
rule.
It efficiently computes one layer at a time, unlike a native direct computation.

It computes the gradient, but it

does not define how the gradient is
used.
 It generalizes the computation in
the delta rule.

Consider the following Back propagation

NN example diagram to understand:
How Backpropagation Algorithm Works
1.Input X, arrives through the preconnected path
2.Input is modeled using real weights W.
 The weights are usually randomly selected.
3. Calculate the output for every neuron from the
input layer, to the hidden layers, to the output
layer.
4. Calculate the error in the outputs.
ErrorB= Actual Output – Desired Output
5. Travel back from the output layer to the hidden
layer to adjust the weights such that the error is
decreased.
Note: Keep repeating the process until the desired
output is achieved
 Benefits of Backpropagation

Most prominent advantages of Backpropagation are:

Backpropagation is

 fast, simple, versatile, and easy to program

It has no parameters to tune apart from the numbers of input

It is a flexible method as it does not require prior knowledge about the network

It is a standard method that generally works well

It does not need any special mention of the features of the function to be learned.

 i.e. no extra function.

 Applications of Backpropagation.

Speech Recognition Character Recognition

Signature Recognition Face Recognition

 Example of Backpropagation

Backpropagation is one of the important concepts of a neural network.

Our task is to classify our data best.
For this, we have to update the weights of parameters and bias, but how can we do that in a
deep neural network?
 In the linear regression model, we use gradient descent to optimize the parameter.
Similarly here we also use a gradient descent algorithm using Backpropagation.
For a single training example,
 the Backpropagation algorithm calculates the gradient of the error function.

 Backpropagation can be written as a function of the neural network.

 Backpropagation algorithms are a set of methods used to efficiently train ANN following a
gradient descent approach which exploits the chain rule.
 Example of Backpropagation
The main features of Backpropagation are the iterative, recursive and efficient method
through which it calculates the updated weight to improve the network until it is not able
to perform the task for which it is being trained.

Derivatives of the activation function to be known at network design time are required for
Backpropagation.

Now, how error function is used in Backpropagation, and how does Backpropagation
work?

 Let’s start with an example and do it mathematically to understand how exactly updates
the weight using Backpropagation.
Example of Backpropagation… Given

 Input values
X1=0.05
X2=0.10

 Initial weight
W1=0.15 w5=0.40
W2=0.20 w6=0.45
W3=0.25 w7=0.50
W4=0.30 w8=0.55

 Bias Values
b1=0.35 b2=0.60

Target Values
T1=0.01
T2=0.99
 Step1: Calculate the values of H1 and H2 by a forward pass.

Forward Pass
• To find the value of H1 we first multiply the input  We will calculate the value of H2 in the
value from the weights as: same way as H1
• H1=x1×w1+x2×w2+b1 H2=x1×w3+x2×w4+b1
• H1=0.05×0.15+0.10×0.20+0.35 H2=0.05×0.25+0.10×0.30+0.35
• H1=0.3775 H2=0.3925
• To calculate the final result of H1, we performed the  To calculate the final result of H1, we
sigmoid function as: performed the sigmoid function as
 Step 2: Calculate the values of y1 and y2 in the same way as we calculate H1 and
H2.

 To find the value of y1, we first multiply the input value

 i.e., the outcome of H1 and H2 from the weights as:  We will calculate the value of y2 in the same
way as y1
y1=H1×w5+H2×w6+b2
y1=0.593269992×0.40+0.596884378×0.45+0.60 y2=H1×w7+H2×w8+b2
y1=1.10590597 y2=0.593269992×0.50+0.596884378×0.55+0.6
0
 To calculate the final result of y1 we performed y2=1.2249214
the sigmoid function as
 Step 3: Find the total error

Our target values are 0.01 and 0.99.

Our y1 and y2 value is not matched with our target values T1 and T2.
Now, we will find the total error, which is simply the difference between the outputs from
the target outputs.
The total error is calculated as

So, the total error is:

 Now, we will backpropagate this error to update the weights using a backward pass.
 Step 4: Backward pass at the output layer

To update the weight, we calculate the error correspond to each weight with the help of a
total error.

The error on weight w is calculated by differentiating total error with respect to w.

 We perform backward process so first consider the last weight w5 as:

 Step 4: Cont’d

From equation two, it is clear that we cannot partially differentiate it with respect to w5
because there is no any w5.

We split equation one into multiple terms so that we can easily differentiate it with respect
to w5 as
  Now, we calculate each term one by one to differentiate
Step 4: Cont’d Etotal with respect to w5 as.

 Putting the value of e-y in equation (5)

 Step 5: put the values of Backpropagation Process in Deep NN

So, we put the values of Backpropagation Process in Deep Neural Network in equation no
(3) to find the final result.
 Step 6: calculate the updated weight
Now, we will calculate the updated weight w5 new with the help of the following formula

 In the same way, we calculate w6new,w7new, and w8new and this will give us the following
values.
w5new=0.35891648
w6new=408666186
w7new=0.511301270
w8new=0.561370121
 Backpropagation Key Points
Simplifies the network structure by elements weighted links that have the least effect on the
trained network

You need to study a group of input and activation values to develop the relationship between
the input and hidden unit layers.

It helps to assess the impact that a given input variable has on a network output.

 The knowledge gained from this analysis should be represented in rules.

Backpropagation is especially useful for deep neural networks working on error-prone projects,
such as image or speech recognition.

Backpropagation takes advantage of the chain and power rules allows backpropagation to
function with any number of outputs.
 What are neural networks used for?
Neural networks have several use cases across many industries, such as the following:
 Medical diagnosis by medical image classification

 Targeted marketing by social network filtering and behavioral data analysis

 Financial predictions by processing historical data of financial instruments

 Electrical load and energy demand forecasting

 Process and quality control

 Chemical compound identification

 speech-to-text transcription

 oil exploration data analysis.

 weather prediction

 Facial recognition.
 Applications of NNs
Computer vision
 Computer vision is the ability of computers to extract information and insights from images
and videos.
 With neural networks, computers can distinguish and recognize images similar to humans.

 Visual recognition in self-driving cars so they can recognize road signs and other road users

 Content moderation to automatically remove unsafe or inappropriate content from image and
video archives
 Facial recognition to identify faces and recognize attributes like open eyes, glasses, and facial
hair
 Image labeling to identify brand logos, clothing, safety gear, and other image details.
 Applications of NNs
Speech recognition
Neural networks can analyze human speech despite varying speech patterns, pitch,
tone, language, and accent.
Virtual assistants like Amazon Alexa and automatic transcription software use
speech recognition to do tasks like these:
 Assist call center agents and automatically classify calls

 Convert clinical conversations into documentation in real-time.

 Accurately subtitle videos and meeting recordings for wider content reach
 Applications of NNs…
Natural language processing (NLP)
 NLP is the ability to process natural, human-created text.

 Neural networks help computers gather insights and meaning from text data and documents.

 NLP has several use cases, including in these functions:

 Automated virtual agents and chatbots

 Automatic organization and classification of written data

 Business intelligence analysis of long-form documents like emails and forms

 Indexing of key phrases that indicate sentiment, like positive and negative comments on social
media
 Document summarization and article generation for a given topic
 Applications of NNs
Recommendation engines
 Neural networks can track user activity to develop personalized recommendations.
 They can also analyze all user behavior and discover new products or services that interest a
specific user.
 For example, Curalate, a Philadelphia-based startup, helps brands convert social media posts into
sales.
 Brands use Curalate’s intelligent product tagging (IPT) service to automate the collection and
curation of user-generated social content.
 IPT uses neural networks to automatically find and recommend products relevant to the user’s
social media activity.
 Consumers don't have to hunt through online catalogs to find a specific product from a social
media image.
 Instead, they can use Curalate’s auto product tagging to purchase the product with ease.
Advantages of Artificial Neural Network (ANN)
 Frequently Asked Questions (FAQs)
1. What is the need of biological neural network?

• Neural network, a network of simple processors (neurons) is found everywhere in the organism: in human brain, in

every animal brain and in our heart, pancreas or lungs. It's a very efficient mechanism, whose functioning principle

is based on the learning process, that makes these systems very adaptive. The study of biological neural networks is

important to understand and simulate the functioning of our own brain, the best known and most complex biological

neural network in the world. This can lead to the development of bio-inspired artificial neural networks.

• 2. What are the characteristics of a biological neural network?

• A biological neural network is a network of neurons that are connected together by axons and dendrites. The

connections between neurons are made by synapses. The axons transport chemicals that cause neurotransmitters to

be released onto dendrites, where the neurotransmitters are then able to excite or inhibit an adjacent neuron. The

neural network is able to learn and remember information, allowing it to solve problems or make decisions.
 Frequently Asked Questions (FAQs)
3. What are the limitations of deep learning?
 The limitations of deep learning are similar to the limitations of all machine learning
techniques. The common problem for all techniques is that they only give you the answers to
the questions you ask them. They can't answer questions that you didn't think of before. Deep
learning is heavily dependent on the data you give them. If your data isn't complete, there are
gaps in it, or the data itself is suspect then your deep learning model won't be very good. As a
result you will get poor performance.
• Backpropagation Process in Deep Neural Network - javatpoint