Neural Network

Neural Networks
• Neural networks are a method of separating space into
classes in a way that can include non-linear separation.
• Neural networks are based on a model of the working
of the brain: the network is organized as a series of
nodes (simulating neurons), which have inputs and
outputs.
• The output of the nodes depends on the input into the
nodes; the different inputs all have relative
importance, which are determined by a set of
parameters known as weights.
• The neural network has the ability to learn by
adjusting the weights.
• It is trained by giving it examples of samples to
be classified; the network adjusts the weights
on the input of the nodes so that it produces
the correct output.
• The network is trained until it shows no
further improvements in predicting the classes
of the training set.
• There are two steps for using a neural network
to predict the class of an individual:
1. Train the neural network using the samples
with known classes.
2. Apply the neural network to the new
individual to determine its class.
• Neural networks have the key advantage of
being able to discriminate non-linearly
separable data.
 Artificial Neural Network (ANN)
What is ANN?
• The ANN is an information-processing paradigm
inspired by the structure of biological nervous
systems such as the human brain.
• The primary characteristic of this paradigm is
modeling the core information-processing system
as a neuron and building a network composed of
a large number of highly interconnected neurons
working to solve specific problems.
This is a very simple example of a neural network
 Key concepts of ANN
The key concepts therefore are
• characteristics of the neuron,
• the neuron interconnections, and
• the neuron activation system consisting of
several mathematical functions.
How is neuron activated in brain function?

• The major parts of a biological neuron (Figure

3.2) are the cell body, which contains the
nucleus and a branching dendritic tree that
collects signals from neighboring neurons.
• The neuron’s output is transmitted via the
axon, which forms a connection or synapse
with the dendrites of neighboring neurons
(Figure 3.3).
• The nucleus is the processing center, which
integrates the input signals and generates a
response that is a nonlinear function of the
inputs and the internal state.
• The neuron alone does not possess
considerable information-processing
capabilities, but a large network formed from
these single neurons gives rise to special
processing abilities.
 Perceptron/Artificial neuron
• The artificial neuron can be modeled as a
system that receives a quantity of data as
inputs and produces a single output.
• An artificial neuron has many similarities with
a biological neuron in that it is referred to as
the perceptron because it mimics the
fundamentals of mammalian visual systems.
• The perceptron is an example of the simplest
feed-forward network.
The similarities between the artificial neuron
and the biological neuron
a) Inputs:
The scalar xi represents the electrical input into
the neuron. In practice, the input consists of data
such as measurements or attributes that mimic
the dendrite trees of the biological neuron. The
importance of each input is represented by the
strength of the synapse connection that is
modeled by the scalar weights Wi, which are pre-
multiplied with the individual input Xi.
b) Summation and activation function:
As in nucleus of a neuron, processing of the weighted
inputs and the output response takes place at a summation
node and is modeled as the following mathematical
function:
)
Here the function f is known as the activation function and
determines the behavior of the neuron. Mathematically, it
is usually taken to be some bounded non decreasing
nonlinear function such as the sigmoid function .
)=
Other commonly used functions are the Gaussian,
step, piecewise linear (ramp). For linear activation
functions, the output activity is directly
proportional to the total weighted output. For
step or threshold functions, the output is set at
one of two levels, depending on whether the total
input is greater than or less than some threshold
value.
Examples of activation functions that have been used in artificial neurons.
(a) Step function; (b) Gaussian; (c) piecewise linear; and (d) sigmoid.
Some activation functions are chosen such that
very large positive or negative numbers are
limited. In this case, they are usually called
squashing functions, for example, sigmoid and
step functions.
c) Output:
The output represents the strength of the
electrical impulse travelling along the axon.
d) Feedback:
In some ANNs, there is some provision to feed
the output back into the input so that the neural
network is adaptive.
The error between the output of the neuron, f()
and the actual class label is usually taken to be
the difference
• If the number of training patterns is n, the
sum of errors for a single neuron is written as,

• The sum of squared errors then is

Block diagram of various neural network
architectures
 Which sector ANN used for?
• Artificial Neural Networks are used for a variety of
tasks, a popular use is for classification. You can collect
datasets of images for example of different breeds of
dogs and then train a neural network on the images,
then if you supply a new image of a dog it will give a
statistical score on how closely the new image matches
the model and then will output what breed of dog the
image is. Neural Networks are also used in Self Driving
cars, Character Recognition, Image Compression, Stock
Market Prediction, and lots of other interesting
applications.
 How does a neural network learn
Information flows through a neural network in two different ways.
When the model is learning (being trained) or operating normally
(after being trained either being used or tested), patterns of
information from the dataset are being fed into the network via the
input neurons, which trigger the layers of hidden neurons, and
these in turn arrive at the output neurons. This is called a feed-
forward network. Not all neurons “fire” all the time. Each neuron
receives inputs from the neurons to its left, and the inputs are
multiplied by the weights of the connections they travel along.
Every neuron adds up all the inputs it receives in this way and (this
is the simplest neural network) if the sum is more than a certain
threshold value, the neuron “fires” and triggers the neurons it’s
connected to (the neurons on its right).
• For an artificial neural network to learn, it has
to learn what it has done wrong and is doing
right, this is called feedback. Feedback is how
we learn what is wrong and right and this is
also what an artificial neural network needs
for it to learn. This is where you start to see
similarities to the human brain.
Neural networks learn things in exactly the same way as the
brain, typically by a feedback process called back-propagation
(this is sometimes shortened to “backprop.”). This is where you
compare the output of the network with the output it was
meant to produce, and using the difference between the
outputs to modify the weights of the connections between the
neurons in the network, working from the output units through
the hidden neurons to the input neurons going backward. Over
time, back-propagation causes the network to learn by making
the gap between the output and the intended output smaller to
the point where the two exactly match, so the neural network
learns the correct output.
 Types of Learning
The learning type can be classified as three. they
are
1. Supervised learning
2. Unsupervised learning
3. Reinforcement learning
 Supervised learning
• A network is fed with a set of training samples
(inputs and corresponding output), and it uses
these samples to learn the general
relationship between the inputs and the
outputs.
• This relationship is represented by the values
of the weights of the trained network.
 Unsupervised learning
• No desired output is associated with the training data.
• Unsupervised learning studies how systems can learn to
represent particular input patterns in a way that reflects
the statistical structure of the overall collection of input
patterns.
• By contrast with Supervised or Reinforcement learning,
there are no explicit target outputs or environmental
evaluations associated with each input; rather the
unsupervised learner brings to bear prior biases as to
what aspects of the structure of the input should be
captured in the output.
 Reinforcement learning
• Reinforcement learning (RL) is a kind of
supervised learning in that some feedback
from the environment is given. However the
feedback signal is only evaluative, not
instructive. Reinforcement learning is often
called learning with a critic as opposed to
learning with a teacher.