ARTIFICIAL INTELLIGENCE AND

MACHINE LEARNING
Course Code: U18CI606
UNIT 2:Neural networks basics

S Siva Priyanka
M.Tech (Ph.D)
Assistant Professor
Dept. of ECE, KITS Warangal
Course Objective
 To understand the basics on neural networks
 To analyze the MLP
 Content to be Covered in Course
U18CI606 ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING

UNIT–II (9)
 Artificial Neural Networks:
 Biological neuron structure,
 artificial neuron, implementations of
 boolean functions using Mc Pitts neuron,
 gradient based learning,
 hidden units,
 architecture design,
 back propagation algorithm,
 Convolution Neural Networks (CNN) - Introduction, data types, efficient
 convolution algorithm,
 activation functions,
 Recurrent Neural Networks (RNN),
 Deep Recurrent Neural Networks (DRNN),
 Long Short Term Memory (LSTM)
Artificial neural network
 Artificial neural networks (ANNs), usually
simply called neural networks (NNs), are
computing systems inspired by the biological
neural networks that constitute animal brains.
 An ANN is based on a collection of connected
units or nodes called artificial neurons, which
loosely model the neurons in a biological brain.
 Each connection, like the synapses in a biological
brain, can transmit a signal to other neurons. An
artificial neuron receives a signal then processes it
and can signal neurons connected to it.
4
 The "signal" at a connection is a real number, and
the output of each neuron is computed by some
non-linear function of the sum of its inputs.
 The connections are called edges.
 Neurons and edges typically have a weight that
adjusts as learning proceeds.
 The weight increases or decreases the strength of
the signal at a connection.
 Neurons may have a threshold such that a signal
is sent only if the aggregate signal crosses that
threshold. 5
 Typically, neurons are aggregated into layers.
 Different layers may perform different
transformations on their inputs.
 Signals travel from the first layer (the input layer),
to the last layer (the output layer), possibly after
traversing the layers multiple times.

6
Artificial neural network Applications
 In image recognition,
 speech recognition
 and natural language processing.
 Recently many papers have been published
featuring AI that can learn to paint, build 3D
Models, create user interfaces(pix2code),

7
Neural Networks
 The definition of a neural network, more properly
referred to as an 'artificial' neural network (ANN),
is provided by the inventor of one of the first
neurocomputers, Dr. Robert Hecht-Nielsen.
 He defines a neural network as:
 "...a computing system made up of a number of
simple, highly interconnected processing elements,
which process information by their dynamic state
response to external inputs."
 Or you can also think of Artificial Neural Network
as computational model that is inspired by the8 way
biological neural networks in the human brain
Biological motivation and connections

 The basic computational unit of the brain is

a neuron.
 Approximately 86 billion neurons can be found in
the human nervous system and they are connected
with approximately 10¹⁴ — 10¹⁵ synapses.

9
 The basic unit of computation in a neural network
is the neuron , often called a node or unit.
 It receives input from some other nodes, or from
an external source and computes an output.
 Each input has an associated weight (w), which is
assigned on the basis of its relative importance to
other inputs.
 The node applies a function to the weighted sum
of its inputs.
 The idea is that the synaptic strengths (the
weights w) are learnable and control the strength
of influence and its direction:
10
 excitory (positive weight) or inhibitory (negative
weight) of one neuron on another.
 In the basic model, the dendrites carry the signal
to the cell body where they all get summed.
 If the final sum is above a certain threshold, the
neuron can fire, sending a spike along its axon.
 In the computational model, we assume that the
precise timings of the spikes do not matter, and
that only the frequency of the firing communicates
information.
 we model the firing rate of the neuron with
an activation function (e.x sigmoid function),
11
which represents the frequency of the spikes along
the axon.
The artificial Neuron
Single neuron
 A singleneuron can be generally described as a
processing unit which gets a stimuli from its
neighbors and respond wrt to a given activation
function.

12
Static Neuron
• Threshold/Binary neuron (simple perceptron)
 First introduced by McCulloch and Pits[1943].
 In that simple model, the single neuron is a binary
threshold unit which computes the weighted sum
of its inputs from other neurons and outputs one or
zero according to its threshold:

where is the output of neuron i at time is the weight of the synapse

(or the edge) between neurons i and j, µi is the units’ threshold value, alternatively
defined (for a biological intuition) as its resting potential.
13
 Linearneuron Here, the single neuron computes
the weighted sum of the inputs from other neurons
and outputs it (activation function is the identity
function):

Differentiable monotonic output (sigmoid) neuron Here, the single neuron

computes the weighted sum of the inputs from other neurons and activates a
sigmoid function

such as Here output of neuron i is:

14
Dynamic neurons (Spiking neurons)
 Adding the time element into the system.
 The summation is the stimulus integration over
time.
 The problem with this kind of system is the
difficulty of analysis.
 However, it is very powerful. It can be simplified
and thus be reduced to a nearly equivalents but
much more simply to analyze systems: recurrent
networks.

15
 Neural Network Architecture
 From the above explanation we can conclude that
a neural network is made of neurons, biologically
the neurons are connected through synapses
where informations flows (weights for out
computational model), when we train a neural
network we want the neurons to fire whenever they
learn specific patterns from the data, and we
model the fire rate using an activation function.

16
Basic neural network architecture

17
Input Nodes (input layer):
 No computation is done here within this layer,
they just pass the information to the next layer
(hidden layer most of the time).
 A block of nodes is also called layer.

18
Hidden nodes (hidden layer):
• In Hidden layers is where intermediate processing
or computation is done, they perform computations
and then transfer the weights (signals or
information) from the input layer to the following
layer (another hidden layer or to the output layer).
• It is possible to have a neural network without a
hidden layer and I’ll come later to explain this.

19
Output Nodes (output layer):
• Here we finally use an activation function that maps to the desired output
format (e.g. softmax for classification).

20
Connections and weights:
 The network consists of connections, each
connection transferring the output of a neuron i to
the input of a neuron j.
 In this sense i is the predecessor of j and j is the
successor of i, Each connection is assigned a
weight Wij.

21
Activation function:
 The activation function of a node defines the
output of that node given an input or set of inputs.
A standard computer chip circuit can be seen as a
digital network of activation functions that can be
“ON” (1) or “OFF” (0), depending on input.
 This is similar to the behavior of the linear
perceptron in neural networks.
 However, it is the nonlinear activation function
that allows such networks to compute nontrivial
problems using only a small number of nodes.
 In artificial neural networks this function is also
22
called the transfer function.
Learning rule
 The learning rule is a rule or an algorithm which
modifies the parameters of the neural network, in
order for a given input to the network to produce a
favored output.
 This learning process typically amounts to
modifying the weights and thresholds.

23
Types of Neural Networks
 There are many classes of neural networks and
these classes also have sub-classes, few are listed
below

24
1. Feedforward Neural Network
 A feedforward neural network is an artificial
neural network where connections between the
units do not form a cycle.
 In this network, the information moves in only
one direction, forward, from the input nodes,
through the hidden nodes (if any) and to the
output nodes.
 There are no cycles or loops in the network.
 We can distinguish two types of feedforward
neural networks:
25
1.1. Single-layer Perceptron
 This is the simplest feedforward neural Network
and does not contain any hidden layer, Which
means it only consists of a single layer of output
nodes.
 This is said to be single because when we count
the layers we do not include the input layer, the
reason for that is because at the input layer no
computations is done, the inputs are fed directly to
the outputs via a series of weights.


26
Simple Perceptron

27
1.2. Multi-layer perceptron (MLP)
 This class of networks consists of multiple layers
of computational units, usually interconnected in a
feed-forward way.
 Each neuron in one layer has directed connections
to the neurons of the subsequent layer.
 In many applications the units of these networks
apply a sigmoid function as an activation function.

28
MLP

29
1.3. Convolutional Neural Network (CNN)

 Convolutional Neural Networks are very similar

to ordinary Neural Networks, they are made up of
neurons that have learnable weights and biases.
 In convolutional neural network (CNN, or
ConvNet or shift invariant or space invariant) the
unit connectivity pattern is inspired by the
organization of the visual cortex, Units respond to
stimuli in a restricted region of space known as
the receptive field.
 Receptive fields partially overlap, over-covering
the entire visual field.
30
 Unit response can be approximated
mathematically by a convolution operation.
 They are variations of multilayer perceptrons that
use minimal preprocessing.
 Their wide applications is in image and video
recognition, recommender systems and natural
language processing.
 CNNs requires large data to train on.


31
32
2. Recurrent neural networks
 In recurrent neural network (RNN), connections
between units form a directed cycle (they
propagate data forward, but also backwards, from
later processing stages to earlier stages).
 This allows it to exhibit dynamic temporal
behavior.
 Unlike feedforward neural networks, RNNs can
use their internal memory to process arbitrary
sequences of inputs.
 This makes them applicable to tasks such as
unsegmented, connected handwriting recognition,
33
speech recognition and other general sequence
Commonly used activation functions
 Every activation function (or non-linearity) takes
a single number and performs a certain fixed
mathematical operation on it.
 Here are some activations functions you will often
find in practice:
 Sigmoid
 Tanh
 ReLU
 Leaky ReLU

34
Course outcome
 able to know the searching strategies of AI
 able to analyze uninformed search strategies

35
3/14/2022
36