Adaptive Networks

Presentation by:
C. Vinoth Kumar
SSN College of Engineering
 Introduction

Neural network is the information processing paradigm

inspired by biological nervous systems.

It is the large number of highly interconnected

processing elements (neurons) working together.

Neural networks are configured for a specific application,

such as pattern recognition or data classification,
through a learning process (inductive learning).

In a biological system, learning involves adjustments to

the synaptic connections between neurons and it is
same for artificial neural networks (ANNs).
 Adaptive Networks

An adaptive network is a network structure consisting of

a number of nodes connected through directional links.

Each node represents a process unit, and the links

between nodes specify the causal relationship between
the connected nodes.

The learning rule specifies how these parameters

should be updated to minimize the error.

The basic learning rule is the well-known steepest

descent method.
 Adaptive Networks

An Adaptive Network is a network structure whose

overall input-output behaviour is determined by a
collection of modifiable parameters.

Each node performs a static node function on its

incoming signals to generate a single node output.

A node function is a parameterized function with

modifiable parameters.

Each link specifies the direction of flow from one node

to another.
 The Neuron
Bias
b
x1 w1
Activation
Local function
Field
Output
Input
signal
x2 w2 ∑ v ϕ(−) y

M M Summing
function

xm wm
Synaptic
weights
A Feed Forward Adaptive Network
- layered representation
 Adaptive Networks

Adaptive Node: If a node’s parameter set is not empty,

then its node function depends on parameter values;
Square is used to represent adaptive node.

Fixed Node: If a node has an empty parameter set, then

its function is fixed; Circle is used to represent fixed
node.

x
x f y f y
a
y=f(x,a) y=f(x,a)
 Adaptive Networks
– Parameter sharing

x u
g x g u
a a

y v y v
h h
 Adaptive Network

A linear single node adaptive network

x1
f3 x3 x3 = f3(x1, x2; a1, a2, a3)
= a1x1 + a2x2 + a3
x2

Perceptron network

x1 x3 = f3(x1, x2; a1, a2, a3)

= a1x1 + a2x2 + a3
f3 x f4 x4
3
x2 x4 = f4(x3)
 Simple Single Unit Adaptive Networks

The network has 2 inputs, and one

output. All are binary. The output is
1 if W0I0 + W1I1 + Wb > 0
0 if W0I0 + W1I1 + Wb ≤ 0

We want it to learn simple OR:

output 1 if either I0 or I1 is 1

Change the weight by an amount

proportional to the difference
between the desired output and the
actual output
∆ Wi = α * (D-Y) * Ii
 Adaptive Network

A Multilayer Perceptron
 Adaptive Feed Forward Networks

Feed Forward Networks -

• Information flow is unidirectional
• Data is presented to Input layer
• Passed on to Hidden Layer
• Passed on to Output layer
• Information is Distributed
• Good at extracting patterns,
generalisation and prediction
• Parallel processing of data
• Training: Back propagation
Backpropagation for Feed Forward Networks

The procedure of finding the gradient vector(derivative of
an error) in a network structure is generally referred as
backpropagation because the gradient vector is
calculated in the direction opposite to the flow of the
output of each node.

Once the gradient is obtained, a number of derivative-

based optimization and regression techniques are
available for updating the parameters.

If the gradient vector uses simple steepest descent

method, the resulting learning procedure is known as
backpropagation learning rule.
Backpropagation for Feed Forward Networks
Backpropagation for Feed Forward Networks

To use steepest descent to minimize the error, first the
gradient vector is obtained.

The learning rate (α) has the effect in both outputs of

nodes, outputs of network and error.

Therefore, the derivative gradient vector is passed

starting from output layer and going backward layer by
layer until the input layer is reached
Backpropagation for Feed Forward Networks
Backpropagation for Feed Forward Networks
Backpropagation for Recurrent Networks

For recurrent adaptive networks, it is possible to derive
an extended version of the backpropagation procedure
that finds gradient vectors.