This action might not be possible to undo. Are you sure you want to continue?
Networks
Applied Problems:
•Image, Sound, and Pattern
recognition
•Decision making
•
Knowledge discovery
•
ContextDependent Analysis
•
…
Artificial Intellect:
Who is stronger and
why?
NEUROINFORMATICS
 modern theory about principles
and new mathematical models of
information processing, which
based on the biological prototypes
and mechanisms of human brain
activities
Introduction to Neural
Networks
2
Massive parallelism
Brain computer as an
information or signal
processing system, is
composed of a large number
of a simple processing
elements, called neurons.
These neurons are
interconnected by numerous
direct links, which are called
connection, and cooperate
which other to perform a
parallel distributed
processing (PDP) in order to
soft a desired computation
tasks.
Connectionism
Brain computer is a highly
interconnected neurons
system in such a way that
the state of one neuron
affects the potential of the
large number of other
neurons which are
connected according to
weights or strength. The
key idea of such principle is
the functional capacity of
biological neural nets
determs mostly not so of a
single neuron but of its
connections
Associative
distributed
memory
Storage of information in
a brain is supposed to be
concentrated in synaptic
connections of brain
neural network, or more
precisely, in the pattern
of these connections and
strengths (weights) of the
synaptic connections.
A process of
pattern recognition
and pattern
manipulation is
based on:
How our brain
manipulates
with patterns ?
Principles of Brain Processing
3
Human brain
contains a
massively
interconnected net
of 10
10
10
11
(10
billion) neurons
(cortical cells)
Biological
Neuron
 The simple
“arithmetic
computing”
element
Brain Computer: What is it?
4
The schematic
model of a
biological
neuron
Synapse
s
Dendrites
Soma
Axon
Dendrit
e from
other
Axon from
other
neuron
1. Soma or body cell  is a large,
round central body in which almost
all the logical functions of the
neuron are realized.
2. The axon (output), is a nerve
fibre attached to the soma which
can serve as a final output channel
of the neuron. An axon is usually
highly branched.
3. The dendrites (inputs)
represent a highly branching tree of
fibres. These long irregularly shaped
nerve fibres (processes) are
attached to the soma.
4. Synapses are specialized
contacts on a neuron which are the
termination points for the axons
from other neurons.
Biological Neurons
5
?
BrainLike
Computer
Brainlike computer –
is a mathematical model of humane
brain principles of computations. This
computer consists of those
elements which can be called the
biological neuron prototypes, which
are interconnected by direct links
called connections and which
cooperate to perform parallel
distributed processing (PDP) in order
to solve a desired computational
task.
Neurons and
Neural Net
The new paradigm of
computing
mathematics consists
of the combination of
such artificial neurons
into some artificial
neuron net.
Artificial Neural Network –
Mathematical
Paradigms of BrainLike Computer
Brainlike Computer
6
NN as an
model of brain
like Computer
An artificial neural network
(ANN) is a massively parallel
distributed processor that has a
natural propensity for storing
experimental knowledge and
making it available for use. It
means that:
Knowledge is acquired by the
network through a learning
(training) process;
The strength of the
interconnections between
neurons is implemented by
means of the synaptic weights
used to store the knowledge.
The learning process is a
procedure of the adapting the
weights with a learning
algorithm in order to capture the
knowledge. On more
mathematically, the aim of the
learning process is to map a
given relation between inputs
and output (outputs) of the
network.
Brain
The human brain is still
not well understood and
indeed its behavior is
very complex!
There are about 10
billion neurons in the
human cortex and 60
trillion synapses of
connections
The brain is a highly
complex, nonlinear and
parallel computer
(informationprocessing
system)
ANN as a BrainLike
Computer
7
Artificial
Intellect
with Neural
Networks
Intellige
nt
Control
Intellige
nt
Control
Technic
al
Diagnist
ics
Technic
al
Diagnist
ics
Intelligent
Data
Analysis
and Signal
Processing
Intelligent
Data
Analysis
and Signal
Processing
Advanc
e
Robotic
s
Advanc
e
Robotic
s
Machine
Vision
Machine
Vision
Image &
Pattern
Recognitio
n
Image &
Pattern
Recognitio
n
Intellige
nt
Security
System
s
Intellige
nt
Security
System
s
Intellige
ntl
Medicin
e
Devices
Intellige
ntl
Medicin
e
Devices
Intellige
nt
Expert
System
s
Intellige
nt
Expert
System
s
Applications of Artificial Neural
Networks
8
Image Recognition:
Decision Rule and Classifier
•
Is it possible to formulate (and formalize!) the
decision rule, using which we can classify or
recognize our objects basing on the selected
features?
•
Can you propose the rule using which we can
definitely decide is it a tiger or a rabbit?
9
Image Recognition:
Decision Rule and Classifier
•
Once we know our decision rule, it is not
difficult to develop a classifier, which will
perform classification/recognition using the
selected features and the decision rule.
•
However, if the decision rule can not be
formulated and formalized, we should use a
classifier, which can develop the rule from the
learning process
10
Image Recognition:
Decision Rule and Classifier
•
In the most of recognition/classification problems,
the formalization of the decision rule is very
complicated or impossible at all.
•
A neural network is a tool, which can accumulate
knowledge from the learning process.
•
After the learning process, a neural network is
able to approximate a function, which is
supposed to be our decision rule
11
Why neural network?
1
( ,..., )
n
f x x
0 1
( , ,..., )
n
w w w
 unknown multifactor decision
rule
Learning process using a representative learning
set
 a set of weighting vectors is
the result of the learning
process
1
0 1 1
ˆ
( ,..., )
( ... )
n
n n
f x x
P w w x w x
·
· + + +
 a partially defined function,
which is an approximation of
the decision rule function
12
m
p
m
1
m
2
m
3
x
i
y
i
n
ℜ
( ) { ¦ t f
f
p n
≡
ℜ ⇒ ℜ
F
:
p
ℜ
1. Quantization of pattern
space into p decision
classes
Input Patterns
Response:
( )
( )
( )
]
]
]
]
]
]
·
1
1
2
1
1
n
x
x
x
i
x
( )
( )
( )
]
]
]
]
]
]
]
·
1
1
2
1
1
n
y
y
y
i
y
2. Mathematical model of
quantization:
“Learning by Examples”
Mathematical Interpretation of
Classification in Decision
Making
13
Data
Acquisitio
n
Data
Analysis
Interpretation
and
Decision
Making
Signals
&
parameters
Characteristics
&
Estimations
Rules
&
Knowledge
Production
s
Data
Acquisition
Data
Analysis
Decision
Making
Knowledge
Base
Adaptive Machine
Learning
via Neural Network
Intelligent Data Analysis in
Engineering Experiment
14
Selforganization – basic
principle of learning:
Structure reconstruction
Input
Images
Teacher
Neuroprocess
or
Responce
The learning
involves
change of
structure
Learning Rule
Learning via SelfOrganization
Principle
15
Symbol
manipulation
Pattern recognition
Which way
of
imaginatio
n is best
for you ?
Dove flies
Lion goes
Tortoise scrawls
Donkey sits
Shark swims
IllFormalizable
Tasks:
•Sound and Pattern
recognition
•Decision making
•Knowledge discovery
•ContextDependent
Analysis
What is
difference
between
human brain
and traditional
computer via
specific
approaches to
solution of ill
formalizing
tasks (those
tasks that can
not be
formalized
directly)?
Symbol Manipulation or Pattern
Recognition ?
16
Artificial Neuron
w
0
w
0
x
1
w x
1 1
Z=
w
1
w x
i i
∑
ϕ( ) Z
. . . Output
ϕ( ) ( ,. .., ) z f x x
n
·
1
x
n
w
n
w x
n n
A neuron has a set of n synapses
associated to the inputs. Each of
them is characterized by a weight .
A signal at the i
th
input is multiplied (weighted) by
the weight
The weighted input signals are
summed. Thus, a linear
combination of the input signals
is
obtained. A "free weight" (or bias)
, which does not correspond to
any input, is added to this linear
combination and this forms a
weighted sum .
A nonlinear activation function
φ is applied to the weighted sum. A
value of the activation function
is the neuron's output.
, 1,...,
i
w i n ·
, 1,...,
i
n x i ·
1 1
...
n n
w x w x + +
0
w
0 1 1
...
n n
z w w x w x · + + +
( ) y z φ ·
Σ
w
1
w
n
w
2
ϕ
x
1
x
2
x
n
y
17
A Neuron
1 0 1 1
( ,..., ) ( ... )
n n n
f x x F w w x w x · + + +
f is a function to be earned
are the inputs
φ is the activation function
1
x
n
x
1
( , ..., )
n
x f x
.
.
.
φ(z)
0 1 1
...
n n
z w w x w x · + + +
1
, ...,
n
x x
Z is the weighted sum
18
A Neuron
•
Neurons’ functionality is determined
by the nature of its activation
function, its main properties, its
plasticity and flexibility, its ability to
approximate a function to be learned
19
( )
z z φ ·
Linear activation
Threshold
activation
Hyperbolic tangent
activation
Logistic
activation
( ) ( )
u
u
e
e
u tanh u
γ
γ
γ ϕ
2
2
1
1
−
−
+
−
· ·
( )
1
1
z
z
e
α
φ
−
·
+
( )
1, 0,
sign( )
1, 0.
if z
z z
if z
φ
≥ ¹
· ·
'
− <
¹
z
z
z
z
1

1
1
0
0
Σ
ϕ
1
1
Artificial Neuron:
Classical Activation Functions
20
Connectionizm
NN is a highly interconnected structure in such a way
that the state of one neuron affects the potential of the
large number of another neurons to which it is
connected accordiny to weights of connections
Not Programming but Training
NN is trained rather than programmed to perform the
given task since it is difficult to separate the hardware
and software in the structure. We program not solution
of tasks but ability of learning to solve the tasks
]
]
]
]
]
]
11 11 11 11
11 11 11 11
11 11 11 11
11 11 11 11
w w w w
w w w w
w w w w
w w w w
Distributed Memory
NN presents an distributed memory so that changing
adaptation of synapse can take place everywhere in
the structure of the network.
Principles of
Neurocomputing
21
( )
2
x y ϕ ·
Learning and Adaptation
NN are capable to adapt themselves (the synapses
connections between units) to special
environmental conditions by changing their
structure or strengths connections.
NonLinear Functionality
Every new states of a neuron is a nonlinear
function of the input pattern created by the firing
nonlinear activity of the other neurons.
Robustness of Assosiativity
NN states are characterized by high robustness
or insensitivity to noisy and fuzzy of input data
owing to use of a highly redundance distributed
structure
Principles of
Neurocomputing
22
Threshold Neuron
(Perceptron)
• Output of a threshold neuron is binary, while
inputs may be either binary or continuous
• If inputs are binary, a threshold neuron
implements a Boolean function
• The Boolean alphabet {1, 1} is usually used
in neural networks theory instead of {0, 1}.
Correspondence with the classical Boolean
alphabet {0, 1} is established as follows:
1 2 ( 0 1 {0 1) {1 1} } 1 1 1
y
; ; , x =  y  , y , x ∈ · ∈ ⇒ − → →
23
Threshold Boolean
Functions
•
The Boolean function is called a
threshold (linearly separable) function, if it is
possible to find such a realvalued weighting vector
that equation
holds for all the values of the variables x from the
domain of the function f.
•
Any threshold Boolean function may be learned by
a single neuron with the threshold activation
function.
f x x
n
( ,..., )
1
W w w w
n
· ( , ,..., )
0 1
) ... ( ) ,... (
1 1 0 1 n n n
x w x w w sign x x f + + + ·
24
Threshold Boolean Functions:
Geometrical Interpretation
“OR” (Disjunction) is an example of the threshold
(linearly separable) Boolean function:
“1s” are separated from “1” by a line
•
1 1 1
•
1 1 1
• 1 1 1
•
1 1 1
XOR is an example of the nonthreshold (not linearly
separable) Boolean function: it is impossible separate
“1s” from “1s” by any single line
• 1 1 1
• 1 1 1
• 1 1 1
• 1 1 1
(1, 1) (1, 1)
(1,1) (1,1)
(1, 1) (1, 1)
(1,1) (1,1)
25
Threshold Neuron: Learning
• A main property of a neuron and of a neural
network is their ability to learn from its
environment, and to improve its performance
through learning.
• A neuron (a neural network) learns about its
environment through an iterative process of
adjustments applied to its synaptic weights.
• Ideally, a network (a single neuron) becomes
more knowledgeable about its environment
after each iteration of the learning process.
26
Threshold Neuron: Learning
•
Let us have a finite set of n
dimensional vectors that describe
some objects belonging to some
classes (let us assume for simplicity,
but without loss of generality that
there are just two classes and that
our vectors are binary). This set is
called a learning set:
( )
{ ¦
1
,..., ; , 1, 2; 1,..., ;
1, 1
j j j j
n k
j
i
X x x X C k j m
x
· ∈ · ·
∈ −
27
Threshold Neuron: Learning
• Learning of a neuron (of a network) is a
process of its adaptation to the automatic
identification of a membership of all
vectors from a learning set, which is
based on the analysis of these vectors:
their components form a set of neuron
(network) inputs.
• This process should be utilized through a
learning algorithm.
28
Threshold Neuron: Learning
• Let T be a desired output of a neuron
(of a network) for a certain input
vector and Y be an actual output of a
neuron.
• If T=Y, there is nothing to learn.
• If T≠Y, then a neuron has to learn, in
order to ensure that after adjustment
of the weights, its actual output will
coincide with a desired output
29
ErrorCorrection Learning
•
If T≠Y, then is the error .
•
A goal of learning is to adjust the
weights in such a way that for a new
actual output we will have the
following:
•
That is, the updated actual output
must coincide with the desired
output.
T Y δ · −
Y T Y · + ∂ ·
%
30
ErrorCorrection Learning
• The errorcorrection learning rule determines
how the weights must be adjusted to ensure
that the updated actual output will coincide
with the desired output:
• α is a learning rate (should be equal to 1 for
the threshold neuron, when a function to be
learned is Boolean)
( ) ( )
0
0 1 1
0
, ,..., ; ,...,
; 1,...,
n n
i i i
W w w w X
w
w w
x
x n
w
x
i
δ
δ
α
α
· ·
· +
· + ·
%
%
31
Learning Algorithm
• Learning algorithm consists of the sequential
checking for all vectors from a learning set,
whether their membership is recognized
correctly. If so, no action is required. If not, a
learning rule must be applied to adjust the
weights.
• This iterative process has to continue either
until for all vectors from the learning set their
membership will be recognized correctly or it
will not be recognized just for some
acceptable small amount of vectors (samples
from the learning set).
32
When we need a network
•
The functionality of a single neuron is
limited. For example, the threshold
neuron (the perceptron) can not learn
nonlinearly separable functions.
•
To learn those functions (mappings
between inputs and output) that can
not be learned by a single neuron, a
neural network should be used.
33
A simplest network
1
x
Neuron 1
2
x
Neuron 2
Neuron 3
34
Solving XOR problem using
the simplest network
1
x
N1
2
x
N2
N3
3
1
3
3
3
1
1
3
3
) , ( ) , (
2 1 2 2 1 1 2 1 2 1 2 1
x x f x x f x x x x x x ∨ · ∨ · ⊕
35
Solving XOR problem using
the simplest network
# Inputs Neuron 1 Neuron 2 Neuron 3 XOR=
Z
output
Z
output
Z
output
1) 1 1 1 1 5 1 5 1 1
2) 1 1 5 1 7 1 1 1 1
3) 1 1 7 1 1 1 1 1 1
4) 1 1 1 1 1 1 5 1 1
2 1
x x ⊕ ·
) 3 , 3 , 1 (
~
− · W ) 1 , 3 , 3 (
~
− · W ) 3 , 3 , 1 (
~
− · W
1
x x
2
) ( sign z ) ( sign z ) ( sign z
36
Threshold Functions and
Threshold Neurons
• Threshold (linearly separable) functions can be learned
by a single threshold neuron
• Nonthreshold (nonlinearly separable) functions can
not be learned by a single neuron. For learning of
these functions a neural network created from
threshold neurons is required (MinskyPapert, 1969)
• The number of all Boolean functions of n variables is equal to
, but the number of the threshold ones is substantially
smaller. Really, for n=2 fourteen from sixteen functions
(excepting XOR and not XOR) are threshold, for n=3 there are
104 threshold functions from 256, but for n>3 the following
correspondence is true (T is a number of threshold functions
of n variables):
• For example, for n=4 there are only about 2000 threshold
functions from 65536
n
2
2
T
n
n 2
0
2
3 >
→
37
Is it possible to learn XOR, Parity n and
other nonlinearly separable
functions
using a single neuron?
• Any classical monograph/text book on neural
networks claims that to learn the XOR function a
network from at least three neurons is needed.
• This is true for the realvalued neurons and real
valued neural networks.
• However, this is not true for the complexvalued
neurons !!!
• A jump to the complex domain is a right way to
overcome the MiskyPapert’s limitation and to
learn multiplevalued and Boolean nonlinearly
separable functions using a single neuron.
38
XOR problem
x
1
x
2
2 2 1 1 0
x w x w w
z
+ + ·
·
) (z P
B
f(x
1
, x
2
)
1 1 1+i 1 1
1 1 1i 1 1
1 1 1+i 1 1
1 1 1i 1 1
1 − ·
B
P
1 − ·
B
P
n=2, m=4 – four sectors
W=(0, 1, i) – the weighting vector
1
i
i
1
1 ·
B
P
1 ·
B
P
39
Blurred Image Restoration
(Deblurring) and Blur
Identification by MLMVN
40
Blurred Image Restoration (Deblurring)
and Blur Identification by MLMVN
•
I. Aizenberg, D. Paliy, J. Zurada, and
J. Astola, "Blur Identification by
Multilayer Neural Network based on
MultiValued Neurons", IEEE
Transactions on Neural
Networks, vol. 19, No 5, May 2008,
pp. 883898.
41
Problem statement:
capturing
• Mathematically a variety of capturing principles can be
described by the Fredholm integral of the first kind
• where x,t ℝ
2
, v(t) is a pointspread function (PSF) of a
system, y(t) is a function of a real object and z(x) is an
observed signal.
2
2
( ) ( , ) ( ) , , z x v x t y t dt x t · ∈
∫
¡
¡
M
i
c
r
o
s
c
o
p
y
T
o
m
o
g
r
a
p
h
y
∈
P
h
o
t
o
42
Image deblurring: problem
statement
• Mathematically blur is caused by the convolution of
an image with the distorting kernel.
• Thus, removal of the blur is reduced to the
deconvolution.
• Deconvolution is an illposed problem, which results
in the instability of a solution. The best way to solve
it is to use some regularization technique.
• To use any kind of regularization technique, it is
absolutely necessary to know the distorting kernel
corresponding to a particular blur: so it is necessary
to identify the blur.
43
Blur Identification
•
We use multilayer neural network based
on multivalued neurons (MLMVN) to
recognize Gaussian, motion and
rectangular (boxcar) blurs.
•
We aim to identify simultaneously both
blur and its parameters using a single
neural network.
44
Degradation in the frequency
domain:
True Image Gaussian Rectangular Horizontal
Motion
Vertical
Motion
Images and log of their Power Spectra
log Z
45
Examples of training
vectors
True Image
Gaussian Rectangul
ar
Horizontal
Motion
Vertical
Motion
46
Neural Network
5 35 6
Hidden layers Output layer
1
2
n
Blur 1
Blur 2
Blur
N
Training
(pattern) vectors
47
Simulation
{ ¦
1, 1.33, 1.66, 2, 2.33, 2.66, 3 ; τ ∈
Experiment 1 (2700 training pattern vectors corresponding
to 72 images): six types of blur with the following
parameters:
MLMVN structure: 5356
1) The Gaussian blur is considered with
2) The linear uniform horizontal motion blur of the lengths
3, 5, 7, 9;
3) The linear uniform vertical motion blur of the length 3, 5,
7, 9;
4) The linear uniform diagonal motion from SouthWest to
North East blur of the lengths 3, 5, 7, 9;
5) The linear uniform diagonal motion from SouthEast to
North West blur of the lengths 3, 5, 7, 9;
6) rectangular has sizes 3x3, 5x5, 7x7, 9x9.
48
Results
Classification Results
Blur MLMVN,
381 inputs,
5356,
2336 weights in total
SVM
Ensemble from
27 binary decision
SVMs,
25.717.500 support
vectors in total
No blur 96.0% 100.0%
Gaussian 99.0% 99.4%
Rectangular 99.0% 96.4
Motion horizontal 98.5% 96.4
Motion vertical 98.3% 96.4
Motion NorthEast Diagonal 97.9% 96.5
Motion NorthWest Diagonal 97.2% 96.5
49
Restored images
Blurred noisy
image:
rectangular 9x9
Restored
Blurred noisy
image:
Gaussian, σ=2
Restored
50
Introduction to Neural Networks
Artificial Intellect: Who is stronger and why?
Applied Problems:
•Image, Sound, and Pattern recognition •Decision making •Knowledge discovery •ContextDependent Analysis •…
NEUROINFORMATICS  modern theory about principles and new mathematical models of information processing, which based on the biological prototypes and mechanisms of human brain 2 activities
Principles of Brain Processing
How our brain manipulates with patterns ?
A process of pattern recognition and pattern manipulation is based on:
Massive parallelism
Brain computer as an information or signal processing system, is composed of a large number of a simple processing elements, called neurons. These neurons are interconnected by numerous direct links, which are called connection, and cooperate which other to perform a
Connectionism
Brain computer is a highly interconnected neurons system in such a way that the state of one neuron affects the potential of the large number of other neurons which are connected according to weights or strength. The key idea of such principle is the functional capacity of
Associative distributed memory
Storage of information in a brain is supposed to be concentrated in synaptic connections of brain neural network, or more precisely, in the pattern of these connections and strengths (weights) of the synaptic connections. 3
Brain Computer: What is it?
Human brain contains a massively interconnected net of 1010 1011 (10 billion) neurons (cortical cells) Biological Neuron  The simple “arithmetic computing” element
4
Biological Neurons
1. Soma or body cell  is a large,
round central body in which almost all the logical functions of the neuron are realized.
2. The axon (output), is a nerve
fibre attached to the soma which can serve as a final output channel of the neuron. An axon is usually highly branched. Synapse s Axon from other neuron Soma
3. The
dendrites
(inputs)
represent a highly branching tree of fibres. These long irregularly shaped nerve fibres (processes) are attached to the soma.
4. Synapses
are specialized contacts on a neuron which are the termination points for the axons from other neurons.
Axon Dendrites
Dendrit e from other
The schematic model of a biological neuron
5
which are interconnected by direct links called connections and which cooperate to perform parallel distributed processing (PDP) in order ? 6 . This computer consists of those elements which can be called the biological neuron prototypes. BrainLike Computer Brainlike computer – is a mathematical model of humanebrain principles of computations.Brainlike Computer Artificial Neural Network – Mathematical The new paradigm of Paradigms of BrainLike Computer computing mathematics consists Neurons and of the combination of Neural Net such artificial neurons into some artificial neuron net.
The strength of the interconnections between neurons is implemented by means of the synaptic weights used to store the knowledge. The learning process is a procedure of the adapting the weights with a learning 7 .ANN as a BrainLike NN as an Computer An artificial neural network model of brainlike Computer Brain The human brain is still not well understood and indeed its behavior is very complex! There are about 10 billion neurons in the human cortex and 60 trillion synapses of connections (ANN) is a massively parallel distributed processor that has a natural propensity for storing experimental knowledge and making it available for use. It means that: Knowledge is acquired by the network through a learning (training) process.
Applications of Artificial Neural Networks Advanc Advanc e e Robotic Robotic ss Machine Machine Vision Vision Intellige Intellige nt nt Control Control Technic Technic al al Diagnist Diagnist ics ics Artificial Intellect with Neural Networks Intelligent Intelligent Data Data Analysis Analysis and Signal and Signal Processing Processing Image & Image & Pattern Pattern Recognitio Recognitio n n Intellige Intellige ntl ntl Medicin Medicin e e Devices Devices Intellige Intellige nt nt Security Security System System ss Intellige Intellige nt nt Expert Expert System System ss 8 .
using which we can classify or recognize our objects basing on the selected features? • Can you propose the rule using which we can definitely decide is it a tiger or a rabbit? 9 .Image Recognition: Decision Rule and Classifier • Is it possible to formulate (and formalize!) the decision rule.
we should use a classifier. it is not difficult to develop a classifier.Image Recognition: Decision Rule and Classifier • Once we know our decision rule. if the decision rule can not be formulated and formalized. which can develop the rule from the learning process 10 . • However. which will perform classification/recognition using the selected features and the decision rule.
• After the learning process. a neural network is able to approximate a function. which can accumulate knowledge from the learning process.Image Recognition: Decision Rule and Classifier • In the most of recognition/classification problems. the formalization of the decision rule is very complicated or impossible at all. • A neural network is a tool. which is supposed to be our decision rule 11 .
.a set of weighting vectors is the result of the learning process .. w1 .. xn ) = = P ( w0 + w1 x1 + . + wn xn ) ...unknown multifactor decision rule Learning process using a representative learning set ( w0 .. xn ) . wn ) ˆ f ( x1 ..Why neural network? f ( x1 .a partially defined function..... which is an approximation of the decision rule function 12 ....
Mathematical model of quantization: “Learning by Examples” Response: y1( 1) ( 1) y yi = 2 ( 1) yn Input Patterns x1( 1) ( 1) x xi = 2 ( 1) xn 13 .Mathematical Interpretation of Classification in Decision Making 1. Quantization of pattern space into p decision classes f: ℜn⇒ ℜ p xi F ≡ { f ( t)} yi ℜn ℜ p m p m 1 m m 3 2 2.
Intelligent Data Analysis in Engineering Experiment Interpretation and Decision Making Rules & Knowledge Production s Data Analysis Data Acquisitio n Signals & parameters Characteristics & Estimations Adaptive Machine Learning via Neural Network Data Analysis Decision Making Data Acquisition Knowledge Base 14 .
Learning via SelfOrganization Principle Selforganization – basic principle of learning: Structure reconstruction The learning involves change of structure Responce Neuroprocess or Learning Rule Teacher Input Images 15 .
Symbol Manipulation or Pattern Recognition ? IllFormalizable Tasks: •Sound and Pattern recognition •Decision making •Knowledge discovery •ContextDependent Analysis What is difference between human brain and traditional computer via specific approaches to solution of illformalizing tasks (those tasks that can not be formalized directly)? Which way of imaginatio n is best for you ? Symbol manipulation Dove flies Lion goes Tortoise scrawls Donkey sits Shark swims Pattern recognition 16 .
. A signali .. A value of the activation function 17 is the neuron's output.. n x at the ith input is multiplied (weighted) by wi .. + wn . i = 1. x1 x2 xn w1 w2 wn Σ ϕ y .. n Output the weight The weighted input signals are w Thus.Artificial Neuron w0 x1 w1 .. which does not=correspond to xn any input... a linear summed. . is added to this linear combination and this forms a weighted sum y = φ ( z ) .1 x1 + .. A nonlinear activation function φ is applied to the weighted sum. . Each of them is characterized by a weight .... xn ) ( z) f A neuron has a set of n synapses associated to the inputs. A "free weight" (or bias) z w0 + w1 x1 + ... + wn xn w combination of the input signals 0 is obtained.. w0 w1 x1 Z= wx ∑ i i ϕ (Z ) xn wn wn x n ϕ =( x1 .. i = 1.
. ....A Neuron f ( x1 .... xn Z is the weighted sum 18 . . + wn xn ) f is a function to be earned x1 x1 .. xn ) = F ( w0 + w1 x1 + ........ n n are the inputs φ is the activation function . xn ) φ(z) z = w0 + w1 x1 + +w x . x n f ( x1 . .
its ability to approximate a function to be learned 19 .A Neuron • Neurons’ functionality is determined by the nature of its activation function. its plasticity and flexibility. its main properties.
if φ ( z ) = sign( z ) = −1. z < 0. Hyperbolic tangent activation 1 − e − 2γu ϕ ( u ) = tanh( γu ) = 1 1 + e −2γu z 1 0 z 20 .Artificial Neuron: Classical Activation Functions Linear activation φ ( z) = z Logistic activation φ ( z) = 1 1 + e −α z 1 Σϕ Threshold activation 1. if z 0 z z ≥ 0.
Principles of Neurocomputing Connectionizm NN is a highly interconnected structure in such a way that the state of one neuron affects the potential of the large number of another neurons to which it is connected accordiny to weights of connections Not Programming but Training NN is trained rather than programmed to perform the given task since it is difficult to separate the hardware and software in the structure. w11 w 11 w11 w11 w11 w11 w11 w11 w11 w11 w11 w11 w11 w11 w11 w11 21 . We program not solution of tasks but ability of learning to solve the tasks Distributed Memory NN presents an distributed memory so that changingadaptation of synapse can take place everywhere in the structure of the network.
Principles of Neurocomputing Learning and Adaptation NN are capable to adapt themselves (the synapses connections between units) to special environmental conditions by changing their structure or strengths connections. y = ϕ( x 2 ) NonLinear Functionality Every new states of a neuron is a nonlinear function of the input pattern created by the firing nonlinear activity of the other neurons. Robustness of Assosiativity NN states are characterized by high robustness or insensitivity to noisy and fuzzy of input data owing to use of a highly redundance distributed structure 22 .
1}. y ∈{0.1}. a threshold neuron implements a Boolean function • The Boolean alphabet {1. x ∈ {1.Threshold Neuron (Perceptron) • Output of a threshold neuron is binary. 1} is established as follows: 0 → 1. Correspondence with the classical Boolean alphabet {0. 1 → 1.2y = (− 1) y 23 . while inputs may be either binary or continuous • If inputs are binary. 1} is usually used in neural networks theory instead of {0.1} ⇒ x = 1.
... if it is possible to find such a realvalued weighting vector that equation domain of the function f. w1 .xn ) = sign(of0the 1variablesn xnfrom the 24 . x n ) is called a threshold (linearly separable) function.. W = ( w0 ...... wn ) f ( x1 the values w + w x1 + ..Threshold Boolean Functions • The Boolean function f ( x1 .. • Any threshold Boolean function may be learned by a single neuron with the threshold activation function. + w x ) holds for all ...
1) (1.1) • • • • 1 1 1 1 1 1 1 1 1 1 1 1 • 1 1 • 1 1 • 1 1 • 1 1 1 1 1 1 25 . 1) (1.1) (1. 1) (1. 1) (1.Threshold Boolean Functions: Geometrical Interpretation “OR” (Disjunction) is an example of the threshold (linearly separable) Boolean function: “1s” are separated from “1” by a line XOR is an example of the nonthreshold (not linearly separable) Boolean function: it is impossible separate “1s” from “1s” by any single line (1.1) (1.1) (1.
26 . a network (a single neuron) becomes more knowledgeable about its environment after each iteration of the learning process. • Ideally.Threshold Neuron: Learning • A main property of a neuron and of a neural network is their ability to learn from its environment. and to improve its performance through learning. • A neuron (a neural network) learns about its environment through an iterative process of adjustments applied to its synaptic weights.
. 2. X x1 .1.... −1} 27 .. j = 1. but without loss of generality that there are just two classes and that j j our=vectorsj are jbinary). xn ) .Threshold Neuron: Learning • Let us have a finite set of ndimensional vectors that describe some objects belonging to some classes (let us assume for simplicity. X ∈ Ck ... k = This set m ( a learning set: called j xi ∈ { 1.is ..
28 .Threshold Neuron: Learning • Learning of a neuron (of a network) is a process of its adaptation to the automatic identification of a membership of all vectors from a learning set. • This process should be utilized through a learning algorithm. which is based on the analysis of these vectors: their components form a set of neuron (network) inputs.
then a neuron has to learn.Threshold Neuron: Learning • Let T be a desired output of a neuron (of a network) for a certain input vector and Y be an actual output of a neuron. there is nothing to learn. its actual output will coincide with a desired output 29 . • If T≠Y. in order to ensure that after adjustment of the weights. • If T=Y.
the updated actual output must coincide with the desired output.ErrorCorrection Learning • If T≠Y. then T − Y δ= is the error . • A goal of learning is to adjust the weights in such a way that for a new Y have the = actual output we will% Y + ∂ = T following: • That is. 30 .
.... wn ) ....ErrorCorrection Learning • The errorcorrection learning rule determines how the weights must be adjusted to ensure that the updated actual output will coincide with the desired output: W = ( w0 .. n • α is a learning rate (should be equal to 1 for the threshold neuron.. xn ) % w0 = w0 + αδ % wi = wi + αδ xi . X = ( x1 .. when a function to be learned is Boolean) 31 . w1 . i = 1...
• This iterative process has to continue either until for all vectors from the learning set their membership will be recognized correctly or it will not be recognized just for some acceptable small amount of vectors (samples from the learning set). whether their membership is recognized correctly. a learning rule must be applied to adjust the weights. If so. If not.Learning Algorithm • Learning algorithm consists of the sequential checking for all vectors from a learning set. no action is required. 32 .
When we need a network • The functionality of a single neuron is limited. a neural network should be used. For example. • To learn those functions (mappings between inputs and output) that can not be learned by a single neuron. 33 . the threshold neuron (the perceptron) can not learn nonlinearly separable functions.
A simplest network x1 x2 34 .
Solving XOR problem using the simplest network x1 ⊕ x2 = x1 x2 ∨ x1 x2 = f1 ( x1 . x2 ) x1 x2 35 . x2 ) ∨ f 2 ( x1 .
−3.3.−1) Z 5 7 1 1 ~ W = (−1.3) Z 5 1 1 5 x1 1) 2) 3) 4) 1 1 1 1 x2 1 1 1 1 Z 1 5 7 1 sign ( z ) 1 1 1 1 output sign ( z ) 1 1 1 1 output sign ( z ) output 1 1 1 1 = x1 ⊕ x 2 1 1 1 1 36 .3.3) ~ W = (3.Solving XOR problem using the simplest network # Inputs Neuron 1 Neuron 2 Neuron 3 XOR= ~ W = (1.
Really. for n=3 there are 104 threshold functions from 256. but the number of the threshold ones is substantially 2 2 smaller. for n=2 fourteen from sixteen functions (excepting XOR and not XOR) are threshold. For learning of these functions a neural network created from threshold neurons is required (MinskyPapert. for n=4 there are only about 2000 nthreshold functions from 65536 37 . but for n>3 the following correspondence is true (T is a number of threshold functions of n variables): T 0 2n → >3 2 • For example.Threshold Functions and Threshold Neurons • Threshold (linearly separable) functions can be learned by a single threshold neuron • Nonthreshold (nonlinearly separable) functions can not be learned by a single neuron. 1969) • The number of all Boolean functions of n variables is equal nto .
• However. 38 . • This is true for the realvalued neurons and realvalued neural networks.Is it possible to learn XOR. this is not true for the complexvalued neurons !!! • A jump to the complex domain is a right way to overcome the MiskyPapert’s limitation and to learn multiplevalued and Boolean nonlinearly separable functions using a single neuron. Parity n and other nonlinearly separable functions using a single neuron? • Any classical monograph/text book on neural networks claims that to learn the XOR function a network from at least three neurons is needed.
m=4 – four sectors PB = −1 i 1 PB = 1 1 W=(0. i) – the weighting vector z= P (z ) = w0 + w1 x1 + w2 x2 B i PB = 1 PB = −1 x1 x2 39 .XOR problem n=2. 1.
Blurred Image Restoration (Deblurring) and Blur Identification by MLMVN 40 .
Paliy. Astola. D. pp. IEEE Transactions on Neural Networks. May 2008. "Blur Identification by Multilayer Neural Network based on MultiValued Neurons". No 5. 19. Zurada. 883898. J. vol.Blurred Image Restoration (Deblurring) and Blur Identification by MLMVN • I. 41 . Aizenberg. and J.
x∈ .t ℝ2.t 2 ¡ ¡ 2 ∈ • where x. o m h To ap gr y Micr osco py 42 . y(t) is a function of a real object and z(x) is an observed signal.t )y (t )dt .Problem statement: capturing Phot o • Mathematically a variety of capturing principles can be described by the Fredholm integral of the first kind z (x ) = ∫ v(x. v(t) is a pointspread function (PSF) of a system.
• To use any kind of regularization technique. which results in the instability of a solution.Image deblurring: problem statement • Mathematically blur is caused by the convolution of an image with the distorting kernel. The best way to solve it is to use some regularization technique. • Thus. 43 . it is absolutely necessary to know the distorting kernel corresponding to a particular blur: so it is necessary to identify the blur. • Deconvolution is an illposed problem. removal of the blur is reduced to the deconvolution.
Blur Identification • We use multilayer neural network based on multivalued neurons (MLMVN) to recognize Gaussian. • We aim to identify simultaneously both blur and its parameters using a single neural network. motion and rectangular (boxcar) blurs. 44 .
Degradation in the frequency domain: True Image Gaussian Rectangular Images and log of their Power Spectralog Horizontal Motion Z Vertical Motion 45 .
Examples of training vectors True Image Gaussian Rectangul ar Horizontal Motion Vertical Motion 46 .
Neural Network 5356 Training (pattern) vectors 1 Blur 1 Blur 2 2 n Blur N Hidden layers Output layer 47 .
5x5.66. 2.West blur of the lengths 3. 7. 6) rectangular has sizes 3x3.33. 5. MLMVN structure: 5356 1) The Gaussian blur is considered with 2) The linear uniform horizontal motion blur of the lengths 3.Simulation Experiment 1 (2700 training pattern vectors corresponding to 72 images): six types of blur with the following parameters: τ ∈ { 1.East blur of the lengths 3. 9. 3) The linear uniform vertical motion blur of the length 3. 7. 9x9.66. 7x7. 2.33. 5. 1. 2. 1. 9. 7. 5. 4) The linear uniform diagonal motion from SouthWest to North. 9. 5) The linear uniform diagonal motion from SouthEast to North. 3} . 5. 7. 48 . 9.
2336 weights in total 96. 381 inputs.5 96.2% SVM Ensemble from 27 binary decision SVMs.0% 99.0% 99.0% 98.4 96.4% 96. 5356.5% 98.4 96.717.4 96.500 support vectors in total 100. 25.9% 97.Results Classification Results Blur MLMVN.3% 97.0% 99.5 49 No blur Gaussian Rectangular Motion horizontal Motion vertical Motion NorthEast Diagonal Motion NorthWest Diagonal .
σ=2 Restored Restored 50 .Restored images Blurred noisy image: rectangular 9x9 Blurred noisy image: Gaussian.
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue reading from where you left off, or restart the preview.