: Dr .


1.
2.
2.1
2.2
2.3
2.3.1 (Perceptron)
2.3.2 (MultiLayer Perceptrons,
MLPs)
3.
3.1
3.2
3.3
XOR
4.
4.1
2
4.2 e x
5.
5.1
5.2
- Logistic map
()

1.
,
.
,
, ,
,
.
2.
2.1
(artificial neural networks)

.


.
, , ,
, .., , ,

.

,
. ,

,
(synapses).

(synaptic weights).
,
,
,
.
.

.
.
, .
,
,

3

. ,
.

.
, .

.
.
,
.

,
. , ,
,
.
2.2
,
(artificial neuron)
.
. ,
.
, .
.
,
.

.
1, ,

.

1:

 , ,
,
. ,
:
,
.
,
(bias, thresold),
(net input, induced local field, activation potential).
,
,
.
2 .

2:

,
,
. ,
,
(linear adaptive filter).
.
, ,
(activation function squashing function)
.
,
(step function).
0,
1, >

( ) =

(1)

 3.
,
0 ( ), 1
( ).

3:

McCulloch-Pitts
.
,


,

, .
,
. ,
, .

, ,
.
[0,1] [-1,1].


(logistic function),

( ) =

1+ e

(2)

. ,
.
,
McCulloch-Pitts. 4
6

,
.

4: , = 0.5 (), = 1 () = 2
()

(hyperbolic
function) (arctangent function).
,
. ,
.

.
,

,
. ,
.

,
,

( ) =

(3)

.
,

m

u j = w ji x i
i =1

(4)

y j = ( j )

(5)

j = u j + bj

(6)

m , x i i-
, w ji j
i, u j , b j

, j y j .

2.
3,
(4), (5) (6)
m

j = w ji x i

(7)

y j = ( j )

(8)

x 0 = +1

(9)

w j0 = bj

(10)

i =0

,
(stochastic neuron).
. ,
, .. 0 1
&
1 P ( )
0,
&
P ( )
1,

yj =

(11)

 P ( )

P ( ) =

1+ e /T

(12)

T ,
.
T 0 McCulloch-Pitts.
2.3

,
,
,
..

.

(perceptron),
Rosenblatt 1958.
2.3.1 (Perceptron)
(perceptron) .
,
.
McCulloch-Pitts
. ,
,
5 .

5: (perceptron), 6 4


,
.
.

, . ,

, :

&
& &

w j (n )

y i

&
w j (n + 1) = w j (n ) (n )x j (n )
y j = 1, & 0 (13)
w (n ) + (n )x (n )
&
y j = 0, & 1
j
j
x j (n ) j, y j (n )

w j (n ) n ,

w j (n + 1) n+1

parameter).

(learning-rate

.
Rosenblatt ,
,
.
(perceptron
convergence theorem).
.
Minsky Papert

.
.

XOR, 1.
x1
0
0
1
1

x2
0
1
0
1

x1 XOR x2
0
1
1
0

1: XOR

, 6,

10

 6: XOR

, ,

0, w 1x 1 + w 2x 2
&

1,

y =

(14)

w 1 w 2 .
,
XOR.

(x 1, x 2 ) . 1, (0,0) (1,1)
0, (0,1) (1,0) 1.
= w 1x1 + w 2x 2 (x 1, x 2 )
. w 1 w 2 ,
.
7 , .

. .

.

7: XOR

XOR,
. Minsky Papert,
XOR.

11


.
,
,
.
. ,
,

.
2.3.2 (MultiLayer Perceptrons, MLPs)

(multilayer
perceptron

multilayer
feedforward networks). ,
,
, (hidden layers).
,
8.

8:

, .
,

.
,
,
.
,

12

 (fully connected),
(partially connected).

,
,

.
,

.

XOR.
,
, 9.

2:2:1

9: XOR 2:2:1

w 11 = w 12 = w 21 = w 22 = w 32 = +1

w 31 = 2

b1 = 2

b =b = 1
3
2
2

(15)

. 10,
, .
(1,1) .
(0,0)
.
.
(0,0) (1,1) 0 (0,1) (1,0)
1.

13

 10: XOR , 2:2:1

,
,
.

,
Rosenblatt.
,
.
,

,
.
(error back-propagation algorithm).
3.
3.1

.
. ,
,
,
, .
, ,
,
.

.

14

3.2

, m m0

. ,
, N
,

Data :{x (i ),d (i );i = 1,2,...,N }

(16)

x (i ) = [x 1(i ), x 2 (i ),..., x m (i )]T ,

d (i ) ,

i.

, .
,
,
.
j- , ,
n-

e j (n ) = d j (n ) y j (n )

(17)

H (error energy)
,

E (n ) =

1 n 2
e j (n )
2 j =1

(18)

Eav =

E (n )

(19)

=1

, Eav
(free parameters)
, .
, , Eav

15

 (cost function)
, .

, ,
. ,

,
.
, ,
,
,
.

11: j.

, ,
, ,
. ,
w ji (0) ,

w ji (1) ,

w ji (2) ...,

E (w (n + 1)) E (w (n ))

(20)

w (n ) n w (n + 1)
n+1.
, ,
, w (n )

16

 (steepest descent),

(w (n )) . w ji (n ) ,
j , i,
, n,
w ji (n ) =

(n )
w ji (n )

(21)

,
. (delta rule).
, ,
(n ) .
(n )

.
w ji (n )
,

(n )
(n ) e j (n ) y j (n ) j (n )
=
w ji (n ) e j (n ) y j (n ) j (n ) w ji (n )

(22)

e j (n ) j, y j (n )

j, j (n ) j,
n- ,
11.
, (18),
(n )
= e j (n )
e j (n )

(23)

(17),
e j (n )

y j (n )

= 1

(24)

, 11 . 14,
j (n ) j, n,
k

j = w ji (n )y i (n )
i =0

17

(25)

 k, ,
, y j (n )
j, n,

y j (n ) = j ( j (n ))

(26)

(25) (26),
y j (n )
j (n )

= ( j (n ))

(27)

j (n )

w ji (n )

= y i (n )

(28)

y i (n ) i .
(27),
.

, .
(23), (24), (27) (28) , (21)

w ji (n ) = j (n )y i (n )

(29)

j (n ) =
& j (n ) =

(n )
j (n )

(n ) e j (n ) y j (n )
e j (n ) y j (n ) j (n )

(30)

& j (n ) = e j (n ) j ( j (n ))
(local gradient).
(30), ,
,
, e j (n ) .
,
,
(17).

18

, ,

. , ,
(30) j (n ) , .
,
, ,
.
,
, ,
(30)

j (n ) =
& j (n ) =

(n )
j (n )

(n ) y j (n )
y j (n ) j (n )

& j (n ) =

(31)

(n )
j ( j (n ))
y j (n )

12: k j.

(n )
.
y j (n )

12,

1 mo 2
(n ) = ek (n )
2 k =1

(32)

ek (n ) k-
. , (32) y j (n ) ,

19

e (n )
(n )
= ek (n ) k
y j (n ) k
y j (n )

e (n ) k (n )
(n )
&
= ek (n ) k
y j (n ) k
k (n ) y j (n )

(33)

, 12, . 17,

ek (n ) = d k (n ) y k (n )
& ek (n ) = d k (n ) k (k (n ))

(34)

mh

k = w kj (n )y j (n )

(35)

j =0

mh k,
.
,
, w k 0 .
(34), k (n ) ,

ek (n )
= k (k (n ))
k (n )

(36)

(35), y j (n ) ,
k (n )
= w kj (n )
y j (n )

(37)

(36) (37) (33),

(n )
= ek (n )k (k (n ))w kj (n )
y j (n )
k
(n )
= k (n )w kj (n )
&
y j (n ) k

(38)

30)
k (n ) , k . ,
(38) (31),

20

 j (n )

j (n ) = j ( j (n )) k (n )w kj (n )

(39)

, w ji (n ) ,
j i,

w ji (n ) = j (n )y i (n )

(40)

j (n ) :
1. j ,

j (n ) = e j (n ) j ( j (n ))

(30)

2. j ,

j (n ) = j ( j (n )) k (n )w kj (n )

(39)

k
.
13 ,
j (n ) , k (n )
.

13:

j (n )

, ,
:

21

. 1,
n- .
,
.
...
,
.

.
. 2,
,
.
,
,
...,
.
,
.
,

.
,
,
,
, Eav .
. ,
, ,
. ,
.
.
,
,
.

, ,
(29)

w ji (n ) = j (n )y i (n )

22

(29)

w ji (n ) = w ji (n 1) + j (n )y i (n )

(41)

,
(momentum constant). (41)
(generalized delta rule)
(29), = 0 .
3.3 XOR
,

, XOR. ,
9, 11.
,
.

.

(-1,1).
, ,
.
, 1.

. ,
(epoch) .

.
XR.cpp. ,
, = 0.01
, = 0 . 24776
, ,

w 11 = 1.49731 w 12 = 1.42548 b1 = 1.96077

w 21 = 3.68129 w 22 = 3.30679 b2 = 0.539174

w = 3.32491 w = 3.16934 b = 0.756056
32
3
31

(42)

,



.
2, :

23

( )
0.01
0.1
0.2
0.3
0.4
0.5
0.5
0.5
0.6
0.7

( )

24776
84262
1389
755
518
200.000
302
149
163
111

0.0
0.0
0.0
0.0
0.0
0.0
0.3
0.9
0.0
0.0

2:

2,
,
.
. , ,
, .
.
. ,
,

.
= 0.5 = 0 .
,
3.
x1

x2

x1 XOR x2

0
0
1
1

0
1
0
1

0
1
1
0

3.43184 10-5
0.99998
0.333304
0.666681

3: R

= 0.5

=0

, .
,

, .
,
,

24


.
.

, .
, ,
,
.
,
, . ,
, ,
.
,
, .


.
2, 22,
, ,

. = 0.3 302
, = 0.9 149.


.
.
simulated annealing.
4.
4.1
m0
.

m0 ,
.
,
.
, ,
,
.
,
.

25

 ,

.

(universal approximation theorem),
:
() , .
I m0 [0,1]m0
I m0 C (I m0 ) . , f C (I m0 )

> 0 ,
i , bi w i , j , i = 1,...,m1 j = 1,..., m0 ,

mL

mo

i =1

j =1

F (x 1,..., x m ) = i (w ij x j + bi )
0

(43)

f ,

F (x1,..., x m ) f (x 1,..., x m ) <

0

(44)

x 1, x 2,..., x m0 .

. , ,
,
,
() , ,
. , (43)
, :
1. m0 , x 1,..., x m0

m1 .

2. i, ,
w i 1,...,w im0 bi .
3.
, 1,...,m1 ,

.
.
,

26

 ,
,
.

,
.
2

4.2 e x
2

e x ,
,
.
,
,
4,
xin
-2.7273
-2.1818
-1.6364
-1.0909
-0.54546
0.0

xout
0.00059
0.0086
0.0086
0.3042
0.7427
1.0

xin
0.54546
1.0909
1.6364
2.1818
2.7273
-

xout
0.7427
0.3042
0.0086
0.0086
0.00059
-

4: e x

, = 0.4
, = 0.3 .
,
, [-3,3], 0.01
. 14

2
, e x ,
15, .

27

 14: , e x

15: , e x

,
, .

16 17.

28

 16: ,

17:

,
.
,
.


.
,
,
.

29


, ,
. , ,

, ,
.

.
5.
5.1



.
,
, ,
.
,

.
,
.
m0
.
,
, . ,
,
.
, .
5.2
- Logistic map

,

x n +1 = rx n (1 x n )

(45)

10000 ,
r = 4 . ,

30

 . x n +1
x n , ,
18.

18:

, ,
,
19.

19:

,

.
, ,

31

 .
.

1000 .
20.

20: .
,

,
3-4 ,
.
.
1000 ,
.
21.
,
(
18) .

32

 21: ()
()

,
. 22 x n +1

x n .
,
.

22:

33

 , .
23

23: ,
. ,

,
.
. , 24

.
. , .


.

34

 24: ()
(),

35


Albano, A.M., A. Passamante, T. Hediger and Mary Eileen Farell.
1992. Using neural nets to look for chaos. Physica D 58: 1-9
, (2001). .
.
Bigus, Joseph P and Bigus, Jennifer (). Constructing Intelligent Agents
Using Java. Willey Computer Publishing
Haykin, Simon (1999). Neural Networks, A Comprehensive Foundation.
Prentice Hall International, Inc, New Jersey
Ilachinski, Andrew (2001). Cellular Automata, A Discrete Universe.
World Scientific Publishing Co. Pte. Ltd.
Masters, Timothy (1993). Practical Neural Network Recipies in C++.
Academic Press

36

 ()

MultiLayer FeedForward Network

.
: NeuralNetwork, HiddenLayer
OutputLayer. Layer
.

Class
Layer

Class
Network
Object: nn

Class
HiddenLayer

Class
OutputLayer

Objects: hl[i]

Objects: ol

classes.h
functions.h
. :
NeuralNetwork
void CreateLayers()
HiddenLayer
OutputLayer ,
Layer, .
void Execute(int i)
i-
, Output Layer
.
void BackPropagation(int i)
Execute
( )
Backpropagation .

37

void Train(int i)
Execute BackPropagation i-
.
double CalcError(int k)
(rms)
.
double Validate(int)
validation
validation

( ).
. validation set.

void Anneal(double Tmin, double Tmax, int Tsteps, double tol)

Simulated Annealing.
( )
.
void showTrainResults()


.
Layer
hidden
Output Layer.

void setValues(int, int, int, double**)

Layer

NumOfNeuronsInCurrentLayer,
NumOfNeuronsInPreviousLayer, NumOfNeuronsInNextLayer w
() .
Inputs, Outputs

Layer ,
NetInput .
Layer

.
void CalculateLayerNetInput(double*)

38

 ,
Layer
.

double* Activate(void)

ActivationFunction
Layer.
HiddenLayer
hidden layers hl[i].

void HiddenLayer::ActivationFunction(double*c, double*y)

Activation function hidden layers.
(NetInput)
, .
hidden layers.
Output Layer (. )

OutputLayer::ActivationFunction.
double HiddenLayer::ActivationFunctionPrime(double c)
activation function
BackPropagation.

CalculateSumDelta

LocalGradient

BackPropagation.
OutputLayer
HiddenLayer Layer
.

.
main()
hidden layers, layer,
,
,
NeuralNetwork.
( createLayers())
HiddenLayer OutputLayer (
),
.

39

 train()
,
.
XOR .
validation ,
, validate()

.
, e-x2 logistic map

.

40