Thut ton bnh phng trung bnh ti thiu Chng 3

1
B cc chng
Trong chng ny, chng ti m t 1 thut ton hc trc tuyn c a chung cao l thut ton
bnh phng trung bnh ti thiu (LMS), c pht trin bi Widrow v Hoff vo nm 1960.
Chng c b cc nh sau:
1. Mc 3.1 l gii thiu, tip sau l Mc 3.2 t ra cc nn mng cho phn cn li ca
chng bng cch m t 1 b lc ri rc thi gian tuyn tnh ca p ng xung thi gian
hu hn.
2. Mc 3.3 n li 2 k thut ti u khng gii hn: phng php xung dc nhanh nht v
phng php Newton.
3. Mc 3.4 tnh ton b lc Wiener, b lc ti u trong kh nng phn on sai s bnh
phng nh nht. Theo truyn thng, hiu sut trung bnh ca thut ton LMS c nh
gi ngc li vi b lc Wiener.
4. Mc 3.5 gii thiu s bt ngun ca thut ton LMS. Mc 3.6 miu t 1 bin th ca
thut ton LMS nh l 1 m hnh Markov. Sau , chun b cho cch hc cch phn
ng hi t (convergence behavior) ca thut ton LMS, Mc 3.7 gii thiu phng trnh
Langevin, bt ngun t nhit ng lc hc bt nh. Cng c khc cn cho phn tch hi
t ca thut ton l phng php Kushner ca trung bnh tuyt i (direct averaging);
phng php ny c tho lun Mc 3.8. Mc 3.9 gii thiu 1 phn tch thng k chi
tit ca thut ton; quang trng nht l, n ch ra rng cch phn ng thng k (statistical
behavior) ca thut ton (s dng 1 thng s t l hc nh) thc t l dng ri rc thi
gian ca phng trnh Langevin.
5. Mc 3.10 gii thiu 1 th nghim my tnh xc nhn hc thuyt t l hc nh ca thut
ton LMS. Mc 3.11 nhc li th nghim phn loi mu ca Mc 1.5 trn perceptron, lc
ny dng thut ton LMS.
6. Mc 3.12 tho lun cc u v nhc im ca thut ton LMS. Mc 3.13 tho lun vn
tranh ci lin quan ca cc thi kha biu rn luyn t l hc.
7. Mc 3.14 cung cp tm tt v tho lun kt thc chng.
3.1. Gii thiu
Perceptron Rosenblatt, tho lun Chng 1, l thut ton hc u tin gii quyt 1 vn
phn loi mu (c kh nng) phn tch tuyn tnh. Thut ton LMS, c pht trin bi Widrow
v Hoff (1960), l thut ton lc thch ng tuyn tnh u tin gii quyt cc vn nh l d
on v knh thng tin ngang bng. S pht trin ca thut ton LMS c thi thc su sc
bi perceptron. Mc d khc nhau trong cc ng dng, 2 thut ton ny c 1 im chung: c 2
cn s s dng ca 1 b kt hp tuyn tnh, v vy s m t tuyn tnh.
Thut ton bnh phng trung bnh ti thiu Chng 3


2
iu ngc nhin v thut ton LMS l n t thit lp khng ch ng tin cy cho cc ng
dng lc thch ng m cn l tiu chun cho cc thut ton lc p ng khc c nh gi. Cc
nguyn nhn pha sau thnh tch ng ngc nhin ny l rt nhiu:
Trong cc iu kin c s dng my in ton phc tp, s phc tp ca thut ton LMS
l tuyn tnh vi s ch ti cc thng s c th iu chnh c, iu ny lm thut ton
c hiu qu s dng my in ton, lc ny thut ton hiu qu trong s thc thi.
Thut ton n gin vit bng m v do d xy dng.
Trn ht, thut ton mnh m vi s ch n cc nhiu lon bn ngoi.
T 1 cch ci nhn k thut, cc c trng ny tt c ht sc ng m c. V vy khng ngc
nhin thy rng thut ton LMS ng vng vi mi th thch ca thi gian.
Trong chng ny, chng ta xut pht t thut ton LMS trong dng c bn nht ca n
v tho lun cc u v nhc im ca n. Quan trng nht, ti liu trnh by y t nn
mng cho thut ton lan truyn ngc c tho lun chng tip theo.
3.2. Cu trc b lc ca thut ton LMS
Hnh 3.1 minh ha th khi ca 1 h thng ng lc hc cha bit c kch thch bi 1 vec-
t u vo bao gm cc thnh phn x
1
(i), x
2
(i), , x
M
(i), trong i biu th thi gian tc thi ti
tc nhn kch thch c tc ng vo h thng. i = 1, 2, , n. p ng li tc nhn kch
thch ny, h thng xut 1 u ra c biu din bi y(i). V vy, hot ng bn ngoi ca h
thng c m t bi tp d liu
T: {x(i), d(i); i = 1, 2, , n, } (3.1)
trong
x(i) = [x
1
(i), x
2
(i), , x
M
(i)]
T
(3.2)
Mu i kt hp T c phn phi tng t nhau theo 1 quy lut xc sut khng bit.Kch thc
ca M gn lin vi vec-t u vo x(i) c ni n nh l kch thc ca khng gian u vo,
hay n gin l kch thc u vo.
Vec-t tc nhn kch thch x(i) c th xut hin trong 1 hoc 2 cch khc nhau v bn cht, 1
l v khng gian, ci kia l v thi gian:
Thnh t M ca x(i) ban u nhng v tr khc nhau trong khng gian; trong trng hp
ny, chng ta ni x(i) l 1 lu nhanh (snapshot) ca d liu.
Thnh t M ca x(i) i din cho tp hp gi tr hin ti v (M 1) gi tr trc ca
mt vi tc nhn kch thch khng gian ng nht theo thi gian (uniformly spaced in
time).
Thut ton bnh phng trung bnh ti thiu Chng 3


3
Vn chng ta cn x l l thit k nh th no 1 m hnh nhiu u vo 1 u ra ca 1 h
thng ng lc hc cha bit bng cch xy dng n quanh 1 n-ron n tuyn tnh. M hnh
n-ron hot ng di nh hng ca 1 thut ton iu khin cc iu chnh quan trng ti khi
lng thuc k tip hp ca n-ron, vi nhng im lu sau:
Thut ton bt u t 1 ci t ty ca khi lng thuc k tip hp ca n-ron.
Cc iu chnh ti cc khi lng thuc k tip hp trong s phn ng vi cc s bin i
thng k trong hnh ng ca h thng c thc hin trn 1 c s ni tip (v d thi
gian khng c kt hp vo trong kt cu ca thut ton).
Cc s tnh ton ca cc iu chnh ti cc khi lng thuc k tip hp c hon thnh
trong 1 khong thi gian l 1 chu k ly mu di.
Mu n-ron va m t c nhc n nh 1 b lc p ng. Mc d s m t c gii thiu
trong tnh hung ca 1 nhim v c nhn ra r rng nh l 1 nhim v ca h thng nhn
dng, tnh cht ca b lc p ng l ph bin c ng dng rng ri.
Hnh 3.1b m t 1 th lung tn hiu ca b lc p ng. Hot ng ca n bao gm 2 qu
trnh ni tip:
1. Qu trnh lc, bao gm s tnh ton ca 2 tn hiu:
1 u ra, biu din bi y(i), c sinh ra theo s phn ng vi M thnh phn ca vec-t
tc nhn kch thch x(i), l x
1
(i), x
2
(i), , x
M
(i);
1 tn hiu li, biu din bi e(i), thu c bi so snh u ra y(i) vi u ra tng ng
d(i) sinh ra bi h thng khng bit. Trong thc t, d(i) ng vai tr nh l 1 phn ng
mong mun, hay mc tiu, tn hiu.
2. Qu trnh p ng, bao gm s iu chnh t ng ca khi lng thuc k tip hp ca
n-ron ph hp vi tn hiu li e(i).
V vy, s kt hp 2 qu trnh ny lm vic cng nhau to thnh 1 vng hi tip hot ng xung
quanh n-ron, nh c thy Hnh 3.1b.
Bi v n-ron l tuyn tnh, u ra y(i) ging ht trng a phng gy ra (induced local
field), l
y(i) = v(i) =

(3.3)
trong w
1
(i), w
2
(i), , w
M
(i) l M cc khi lng thuc k tip hp ca n-ron, c o ti thi
im i. Trong dng ma trn, chng ta c th biu din biu din y(i) nh 1 tch v hng ca cc
vec-t x(i) v w(i) l
y(i) = x
T
(i)w(i) (3.4)
Thut ton bnh phng trung bnh ti thiu Chng 3


4
trong
w(i) = [w
1
(i), w
2
(i), , w
M
(i)]
T
Ch rng k hiu cho 1 khi lng thuc k tip hp c n gin y bng cch khng
bao gm 1 ch s di dng thm vo xc nh n-ron, bi v chng ta ch c 1 n-ron n
x l. Hnh ng ny c xuyn sut cun sch, bt c khi no 1 n-ron n c nhc n.
u ra c xuyn sut cun sch, bt c khi no 1 n-ron n c nhc n. u ra y(i) ca
n-ron c so snh vi u ra tng ng d(i) nhn c t h thng khng bit ti thi im i.
Thnh c tnh, y(i) khc vi d(i); do , s so snh ca chng dn n kt qu l tn hiu li
e(i) = d(i) y(i) (3.5)
Cch m trong tn hiu li e(i) c s dng iu khin cc iu chnh ti khi lng
thuc k tip hp ca n-ron c xc nh bi hm gi tr c dng ly t thut ton lc
p ng quan trng. Vn ny lin h gn gi vi s ti u ha. N v vy thch hp trnh
by 1 s xem xt li ca cc phng php ti u ha t nhin. Ti liu ny c th p dng khng
ch vi cc b lc p ng tuyn tnh m cn vi mng n-ron thng thng.
3.3. Ti u ha t nhin: 1 s xem xt li
Xem xt 1 hm gi trE(w) l 1 hm lin tc c th phn bit c ca mt vi khi lng (thng
s) khng bit vec-t w. Hm E(w) sp xp cc phn t ca w vo cc s thc. N l 1 phng
php ca cch chn khi lng (phn t) vec-t w ca 1 thut ton b lc p ng v vy n hot
ng trong 1 trng thi ti u. Chng ta mun tm 1 gii php ti u w* tha mn iu kin
E(w*) E(w) (3.6)
iu m chng ta cn gii quyt 1 vn ti u ha t nhin c pht biu di y:
Ti thiu hm gi trE(w) i vi khi lng vec-t w.
iu kin cn i vi s ti u l
E(w*) = 0 (3.7)
trong l ton t gradient,
*

(3.8)
v E(w) l vec-t gradient ca hm gi tr
E(w) = *

(3.9)
Thut ton bnh phng trung bnh ti thiu Chng 3


5
(Php ly vi phn vi s ch ti 1 vec-t c tho lun Ghi ch 1 cui chng ny.)
Mt lp ca cc thut ton ti u ha t nhin m c bit ph hp vi thit k ca cc
b lc p ng c da trn tng ca s k tha lp i lp li cc b (local iterative
descent):
Bt u vi 1 d on ban u biu din bi w(0),to ra 1 chui cc vec-t khi lng
w(1), w(2), , v vy hm gi trE(w) c gim ti mi ln lp li ca thut ton, nh
c trnh by bi
E(w(n + 1)) < E(w(n)) (3.10)
trong w(n) l gi tr c ca vec-t khi lng v w(n + 1) l gi tr mi ca n.
Chng ta hi vng rng thut ton cui cng s hi t ln trn gii php ti u w*. Chng ta ni
hi vng bi v c 1 kh nng r rng rng thut ton s phn ly (v d tr nn khng n nh)
tr khi cc s phng trc c thc hin.
Trong mc ny, chng ta m t 3 phng php ti u ha t nhin da vo tng ca
s k tha lp i lp li (iterative descent) trong 1 dng hoc 1 dng khc (Bertsekas, 1995).
Phng php k tha nhanh nht
Trong phng php k tha nhanh nht (steepest descent), nhng s iu chnh lin tip p dng
ti vec-t khi lng w l trong 1 hng ca k tha nhanh nht (steepest descent), l, trong
1 hng i din vec-t gradient E(w). trnh by thun tin, chng ta vit
g = E(w) (3.11)
Do , thut ton k tha nhanh nht (steepest descent) c chnh thc m t bi
w(n + 1) = w(n) g(n) (3.12)
trong l 1 hng s dng gi l stepsize, hay t l hc, thng s, v g(n) l vec-t gradient
c xc nh ti im w(n). Tip tc t s lp li n ti n + 1, thut ton p dng s sa li
w(n) = w(n + 1) - w(n) (3.13)
= g(n)
Phng trnh (3.13) trong thc t l 1 pht biu chnh thc ca nguyn tc sa li c m t
trong chng m u.
chng minh rng s to thnh cng thc ca thut ton k tha nhanh nht (steepest
descent) tha mn iu kin ca phng trnh (3.10) i vi s k tha lp i lp li (iterative
Thut ton bnh phng trung bnh ti thiu Chng 3


6
descent), chng ta s dng 1 chui khai trin bc nht Taylor quanh w(n) xp x E(w(n + 1))
nh l
E(w(n + 1)) E(w(n)) + g
T
(n)w(n)
s s dng ca chui khai trin c chng minh i vi nh. Thay th phng trnh (3.13)
thnh cc kt qu tng quan gn ng
E(w(n + 1)) E(w(n)) - g
T
(n)g(n)
= E(w(n)) - ||g(n)||
2
iu ny ch ra rng, i vi 1 thng s t l hc dng, hm gi tr c gim khi thut ton
tin hnh t 1 ln lp i lp li ti ln k tip. Cch lp lun c trnh by y l s xp x
trong kt qu cui cng ch ng i vi cc t l hc nh.
Phng php ca k tha nhanh nht (steepest descent) hi t ti gii php ti u w* l
chm. Hn na, thng s t l hc c 1nh hng su sc i vi cch hnh ng hi t ca n:
Khi nh, phn ng nht thi ca thut ton l s tt dn qu (overdamped), trong
ng i cong ny c vch ra bi w(n) theo 1 ng phng trong mt phng W, nh
c minh ha trong Hnh 3.2a.
Khi ln, phn ng nht thi ca thut ton l s tt dn chm (underdamped), trong
ng i cong ny c vch ra bi w(n) theo 1 ng zigzag (dao ng), nh c
minh ha trong Hnh 3.2b.
Khi vt qu 1 gi tr ti hn, thut ton tr nn khng n nh (v d, n phn ly).
Phng php Newton
i vi 1 k thut ti u ha t m hn, chng ta c th da vo phng php Newton, tng c
bn ca n l gim n mc ti thiu php tnh xp x bc 2 ca hm gi tr E(w) xung quanh
im hin ti w(n); s ti thiu ha ny c th hin mi ln lp i lp li ca thut ton. C
th, s dng 1 chui khai trin bc 2 Taylor ca hm gi tr xung quanh im w(n), chng ta c
th vit
E(w(n)) = E(w(n + 1)) - E(w(n))
g
T
(n)w(n) +

w
T
(n)H(n) )w(n)
Nh ni, g(n) l vec-t gradient M 1 ca hm gi tr E(w) c xc nh ti im w(n). Ma
trn H(n) l Hessian m m ca E(w), cng c xc nh ti im w(n). Hessian ca E(w) c
nh ngha bi
(3.14)
Thut ton bnh phng trung bnh ti thiu Chng 3


7
H =

E(w)
=
[

Phng trnh (3.15) yu cu hm gi tr E(w) c th phn bit lin tip gp i vi s ch cc
yu t ca w. Ly vi phn phng trnh (3.14) vi s ch n w, chng ta gim n mc ti
thiu nguyn nhn s thay i E(w) khi
g(n) + H(n) )w(n) = 0
Gii phng trnh ny vi cc kt qu w(n)
w(n) = - H
-1
(n)g(n)
l,
w(n + 1) = w(n) + w(n)
= w(n) - H
-1
(n)g(n)
trong H
-1
(n) l nghch o ca Hessian ca E(w).
Thng thng m ni, phng php Newton hi t tim cn nhanh v khng biu l hot
ng zigzag m i khi tiu biu cho phng php k tha nhanh nht (steepest descent). Tuy
nhin, thc hin phng php Newton, Hessian H(n) phi l 1 ma trn dng xc nh vi
mi n. Khng may, thng thng khng c s m bo no rng H(n) l dng xc nh ti mi
ln lp i lp li ca thut ton. Nu Hessian H(n) khng dng xc nh, s bin ci ca
phng php Newton l cn thit (Powell, 1987; Bertsekas, 1995). Trong bt k trng hp no,
1 hn ch ch yu ca phng php Newton l s phc tp dng my in ton ca n.
Phng php Gauss Newton
gii quyt s phc tp dng my in ton ca phng php Newton m khng lm gim
nghim trng hnh ng hi t ca n, chng ta c th s dng phng php Gauss Newton.
p dng phng php ny, chng ta chp nhn 1 hm gi tr c biu din l tng ca cc
bnh phng li. t
E(w) =

(3.17)
(3.15)
(3.16)
Thut ton bnh phng trung bnh ti thiu Chng 3


8
trong h s t l

c k n n gin ha cc vn trong phn tch sau ny. Tt c cc

s hng li trong cng thc ny c tnh trn c s ca 1 vec-t khi lng w m c c nh
trn ton b khong thi gian quan st 1 .
Tn hiu li e(i) l 1 hm ca vec-t khi lng c th iu chnh c w. Cho 1 im ang hot
ng w(n), chng ta chuyn s ph thuc ca e(i) vo w thnh tuyn tnh bng cch xut 1 s
hng mi
e(i, w) = e(i) + *

(w - w(n)), i = 1, 2, , n
Mt cch tng ng, bng cch s dng k hiu ma trn, chng ta c th vit
e(n, w) = e(n) + J(n) (w - w(n)) (3.18)
trong e(n) l vec-t li
e(n) = [e(1), e(2), , e(n)]
T
v J(n) l Jacobi n m ca e(n):
J(n) =
[

Jacobi J(n) l chuyn v ca ma trn gradient m n , trong
= [ ]
Vec-t khi lng cp nht w(n + 1) by gi c nh ngha bi
w(n + 1) = arg

- (3.20)
S dng phng trnh (3.18) xc nh quy tc bnh phng Euclid ca , chng ta
c

+ e
T
(n)J(n)(w - w(n)) +

(w - w(n))
T
J
T
(n) J(n)(w - w(n))
Do , ly vi phn biu thc ny vi s ch ti w v t kt qu bng 0, chng ta thu c
J
T
(n)e(n) + J
T
(n) J(n) (w - w(n)) = 0
(3.19)
Thut ton bnh phng trung bnh ti thiu Chng 3


9
Gii phng trnh ny vi w, chng ta v vy c th vit, t phng trnh (3.20),
w(n + 1) = w(n) (J
T
(n) J(n))
-1
J
T
(n)e(n) (3.21)
miu t dng thun ty ca phng php Gauss Newton.
Khng nh phng php Newton cn hiu bit v Hessian ca hm gi tr E(w), phng
php Gauss Newton ch cn Jacobi ca vec-t li e(n). Tuy nhin, s lp i lp li Gauss
Newton c th tnh ton c, tch ma trn J
T
(n) J(n) phi khng n nht.
Vi s quan tm thi gian gn y, chng ta nhn ra rng J
T
(n) J(n) lun lun ln hn
hoc bng 0. m bo n khng n nht, Jacobi J(n) phi c hng bc n; l, n hng ca
J(n) trong phng trnh (3.19) phi c lp tuyn tnh. Khng may, khng c s bo m no
rng iu kin ny s lun lun gi vng. hn ch kh nng J(n) l bc khng y , thng
thng cng ma trn cho I vo ma trn J
T
(n) J(n), trong I l ma trn xc nh. Thng s
l 1 hng s dng nh c chon m bo rng
J
T
(n) J(n) + I xc nh dng vi mi n
Trn c s ny, phng php Gauss Newton c thc hin trong dng c b sung nh
w(n + 1) = w(n) (J
T
(n) J(n) + I)
-1
J
T
(n)e(n) (3.22)
Hiu qu ca vic thm s hng I gim dn khi s ln lp i lp li n tng ln. Cng ch rng
phng trnh quy (3.22) l gii php ca hm gi tr c b sung
E(w) =

} (3.23)
trong w(n) l gi tr hin ti ca vec-t khi lng w(i).
Trong cc ti liu v x l tn hiu, vic thm s hng I phng trnh (3.22) c nhc
n nh l ti cho (diagonal loading). Vic thm s hng ny c gii thch bi s khai trin
hm gi tr E(w) theo cch c m t trong phng trnh (3.23), ti chng ta by gi c 2 s
hng (b qua h s t l

):
S hng u tin,

, l tng tiu chun ca cc bnh phng li m ph thuc

vo d liu o to (training data).
S hng th 2 cha quy tc bnh phng Euclid,

, ph thuc vo cu trc
b lc. Trong thc t, s hng ny hot ng nh l b thng bng.
H s t l thng c nhc n nh l 1 thng s theo quy tc, v nguyn nhn s b sung
ca hm gi tr do c nhc n nh l cu trc theo quy tc. Vn ca s lm theo quy
tc c tho lun chi tit trong Chng 7.
Thut ton bnh phng trung bnh ti thiu Chng 3


10
3.4. B lc Wiener
B nh gi bnh phng ti thiu thng thng c tho lun Chng 2, ti cch tip
cn truyn thng ti s gim n mc ti thiu c s dng tm gii php bnh phng
ti thiu t 1 m hnh quan st ca mi trng. ph hp vi thut ng c k tha trong
chng ny, chng ta s nhc n n nh l b lc bnh phng ti thiu. Ngoi ra, chng ta s
ly li cng thc cho b lc ny bng cch s dng phng php Gauss Newton.
Tip theo, chng ta s dng phng trnh (3.3) v (3.4) xc nh vec-t li nh l
e(n) = d(n) [x(1), x(2), , x(n)]
T
w(n) = d(n) X(n)w(n) (3.24)
trong d(n) l vec-t hi p mong mun n 1,
d(n) = [d(1), d(2), , d(n)]
T

v X(n) l ma trn d liu n M,
X(n) = [x(1), x(2), , x(n)]
T

Ly vi phn vec-t li e(n) vi s ch ti w(n) mang li ma trn gradient
e(n) = - X
T
(n)
Do , Jacobi ca e(n) l
J(n) = - X(n) (3.25)
Bi v phng trnh li (3.18) tuyn tnh theo vec-t khi lng w(n), phng php Gauss
Newton hi t trong 1 ln lp i lp li n, nh c trnh by y. Thay phng trnh (3.24)
v (3.25) vo (3.21) thu c
w(n + 1) = w(n) + (X
T
(n) X(n))
-1
X
T
(n)(d(n) X(n)w(n))
= (X
T
(n) X(n))
-1
X
T
(n)d(n)
S hng (X
T
(n) X(n))
-1
X
T
(n) c gi l gi nghch o ca ma trn d liu X(n); l
X
+
(n) = (X
T
(n) X(n))
-1
X
T
(n) (3.27)
V vy, chng ta c th vit li phng trnh (3.26) di dng c ng
w(n + 1) = X
+
(n) d(n) (3.28)
Cng thc ny m t 1 cch thun tin pht biu di y:
(3.26)
Thut ton bnh phng trung bnh ti thiu Chng 3


11
Vec-t khi lng w(n +1)gii quyt vn bnh phng ti thiu, c xc nh trn s
quan st trong khong thi gian n, l tch ca 2 tha s: gi nghch o X
+
(n) v vec-t hi p
mong mun d(n).
B lc Wiener: Dng gii hn ca bnh phng ti thiu,
B lc cho mi trng Ergodic
Cho w
0
biu th dng gii hn ca b lc bnh phng ti thiu khi s ln quan st, n, c cho
php tin ti v cng ln. Chng ta by gi c th dng phng trnh (3.26) vit
w
0
=

X
T
(n)d(n)
=

(3.29)
Gi s by gi vec-t u vo x(i) v hi p mong mun tng ng d(i) c a ra t
1 mi trng Ergodic chung m cng n nh. Chng ta sau c th dng thi gian trung bnh
thay cho trung bnh ton b. Theo nh ngha, dng trung bnh ton b ca ma trn tng quan
ca vec-t u vo x(i) l
R
xx
= E[x(i)x
T
(i)] (3.30)
v do , dng trung bnh ton b ca vec-t tng quan cho gia vec-t u vo x(i) v hi
p mong mun tng ng d(i) l
r
dx
= E[x(i)d(i)] (3.31)
trong E l ton t k vng. Do , di ergodicity gi nh, chng ta by gi c th vit
R
xx
=

v
r
dx
=

V vy, chng ta c th vit li phng trnh (3.29) trong iu kin ca cc thng s tng quan
trung bnh ton b l
w
0
=

r
dx
(3.32)
Thut ton bnh phng trung bnh ti thiu Chng 3


12
trong

l nghch o ca ma trn tng quan R

xx
. Cng thc ca phng trnh (3.32) l
kiu trung bnh ton b ca gii php bnh phng ti thiu c nh ngha trong phng trnh
(2.32).
Vec-t khi lng w
0
c gi l gii php Wiener

cho vn lc tuyn tnh ti u
(Widrow v Stearns, 1985; Haykin, 2002). Do , chng ta c th pht biu:
Vi 1 qu trnh ergodic, b lc bnh phng ti thiu t ti tim cn b lc Wiener khi
s ln quan st tin ti v cng.
Thit k b lc Wiener yu cu hiu bit v thng k bc 2: ma trn tng quan R
xx
ca
vec-t u vo x(n), v vec-t tng quan cho r
dx
gia x(n) v phn hi mong mun d(n). Tuy
nhin, thng tin ny l khng c sn khi mi trng trong b lc hot ng l khng bit.
Chng ta c th x l mi trng nh vy bng cch s dng 1 b lc thch ng tuyn tnh, thch
ng theo cm gic rng b lc c th iu chnh cc thng s t do ca n tng ng vi cc
bin i thng k trong mi trng. Mt thut ton v cng ph bin lm dng iu chnh ny
trong 1 thi gian lin tc l thut ton trung bnh ti thiu, c tho lun tip theo y.
3.5. Thut ton bnh phng trung bnh ti thiu
Thut ton bnh phng trung bnh ti thiu (LMS) c nh hnh gim n mc ti thiu
gi tr tc thi ca hm gi tr,
E () =

(3.33)
trong e(n) l tn hiu li o c ti thi im n. Ly vi phn E () vi s ch ti vec-t
khi lng () thu c

(3.34)
Cng nh vi b lc bnh phng ti thiu, thut ton LMS hot ng vi 1 n-ron tuyn tnh, v
vy chng ta c th biu din tn hiu li nh l

(3.35)
V vy,

v
Thut ton bnh phng trung bnh ti thiu Chng 3


13

S dng kt qu cui ny nh l nh gi tc thi ca vec-t gradient, chng ta c th vit
(3.36)
Cui cng, s dng phng trnh (3.36) cho vec-t gradient trong phng trnh (3.12) vi
phng php k tha nhanh nht (steepest descent), chng ta c th to thnh cng thc thut
ton LMS nh di y:
(3.37)
Cng ng lu rng nghch o ca thng s t l hc c vai tr nh l 1 tiu chun nh gi
ca b nh ca thut ton LMS: chng ta lm cng nh, kh nng ghi nh trong thi gian ngn
ti thut ton LMS ghi nh d liu c s cng di ra. Do , khi nh, thut ton LMS th
hin chnh xc, nhng t l hi t ca thut ton chm.
Thu c phng trnh (3.37), chng ta s dng ti v tr ca w(n) nhn
mnh thc t rng thut ton LMS sinh ra 1 nh gi tc thi ca vec-t khi lng m c th
do vic s dng ca phng php k tha nhanh nht (steepest descent). Nh mt h qu, trong
s dng thut ton LMS chng ta t b 1 c im phn bit ca k tha nhanh nht (steepest
descent). Trong thut ton k tha nhanh nht (steepest descent), vec-t khi lng w(n) theo 1
ng cong r rng trong khng gian khi lng W theo 1 c quy nh. Ngc li, trong
thut ton LMS, vec-t khi lng theo 1 ng cong ngu nhin. V l do ny, thut ton
LMS i khi c nhc n nh l 1 thut ton gradient ngu nhin. Bi v s ln lp i lp li
trong thut ton LMS tin ti v cng, biu din 1 bc ngu nhin (chuyn ng Brown)
v gii php Wiener w
0
. Tuy nhin, iu quan trng cn ch l thc t, khng nh phng
php k tha nhanh nht (steepest descent), thut ton LMS khng cn hiu bit v thng k ca
mi trng. c im ny ca thut ton LMS l quan trng theo 1 cch nh gi trn thc t.
Mt tng kt ca thut ton LMS, da trn phng trnh (3.35) v (3.37), c trnh by
Bng 3.1, minh ha r rng tnh n gin ca thut ton. Nh c ch ra trong bng ny, s
khi chy ca thut ton c thc hin n gin bng cch t gi tr ca vec-t khi lng
.




Thut ton bnh phng trung bnh ti thiu Chng 3


14
BNG 3.1. Tng kt ca thut ton LMS
Mu hun luyn:

Vec-t tn hiu vo = x(n)
Phn hi mong mun = d(n)
Thng s ngi dng chn la:

Khi chy:
t
Vn hnh:
Vi n = 1, 2, , tnh

S biu din biu lung tn hiu ca thut ton LMS
Bng cch kt hp phng trnh (3.35) v (3.37), chng ta c th biu din tin trnh ca vec-t
khi lng trong thut ton LMS nh l

trong I l ma trn n v. S dng thut ton LMS, chng ta nhn ra rng

(3.39)
trong z
-1
l ton t tr n v thi gian (unit-time delay operator), bao hm lu tr (implying
storage). S dng phng trnh (3.38) v (3.39), chng ta do c th miu t thut ton LMS
bng biu lung tn hiu c v trong Hnh 3.3. Biu lung tn hiu ny cho thy rng
thut ton LMS l 1 v d ca 1 h thng phn hi ngu nhin. S c mt ca phn hi c 1 tc
ng mnh ln cch phn ng hi t (convergence behavior) ca thut ton LMS.
3.6. M hnh Markov m t lch ca thut ton LMS t b lc
Wiener
biu din phn tch thng k ca thut ton LMS, chng ta thy n thun tin hn nhiu khi
lm vi vec-t khi lng li, xc nh bi
(n) = w
0
- (3.40)
(3.38)
Thut ton bnh phng trung bnh ti thiu Chng 3


15
trong w
0
l gii php Wiener ti u c xc nh bi phng trnh (3.32) v l c
lng tng ng ca vec-t khi lng c tnh ton bi thut ton LMS. V vy, trong iu
kin ca (n), tha nhn vai tr ca trng thi, chng ta c th vit li phng trnh (3.38) di
dng c ng
(n + 1) = A(n)(n) + f(n) (3.41)
y chng ta c
A(n) =

(3.42)
trong I l ma trn n v. S hng nhiu ph thm trong v phi ca phng trnh (3.41) c
xc nh bi
f(n) =

(3.43)
trong

(3.44)
l c lng li sinh ra bi b lc Wiener.
Phng trnh (3.41) biu din 1 m hnh Markov ca thut ton LMS, vi m hnh c
m t di y:
Trng thi cp nht ca m hnh, c biu din bi vec-t (n + 1), ph thuc vo trng
thi c (n), vi s ph thuc chnh n c xc nh bi ma trn chuyn tip A(n).
S tin trin ca trng thi qua thi gian n b xo trn bi nhiu c pht sinh bn cht
f(n), ng vai tr nh l lc truyn ng.
Hnh 3.4 minh ha 1 s biu din m hnh lung tn hiu vec-t gi tr ca m hnh ny. Nhnh
z
-1
I biu din b nh ca m hnh, vi z
-1
ng vai tr ton t tr n v thi gian (unit-time
delay operator), nh c ch ra bi
z
-1
[(n + 1)] = (n) (3.45)
Hnh v ny nhn mnh s c mt ca hi tip trong thut ton LMS trong 1 dng c ng hn
chnh n trong Hnh 3.3.
M hnh lung tn hiu ca Hnh 3.4 v phng trnh km theo cung cp khung cho phn
tch hi t ca thut ton LMS di gi thit 1 thng s t l hc nh . Tuy nhin, trc khi tip
tc vi phn tch ny, chng ta s ngoi l mt cht gii thiu 2 khi lm sn (building block)
vi mc tiu trong u: phng trnh Langevin, gii thiu mc 3.7, tip theo sau l phng
php trung bnh trc tip Kushner, gii thiu mc 3.8. Vi 2 khi lm sn (building block)
trong tay, chng ta sau s tip tc hc phn tch hi t ca thut ton LMS trong mc 3.9.
Thut ton bnh phng trung bnh ti thiu Chng 3


16
3.7. Phng trnh Langevin: s m t ca chuyn ng Brown