Thut ton Q phn tn cho h a tc t

Distributed Q-Learning for Multiagent Systems

T Minh Phng

Abstract: An important issue in the design and

implementation of a multiagent system is to ensure the
coordination of agents activities so that the overall
activities of the system lead to expexted results. One way to
achieve the necessary coordination is allowing agents to
learn how to coordinate themselves. Reinforcement
learning is the most popular automated learning technique
for single agents and has been recently extended for
multiagent systems. This paper presents a distributed
reinforcement learning algorithm for systems with multiple
cooperative agents. The algorithm is an extension of the
popular Q-learning algorithm, in which Q tables are
distributed among agents and are updated independently.
The algorithm was implemented and evaluated in the block
pushing problem with two agents. The experimental results
are promising and demonstrate the efficiency of the
algorithm.

I. T VN
H a tc t (multiagent system) l h thng trong
nhiu tc t t ch tng tc vi nhau thc hin
mt s nhim v no . Yu cu quan trng i vi
h a tc t l tc t phi c kh nng phi hp hnh
ng vi nhau sao cho ton h thng hot ng hiu
qu. C ch phi hp c th ci t sn khi xy dng
tc t di dng cc quy c, cc k hoch lp sn,
cc k thut ng b... [15]. Mt phng php m
bo phi hp khc l khng ci t sn c ch ng
b m tc t t hc cch phi hp hnh ng thng
qua kinh nghim thu c trong qu trnh tng tc
vi nhau. Phng php ny c mt s u im nh
trc quan, cho kt qu n nh. Vn t ra l cn
xy dng cc thut ton hc t ng ph hp vi tnh

cht phn tn v khng ng b vn c ca h a tc

t.
Hc tng cng (reinforcement learning) truyn
thng l k thut hc t ng dng cho mt tc t
ring l. Tc t phi hc cch ra quyt nh hnh ng
thng qua chui cc tng tc vi mi trng v quan
st kt qu. Thng thng bi ton hc tng cng
c cho bi tp cc trng thi mi trng S, tp cc
hnh ng ca tc t A. mi trng thi s, tc t thc
hin hnh ng a. Do kt qu hnh ng a, mi
trng chuyn sang trng thi mi, ng thi tc t
nhn c mt kt qu gi l phn thng hoc tn
hiu tng cng. Nhim v ca tc t l hc cch
hnh ng sao cho tng gi tr phn thng theo thi
gian l ln nht.
Khi tc t cng hot ng trong h thng vi nhng
tc t khc, trng thi mi trng v gi tr phn
thng khng ch ph thuc vo hnh ng ca tc t
m cn ph thuc vo hnh vi ca cc tc t cn li.
Nhim v ca tc t khi l hc cch hnh ng sao
cho ph hp vi vi hnh vi ca cc tc t trong h
thng. K thut hc tng cng truyn thng (cho mt
tc t) khng cho kt qu mong mun trong trng
hp ny: hoc tc t khng tm c hnh vi ti u
hoc hnh vi ch ti u khi tc t khc hnh ng
theo mt cch nht nh. T y xut hin vn m
rng k thut hc tng cng cho h a tc t.
H a tc t c th chia thnh hai loi chnh: h bao
gm cc tc t cng tc v h bao gm cc tc t
khng cng tc. Cc nghin cu v hc tng cng
cho h tc t cng phn bit ring k thut hc cho
hai trng hp ny.

Littman [8], Hu v Wellman [5] m t k thut hc

cho h bao gm cc tc t cnh tranh da trn l
thuyt ton hc tng cng v l thuyt tr chi. Tc
t c m hnh ho di dng cc bn tham gia mt
tr chi, mi bn c gng cc i ha hm tin ch
ca mnh, chnh sch hnh ng ti u c xc nh
di dng cn bng theo Nash. Bowling v Veloso [1]
m t bi ton tng t v xut thut ton WoLF
trong h s hc thay i theo thi gian. Mt thut
ton dnh cho trng hp bi ton c nhiu im cn
bng Nash c xut trong [12].
Trng hp h tc t cng tc c nghin cu
trong [3,16,7, 9]. Trong [3], mi tc t hoc s dng
thut hc tng cng Q mt cch c lp (khng quan
tm n h tc t khc) hoc tin hnh hc cho hnh
ng chung. Mt gii php khc c xut trong
[7,16] trong hm phn thng c phn tn trn
cc tc t. Tc t c th thng bo cho nhau gi tr
phn thng ca mnh, thng qua hc cch phi
hp vi tc t khc. Phng php ny cho kt qu hc
tng i tt mc d i hi tc t trao i thng tin
vi nhau.
Ngoi cc nghin cu v l thuyt, k thut hc cho
h tc t c nghin cu p dng cho nhiu bi
ton c th. Cc ng dng c bit n nhiu nht l
hc iu khin nhm cu thang my [4], nh tuyn
trong mng [2], iu khin li in [9], phn phi
knh truyn cho mng in thoi di ng [11]. Kt
qu cho thy hc tng cng c th m rng vi cc
mc khc nhau cho h thng a tc t v thch hp
vi nhiu ng dng thc t.
Trong bi bo ny, i tng nghin cu c gii
hn l h thng vi cc tc t cng tc. Bi bo trnh
by thut ton cho php phn tn qu trnh hc trn
cc tc t. Thut ton da trn c s thut ton hc Q,
mt trong nhng thut ton hc tng cng c s
dng nhiu nht. Ni dung c bn ca thut ton l
mi tc t lu tr v cp nht mt bng Q ring cho
mnh. Tc t ch cn c thng tin v gi tr phn
thng chung v trng thi chung ca h thng m
khng cn thng tin v hnh ng ca tc t khc.
Thut ton cho php tit kim chi ph lin lc gia tc

t trong khi vn cho php hc c hnh vi ti u.

Thut ton c ci t v th nghim cho bi ton
trong hai tc t cng tc vi nhau di chuyn mt
vt ti mt v tr cho trc. Kt qu th nghim cho
thy tc t hc c cch phi hp hnh ng cn
thit ng thi c thi gian hc ngn hn so vi thut
ton Bucket Brigade c s dng trong [10].
Cc phn cn li c b cc nh sau. Phn II trnh
by m hnh qu trnh quyt nh Markov m rng
c s dng m hnh ho bi ton hc t ng
cho h a tc t. Phn III m t thut ton hc Q phn
tn cho cc tc t cng tc. Phn IV gii thiu bi
ton di chuyn vt nng bi nhiu tc t v trnh by
kt qu th nghim thut ton cho bi ton . Phn V
l kt lun ca bi bo.
II. PHT BIU BI TON HC T NG
CHO H A TC T
Bi ton hc t ng cho h a tc t c th pht
biu v phn tch trn c s kt hp m hnh qu trnh
quyt nh Markov (Markov Decission Process
MDP) v l thuyt tr chi.
Qu trnh quyt nh Markov (MDP) l m hnh
h thng vi mt tc t v nhiu trng thi. MDP
c nh ngha di dng b bn (S,A,T,R) trong
S l tp cc trng thi, A l tp cc hnh ng ca tc
t, T l hm chuyn tip T : S A S [0,1] , v R
l hm thng R : S A R . Hm chuyn tip T
xc nh xc sut chuyn tip sang trng thi tip theo
tu thuc vo trng thi hin thi v hnh ng ca
tc t. Trong trng hp T ch c th bng 0 hoc 1 h
thng c gi l xc nh (deterministic). Hm
thng R xc nh gi tr phn thng hay li ch m
tc t nhn c khi thc hin mt hnh ng i vi
trng thi cho trc. M hnh c gi l Markov nu
hm chuyn tip ch ph thuc vo trng thi v hnh
ng hin thi ca tc t m khng ph thuc vo
trng thi v hnh ng trc . Yu cu ca hc
tng cng i vi MDP l tm ra chnh sch hnh
ng : S A cho php la chn hnh ng tu
thuc vo trng thi sao cho tng phn thng nhn

c trong tng lai l ln nht. Tng phn thng

c tnh bi:

r
t =0

trong l yu t chit khu (discount factor).

ngha ca yu t chit khu l phn thng nhn c
trong tng lai gn c gi tr hn trong tng lai xa.

vi tc t trong tng lai l ln nht.

i tng nghin cu trong bi bo ny l h cc
tc t cng tc. Hm phn thng chung ca h thng
c xc nh nh tng cc hm phn thng thnh
phn (ca tng tc t). Mc tiu ca hc tng cng
l xc nh chnh sch hnh ng cho php cc i
ho hm phn thng chung ny. Gi l chnh
sch chung ca h thng v c to thnh t cc

Tr chi ma trn (matrix game). Khc vi MDP,

tr chi ma trn l m hnh h thng vi mt trng
thi duy nht v nhiu tc t. Tr chi ma trn c

chnh sch ring ca tc t ( s ) = ( 1 ( s ),..., n ( s )) ,

cho bi ( n, A1.. n , R1.. n ) , trong n l s lng tc t

thng theo thi gian t

cn tm sao cho tng phn thng chung ca h

t =0

(ngi chi), Ai l tp hnh ng m tc t i c th

R( st , ( st )) l ln nht.

la chn, Ri l hm thng ca tc t i Ri : A R ,

III. THUT TON

y A l tp cc hnh ng chung ca tt c tc t

1. Thut ton hc Q tp trung

tham gia tr chi A = A1 A2 ... An . Mi tc t

la chn mt hnh ng trong tp hnh ng cho
php ca mnh v nhn c phn thng tu thuc
vo hnh ng chung ca tt c tc t. Hm Ri
thng c cho di dng ma trn v do vy m
hnh c tn l tr chi ma trn. y cn phn bit
hai trng hp: h thng bao gm cc tc t cng tc
v h thng bao gm tc t cnh tranh. i vi tc t
cng tc, hm phn thng Ri l nh nhau cho tt c
tc t v tc t hnh ng cc i ho hm phn
thng chung ny. Trong trng hp cnh tranh, hm
phn thng ca tc t l khc nhau.
M hnh MDP m rng. Trong trng hp tng
qut, h thng c th c nhiu trng thi khc nhau v
bao gm nhiu tc t. biu din bi ton hc tng
cng cho h thng nh vy, hai m hnh trn c
m rng bng cch kt hp vi nhau [1]. M hnh
MDP m rng c cho bi b ( n, S , A1... n , T , R1... n ) ,
trong cc thnh phn c nh ngha tng ng
nh trong hai m hnh trn. M hnh ny khc vi
MDP ch hnh ng c xc nh bi nhiu tc t
khc nhau v hm phn thng ph thuc vo hnh
ng chung ca ton b tc t. Yu cu t ra i vi
qu trnh hc l xc nh chnh sch hnh ng cho
tng tc t i : S A sao cho tng phn thng i

Thut hc Q (Q-learning) [14] l mt trong nhng

thut ton hc tng cng c s dng nhiu nht
cho MDP. Ni dung c bn ca thut hc Q l s
dng khi nim gi tr tng cng Q ( s, a ) . Q ( s, a )
c nh ngha nh gi tr tng cng (c tnh ti
chit khu) nhn c nu thc hin hnh ng a
trong trng thi s. Gi tr Q ( s, a ) c khi to ngu
nhin (chn hn bng 0) cho tt c cc i (s,a) v
c cp nht theo quy tc quy:
Q ( s, a ) = (1 )Q ( s, a ) + ( R ( s, a ) + max Q (s ', a ')) (1)
a'

trong s l trng thi trc khi chuyn tip, a l

hnh ng c chn, R(s,a) l phn thng nhn
c tc th nh hnh ng a, s l trng thi tip
theo, a l hnh ng tip theo, l h s hc ,

0 1 v xc nh mc thay i gi tr Q sau
mi bc hc. Nu mi hnh ng c thc hin
mt s ln v hn cho mi trng thi th gi tr Q s
hi t ti gi tr ti u Q*. Chnh sch ti u * khi
c xc nh t Q* nh sau:

* ( s ) = arg max Q * ( s, a )

(2)

2. Thut ton hc Q tp trung cho h a tc t

Thut ton Q c xy dng cho trng hp mt
tc t. i vi h thng bao gm nhiu tc t, cch
n gin nht l tin hnh hc tp trung. Qu trnh

hc c thc hin tp trung trn mt tc t duy nht.

Mi hnh ng ca tc t hc l mt hnh ng trong
khng gian hnh ng chung A. Sau khi tm c
chnh sch hnh ng chung, chnh sch ny s c
chia thnh cc chnh sch thnh phn v gi cho tc t
tng ng.
Thut ton hc Q tp trung cho h a tc t c
m t theo quy tc sau

thng chung R v trng thi chung s ca ton h

thng.
Do mi tc t ch phn bit c hnh ng ca
mnh, thay v s dng bng Q chung, mi tc t i s
lu tr v cp nht bng Qi ring. Gi tr trong bng
Qi ring ph thuc vo trng thi s ca h thng v
hnh ng ring ai ca tc t. Vn t ra l xc
nh gi tr Qi nh th no v gi tr quan h vi

Khi to Q ( s, a ) = 0

gi tr trong bng Q trung tm ra sao. Vic la chn

Cp nht

gi tr Qi quyt nh thut ton v cht lng hc

Q( s, a) = (1 )Q( s, a) + ( R( s, a) + max Q(s ', a ')) (3) phn tn.

a'
trong a , a ' A , A = ( A1 A2 ... An ) .
Thut ton hc Q tp trung rt ging thut hc Q
truyn thng. C th, nu mi cp (s,a) c xut
hin vi s ln v hn th gi tr Q s hi t.
Nhc im ln nht ca gii php hc tp trung l
thng tin v trng thi, hnh ng cng nh hm phn
thng phi c thu thp v x l tp trung. Vic x
l tp trung nh thng lm gim tnh n nh v
tin cy ca h thng, gy tc nghn nt (tc t)
trung tm, tng chi ph truyn thng gia tc t ng
thi khng tn dng c u th v tc ca x l
song song. Ngoi ra, vic thu thp thng tin tp trung
i hi tc t phi hiu c cch biu din hnh
ng ca tc t khc. Yu cu ny c th khng tho
mn i vi h thng bao gm cc tc t hn tp. Do
vy, yu cu t ra l xy dng thut ton hc phn
tn sao cho tng tc t c th hc chnh sch hnh
ng ca mnh vi t yu cu trao i thng tin nht.
3. Thut ton hc Q phn tn
Trong trng hp thut hc Q tp trung, thut ton
hc lu tr mt bng cc gi tr Q cho tng cp (s,a),

(a = (a1 ,..., an )) . Tuy nhin, khi tc t tin hnh hc

c lp vi tc t khc, tng tc t khng th bit ht
cc t hp hnh ng chung a m ch c kh nng
phn bit cc hnh ng ca ring mnh. Mc tiu ca
thut ton hc phn tn l cho php tng tc t hc
da trn hnh ng ca ring mnh m khng cn ti
thng tin v hnh ng ca tc t khc. y ta vn
gi thit l mi tc t u bit c gi tr hm phn

Theo phng php n gin nht [3] , mi tc t i

t cp nht bng Qi ring m khng quan tm ti tc
t khc theo cng thc (4).

Qi ( s, ai j ) =

a = ( a1 ,..., an ), ai = ai j

(Pr(a | ai )( R( s, a)
+ max Qi ( s ', ai ')))

(4)

ai ' Ai

trong Pr( a | ai ) l xc sut ca hnh ng

chung a khi tc t i chn hnh ng ai trong trng thi
s. Xc sut ny c th tnh bng cch ghi li tn s
xut hin a cng vi ai trc . Tuy nhin, iu ny
i hi tc t phi c thng tin v hnh ng chung
(khng m bo yu cu phn tn). Ngoi ra thut
ton ny khng cho php tm ra chnh sch hnh ng
ti u trong nhiu trng hp [3].
Di y l phng php cho php tnh gi tr Qi
m khng cn thng tin v hnh ng chung. Ni
dung c bn ca phng php ny l vi mi trng
thi s v hnh ng aij, tc t i ch cp nht gi tr Qi
tng ng nu gi tr Qi mi ln hn gi tr c. Ni
cch khc, mi tc t ch ghi li gi tr tng cng cho
hnh ng chung tt nht ng vi mi trng thi v
hnh ng ring ca mnh cho ti thi im hin ti.
Gi tr Qi ring ca tc t khi c cp nht theo
quy tc (5).

max{Qi ( s, ai j ), R ( s, ai j )

Qi ( s, ai ) =
j
Qi ( s ', ai ')}, (ai Ai )
+ max
'

a
A
i
i

(5)

thi im bt u, Qi ( s, ai j ) c khi to bng 0.

Nu mi cp trng thi hnh ng ring ( s, ai ) xy

ra nhiu th c th d dng chng minh bng quy

np l vic cp nht theo quy tc (5) cho kt qu sau

Qi ( s, ai j ) =

max

a =( a1 ,...,an ),ai = ai j

Q ( s, a )

(6)

Tc l bng Qi ring cha gi tr ln nht ca bng

Q chung ng vi mi hnh ng ring aij ca tc t.
T y c th thy bng Q ring c cp nht theo
(5) s cha gi tr Q* ti u ca bng Q chung c
tnh theo (4) nu mi i trng thi hnh ng
chung xy ra nhiu.
V d. minh ho cho thut ton trn, xt v d
n gin sau [3,5]. V d cho di dng tr chi bao
gm hai tc t cng tc, mt trng thi duy nht s0 , v

u cn b sung kh nng phi hp hnh ng ca tc

t. Trc ht, do tt c cc i trng thi - hnh ng
u lp li vi mt s ln ln, s xut hin hnh
ng ng vi chnh sch ti u. Nu thut ton hc
ghi li hnh ng tt nht cho n thi im hin ti
th khi kt thc qu trnh hc, hnh ng tt nht c
ghi li cho mi trng thi cng chnh l chnh sch ti
u. Nh vy, tc t tin hnh cp nht chnh sch i
ng thi vi cp nht bng Qi ch khng xc nh
chnh sch mt ln duy nht sau khi hc xong theo
cng thc (2). Thut ton hc phn tn (5) khi
c m rng thnh (Hnh 1):

ba hnh ng ring Ai = {ai1 , ai 2 , ai 3}, i = 1, 2 (c

(H s t trn c thm vo tin phn bit cc

bc thi gian ca qu trnh hc).

th tng qut ho d dng cho trng hp nhiu trng

thi v nhiu tc t). Hm phn thng c cho di
dng ma trn nh sau

Tr li v d trn. Do hnh ng chung (a11,a21)

v (a13,a23) cho gi tr Qi (i=1,2) ln nht, trong qu
trnh hc, tu thuc vo (a11,a21) hay (a13,a23) xut

a11
A12
a13
A2
10
0
k
A22
0
2
0
3
A2
k
0
10
trong k l tham s sao cho k<10. S dng quy tc
(5), tc t s xy dng bng Q ring nh sau
1

Q1(s0,ai)
Q2(s0,ai)
Nu s dng xc
t bng Q ring

i * ( s ) = arg max a

ai
ai
ai
10
2
10
10
2
10
nh chnh sch hnh ng ti u
da trn cng thc (2) ta c
Ai

Qi ( s, ai j ) . Theo cng thc ny

c th tm c hai chnh sch ring cho mi tc t

1* ( s0 ) = {a11 , a13} v 2* ( s0 ) = {a21 , a2 3} . T y

c th suy ra bn chnh sch chung (a11,a21), (a13,a23),
(a11,a23) v (a13,a21). D dng nhn thy ch c hai
chnh sch u l ti u v cho phn thng l 10
trong khi hai chnh sch sau khng ti u vi gi tr
phn thng nh hn 10. Nguyn nhn ca vic xut
hin cc chnh sch chung khng ti u l do ty tng
tc t tm c hnh ng thnh phn ti u nhng
khng phi hp c vi hnh ng tng ng ca
tc t khc.
m bo thut ton hc hi t chnh sch ti

hin trc, thut ton s lu li 1 ( s ) = a11 v

2 ( s ) = a21 hay 1 ( s ) = a13 v 2 ( s ) = a2 3 . y

chnh l chnh sch hnh ng ti u cn tm.
IV. TH NGHIM - BI TON DI CHUYN
VT NNG
kim nghim thut ton trnh by trong phn
trc, thut ton c ci t v th nghim cho bi
ton di chuyn vt nng [10]. Trong bi ton ny, hai
tc t (rbt) c phn cng di chuyn mt vt nng
t im xut pht S n im ch G theo mt qu
o P trn mt mt phng (c biu din trn h to
cc) nh trn Hnh 2.
Cc tc t c m phng bng phn mm. Hai tc
t khng bit trc kh nng hnh ng ca tc t kia
v cn hc cch thch ng vi nhau. Mi tc t u c
kh nng nh gi khong cch t v tr hin ti ca
vt cn di chuyn n qu o cn i (mt pht biu
khc kh hn ca bi ton l tc t ch c kh nng
phn bit v tr hin ti l ch cha, tuy nhin pht
biu ny i hi thi gian hc lu hn v khng c
cp n y). Ti mi thi im, mi tc t i

(i=1,2) c th tc ng vo vt nng mt lc Fi ,

hnh ng ca tc t. Mt vng bt u t
v tr xut pht S v kt thc nu xy ra mt
trong ba tnh hung sau: 1) tc t y c
vt nng ti ch; 2) vt nng b y ra khi
khng gian cho php; 3) s hnh ng vt
qu mt gii hn cho trc m khng xy
ra hai tnh hung trc. Vic hn ch s
lng hnh ng cho php trnh tnh trng
tc t khng lm g c khi vt nng nm
trn qu o mong mun nhng cha t
ti ch.

Vi mi tc t i
Ti thi im 0:
Khi to

Qi 0 ( s, ai j ) = 0

0 ( s) A

i
Khi to ngu nhin i
Ti thi im t:
Thc hin hnh ng aij
Cm nhn trng thi mi trng s

Xc nh gi tr phn thng
Cp nht

R( s, ai j )

Qi t +1 ( s, ai j ) = max{Qi t ( s, ai j ), R ( s, ai j ) + max Qi t ( s ', ai ')}

ai 'Ai

t +1

Nu

khng gian cho php

max
Qi ( s, ai ) > max
Qi ( s, ai )
j
j
j

ai Ai

th cp nht

v tr ch G

ai Ai

i ( s ) = ai j
Hnh 1. Thut ton hc Q phn tn

qu o
c th

qu o
tt nht P

r
0 | Fi | Fmax di mt gc i . Tng hp tc ng
ca hai tc t c xc nh bi vct lc

r r r
F = F1 + F2 . Lc ny lm vt nng dch chuyn theo
r
trc x v y khong cch tng ng l | F | cos( ) v
r
| F | sin( ) n v. Gi s v tr hin ti l ( x, y ) ,

Px ( y ) l ta x ca qu o P ti cng tung y,
x =| x Px ( y ) | l khong cch theo trc x gia vt
nng v qu o P. Khi phn hi (phn thng) m
mi tc t nhn c cho hnh ng cui cng l

K * a x . Gi tr K v a c chn ging nh trong

[10]: K=50, a=1.15. n gin, cc i lng c gi
tr lin tc nh lc, gc, khong cch c ri rc ho
bng cch chia u thnh nhng khong bng nhau.

Trng thi ca bi ton c xc nh bi v tr ca

vt nng. D dng nhn thy, t mt v tr (trng thi)
hin ti, mt hnh ng chung c th lun dn n
mt v tr (trng thi) xc nh khc. Nh vy, bi
ton di chuyn vt nng c dng MDP xc nh nh
nh ngha phn II.

S
Tc t 1

vt nng
Tc t 2

Hnh 2: Bi ton di chuyn vt nng vi hai agent

Nhim v ca tc t l hc cch cng nhau di

chuyn vt nng ti ch sau khi lp li cc vng hc.
Trong qu trnh hc, cc tham s nh K, c gi
nguyn khng thay i. Qu trnh hc kt thc khi tc
t di chuyn vt nng thnh cng trong N vng lin
tip (N c chn bng 10) hoc khi s vng hc
vt qua mt ngng nht nh (1500). Mc d bi
ton c m t v thc nghim ch gm hai tc t, c
th m rng bi ton cho trng hp vi nhiu tc t
hn.
V tr bt u v kt thc c la chn ging nh
trong [10] tng ng l (40,0) v (40,100). Gc tc
ng ca lc c chia thnh 11 khong ri rc,
cng lc tc ng c chia thnh 10 khong
bng nhau. H s chit khu c chn =0,9. Vi cc
tham s c la chn nh vy, khng gian trng thi
s bao gm 104 trng thi. Kch thc khng gian
trng thi trong trng hp ny l tng i nh v

do vy trong qu trnh ci t thut ton, gi tr Q

c biu din di dng bng. Trong trng hp
khng gian trng thi ln hn, c th s dng cc
phng php biu din bng Q di dng rt gn nh
dng mng nron hay cy hi quy (regression tree)
[6].

khi thut ton hi t. Mt thng s na lin quan n

qu trnh hc l khong cch trung bnh t qu o
thc n qu o mong mun. Gi tr cc tiu ch trn
c xc nh bng cch ly gi tr trung bnh qua
100 ln thc hin thut ton. Gi tr Q c coi l hi

Mt vn quan trng trong hc tng cng l tc

t cn kho st s lng ln trng thi v hnh
ng. gii quyt vn ny, ti mi trng thi, tc
t la chn hnh ng (hng v cng tc ng)
ngu nhin theo mt phn b xc sut no . C hai
phng php chnh. Phng php th nht chn mt
trong s cc hnh ng cho php vi xc sut bng
nhau. Phng php ny khng quan tm n kinh
nghim trc . Phng php th hai tnh xc sut
ca hnh ng tip theo ph thuc vo gi tr ca Q.
Theo phng php ny, tc t chn hnh ng tt
nht ti thi im (hnh ng c gi tr Q tng
ng ln nht) vi xc sut p v chn cc hnh ng
khc vi xc sut (1-p). Xc sut p thng c iu
tng dn theo thi gian do tc t c thm nhiu
kinh nghim. Mt phng php thng dng loi ny
l phng php Boltzman c s dng cho thc
nghim trnh by trong bi bo ny. Theo phng
php kho st theo Boltzman (Boltzman exploration):
hnh ng a c chn vi xc sut

t nu Q t +1 Q t vi l mt s dng nh tu
Trong [10], bi ton di chuyn vt nng vi hai tc
t c gii quyt bng cch s dng thut ton hc
tng cng Bucket Brigade (BB) cho h a tc t. Kt
qu th nghim thut ton trnh by trn c so
snh vi kt qu thut ton BB nh trn th Hnh
3.
Gi tr theo trc tung l khong cch trung bnh t
qu o thc n qu o mong mun. Gi tr theo
trc honh l s lng vng hc ca tc t. Kt qu
thc hin Bucket brigade ly t [10] vi h s hc
=0,6 v cc tham s khc nh K, a, tham s dng
ri rc ho, kch thc khng gian hc, hn ch v s
bc trong mi vng c la chn ging nh trnh
by trn.
Thut ton trnh by trn hi t sau khong 50
vng. Sau khi hi t, tc t hnh ng rt n nh v
di chuyn vt nng theo ng qu o mong mun.

T l tham s v c gi
tr gim dn theo thi
gian. Mc ch chnh ca
phng php ny l cho
php tc t p dng s
dng kinh nghim trc
thu hp khng gian
kho st.
V. KT QU.
Tiu ch chnh nh
gi thut ton hc l s
vng hc cn thit trc

Khong cch ti qu o ti u

eQ ( a ) / T
12
a ' eQ ( a ') / T

thut ton xut

10
thut ton Bucket
Brigade

8
6
4
2
0
0

20

40

60

80

S vng hc

Hnh 3. Kt qu hc t ng cho bi ton di chuyn vt nng

100

Trong khi , theo kt qu ca [10] thut ton Bucket

brigade cn khong 80 vng trc khi hi t.
VI. KT LUN
Bi bo trnh by mt thut ton hc tng cng
cho h a tc t cng tc. Thut ton l mt ci tin
ca thut ton hc Q trong thay v s dng bng Q
chung duy nht, mi tc t lu tr v cp nht bng Q
ring ca mnh. Tc t ch tin hnh cp nht gi tr
bng Q ring nu gi tr mi ln hn gi tr c. Bng
cch ng thi ghi li chin lc hnh ng tt nht
tng gp, thut ton m bo tm c hnh vi ti u
cho tc t. Kt qu th nghim cho thy tc t hc
c cch phi hp hnh ng trong bi ton di
chuyn vt nng vi thi gian hc tng i ngn.
Thut ton ch c th p dng trong trng hp hm
chuyn tip gia cc trng thi l xc nh. Mt hn
ch khc ca thut ton l mi tc t phi c thng tin
v trng thi v gi tr phn thng chung ca h
thng. Trong trng hp phn thng chung c
tng hp t phn thng ca tng tc t, h thng cn
tnh ton phn thng chung v thng bo n tng
thnh vin. Qu trnh ny c th i hi chi ph lin
lc tng i ln.
LI CM N

[4] R.H. CRITES, A.G. BARTO, Elevator group

control using multiple reinforcement learining.
Machine learning, vol. 33, pp 235-262, Kluwer
Acad., 1998.
[5] J. HU, M.P. WELLMAN, Multiagent reinforcement
learning: theoretical framework and an algorithm,
In Proc. of the 15th International Conference on
Machine Learning, San Francisco, Morgan
Kaufman, 1998.
[6] L.P. KAEBLING, M.L. LITTMAN, A.W. MOORE,
Reinforcement learning: A survey, Journal of
Artificial Intelligence Research, Vol. 4, pp 237-285,
Morgan Kaufman, 1996.
[7] M. LAUER, M. RIEDMILLER, An algorithm for
distributed reinforcement learning in cooperative
multiagent system, In Proc. of the 17th International
Conference on Machine Learning, Stanford, CA,
Morgan Kaufman, 2000.
[8] M.L. LITMAN, Markov games as a framework for
multiagent reinforcement learning, In Proc. of the
11th International Conference on Machine Learning,
New Brunswick, Morgan Kaufman, 1994.
[9] J. SCHNEIDER, W.K. WONG, A.W. MOORE, M.
RIEDMILLER, Distributed value functions, In Proc.
of the 16th International Conference on Machine
Learning, San Francisco, Morgan Kaufman, 1999.
[10] S. SEN, M. SEKARAN, Individual learning of
coordination knowledge, Journal of Experimental
and Theoretical AI, pp 156-170, 1998.

Nghin cu c thc hin vi s h tr kinh ph

ca Hi ng khoa hc t nhin.
TI LIU THAM KHO
[1] M. BOWLING, M. VELOSO, Multiagent learning
using a variable learning rate, Artificial
intelligence, vol. 136, pp 215-250, Elsevier, 2002.
[2] J.A. BOYAN, M.L. LITTMAN, Packet routing in
dynamically changing network: A reinforcement
learning approach. Advances in Neural Information
Processing Systems, 6, Morgan Kaufman, 1994.
[3] C. CLAUS, C. BOUTILIER, The dynamics of
reinforcement learning in cooperative multiagent
systems, In Proc. of the 15th National Conference on
Artificial Intelligence, Madison, WI, AAAI press,
1998.

[11] S. SINGH, D. BERTSEKAS, Reinforcement

learning for dynamic channel allocation in cellular
phone systems, Advances in Neural Information
Processing Systems, 9, MIT Press, 1997.
[12] N. SUEMATSU, A. HAYASHI, A multiagent
reinforcement learning algorithm using extended
optimal response, In Proc. of AAMAS , Bologna,
Italy, ACM Press, 2002.
[13] R. SUTTON, A. Barto. Reinforcement Learning:
An Introduction, MIT Press, Cambridge, 1998.
[14] C. WATKINS, P. DYAN, Q-learning. Machine
learning, Vol. 8, 279-292, Kluwer Acad., 1992.
[15] G. WEISS (EDS), Multiagent systems, a modern
approach to DAI, MIT Press, Cambridge, 1999.

[16] D.H. WOLPERT, K.R.WHEELER, K. TUMER,

General principles of learning-based multiagent
systems. In Proc. of the 3th Conference on
Autonomous Agents, New York, ACM, 1999.

Ngy nhn bi: 17/10/2003

S LC TC GI
T MINH PHNG
Sinh nm 1971 ti H Ni.
Tt nghip i hc ti H Bch khoa Taskent nm
1993, bo v tin s ti Vin hn lm khoa hc
Uzbekistant, Taskent, nm 1995
Hin l ging vin khoa Cng ngh thng tin 1, Hc
vin cng ngh Bu chnh Vin thng.
Hng nghin cu: tr tu nhn to, h tc t, logic
m, bioinformatics.
Email: phuongtm@fpt.com.vn