10/9/2012 1

Mng vin thng -

l thuyt thng tin
By: Vinh Dang
2
10/9/2012
Outline
Gii thiu Mng vin thng
Khi nim c bn
Thng tin
Entropy
M ha ngun
3
10/9/2012
Gii thiu
Vin thng l g?
Ting Hi Lp : "tele - xa; v
v Latin, "communicate chia
s . Vin thng l truyn thng
c ly xa.
Bn cht ca Vin thng l
truyn Thng tin n 1 hay
nhiu ngi dng trong bt k
dng no c th
4
10/9/2012
Vin thng
C xu hng ngh Vin thng l in thoi,
mng my tnh, Internet, truyn hnh cp.
Bao gm phng tin in, in t, v quang
c nhc n, ng thi Bao gm dy dn
n gin, radio, hay dng khc.

5
10/9/2012
Vin thng thi k u
trng v ci
khi/Tn hiu la
nh sng
b cu
trng khi
b cu
Tower using mirrors
horn
6
10/9/2012
Vin thng pht trin

in tn (Morse)
in thoi (Bell)
truyn thng khng dy
truyn thng v tinh
A BTS (Mobile
Communication)
VINASAT-1 satellite
7
10/9/2012
h thng chuyn mch
h thng truyn thng quang
h thng thoi v truyn hnh
H thng truyn thng v tinh
h thng truyn thng di ng
H thng truyn thng khng dy
h thng truyn thng d liu
Mng vin thng
8
10/9/2012
Mng vin thng l mng gm lin kt v
nt mng vin thng c sp xp
thng tin chuyn t 1 phn ca mng n
mng khc qua nhiu kt ni v thng
qua ch khc nhau.
9
10/9/2012
Mng vin thng
Mng vin thng c chia thnh:











thit b chuyn mch v thit b truyn nhn cng nhau t mng li
thit b u cui hng khch hng kt ni n mng li thng qua
truy nhp mng
truyn ti v chuyn mch l 2 chc nng c bn truyn ti
Thng tin ngi dng.
Transport thit b truyn nhn
chuyn mch Switch hay Exchange hay Central Office
(CO)
truy nhp thit b truy nhp ca thu bao (truy
nhp mngs AN)
Customer
Premises
Equipment (CPE)
thit b u cui thu bao

10
10/9/2012
Transport
AN AN
N
T
N
T
AN
N
T
AN
N
T
Exchange
truy nhp mng (AN)
vi u cui thu bao
(CPE)
truyn ti knh truyn(trao i lin kt kt ni):
trung k truynThng tin ngi dng
tn hiu kt ni truyn thng tin iu khin
SIEMENS NIXDORF SIEMENS NIXDORF
SIEMENS NIXDORF
SIEMENS NIXDORF
11
10/9/2012
Khi nim c bn
s khi ca h thng truyn thng s
m ha
ngun
m ha
knh
truyn
b iu
ch
knh
truyn
nhiu
b gii
m knh
b gii
m
ngun
b gii iu
ch
ngun
ch
12
10/9/2012
l thuyt thng tin?
l thuyt thng tin cugn cp cch o lng ca ngun
Thng tin, dung lng thng tin ca mt knh truyn
m ha l phng tin ti u ha dung lng knh
truyn lu chuyn Thng tin
l thuyt m ha ca Shan khng c n:
Nu tc ca Thng tin t ngun ko vt qu dung
lng ca knh truyn thng tin, tn ti k thut m ha
m Thng tin c pht qua knh truyn vi xc sut
li nh, bt chp s tn ti ca li
13
10/9/2012
o lng Thng tin
l thuyt thng tin: bao
nhiu Thng tin
cha trong tn hiu?
h thng pht sinh?
mt knh truyn truyn
ti?
Thng tin l 1 dng hng
ha sn xut bi ngun
n vi ngi dng
ch
V d : Barcelona vs
GT-LA
14
10/9/2012
o lng Thng tin
3 kt qu: thng, ha, thua






S kin cng t kh nng, cng cha nhiu thng tin
Thng tin nh ngha ton hc nh th no?
Cases s kins Thng tin kh nng
1 Barca thng khng c Thng tin ~1, chc chn
2 Barca ha vi
GT-LA
Thng tin nhiu hn Tng i thp
3 Barca thua Lng Thng tin ln Xc sut thp xut
hin trong tnh
hung c bit
wins
15
10/9/2012
Thng tin
x
j
l s kin vi p(x
j
) l kh nng ca s kin x
j
c
chn truyn dn
Nu x
j
xy ra

1
( ) log log ( )
( )
j a a j
j
I x p x
p x
= =
n v ca Thng tin
I(x
j
) c gi self-information
I(x
j
)>0 for 0s p(x
j
)s1
I(x
j
)0 for p(x
j
)1
I(x
j
)>I(x
i
) for p(x
j
)<p(x
i
)
(1)
16
10/9/2012
Thng tin
c s ca logarithm
10 o lng ca Thng tin l hartley
e o lng ca Thng tin l nat
2 o lng ca Thng tin l bit
V d : th nghim ngu nhin vi 16 kt qu tng ng:
Thng tin tng ng vi mi kt qu :
I(x
j
)=-log
2
(1/16)=log
2
16=4 bits
Thng tin l ln hn1 bit,do kh nng ca mi kt qu l t hn
.
17
10/9/2012
Entropy v Thng tin tc
xem xt ngun Thng tin pht 1 chui symbols t tp
X={x
1
,x
2
..,x
M
}
mi symbol x
i
c xem nh 1 thng tin vi kh nng p(x
i
) v
self-informationI(x
i
)
Ngun c tc trung bnh r symbols/sec
Ngun ko b nh gin on
lng Thng tin cugn cp bi ngun trong khong ty
symbol l bin X gin on ngu nhin .
Thng tin trung bnh mi symbol l n :



Entropy = Thng tin = s ko chc chn
Nu tn hiu l hon ton d on c, n c 0 entropy v
khng c Thng tin
Entropy = s trung bnh bits yu cu truyn ti tn hiu
bit/symbol ) ( log ) ( )} ( { ) (
1
2
=
= =
M
j
j j j
x p x p x I E X H (2)
emitting
disrete
18
10/9/2012
V d
bin ngu nhin vi phn b chnh tc
qua 32 kt qu
H(X) = - (1/32).log(1/32) = log 32 = 5
# bits yu cu = log 32 = 5 bits!
do H(X) = s bits yu cu i din
s kin ngu nhin
Bao nhiu bits l cn thit :
kt qu ca thy ng xu
ngy mai l Th 5
19
10/9/2012
Entropy
gi tr ca H(X) t 1 ngun ph thuc vo xc sut
symbol p(x
i
) v M
Tuy nhin ,


Bin thp tng xng n khng c s ko chc chn
bin trn tng xng n s ko chc chn ln nht,
xy ra khi mi symbol l tng ng c th xy ra
chng minh ca s khng cn bng c th hin
trong [2] Chapter 15
M X H
2
log ) ( 0 s s
(3)
20
10/9/2012
chng minh
Bin thp vi ty M d dng chng minh, ghi nh
l a.log(a)0 as a0
chng minh ca bin trn l phc tp hn




Da vo bt ng thc
ln a s (a-1)
M X H
2
log ) ( 0 s s
M x p x p
M x p x p X H
M
i
i i
X
2
1
2
log ) ( log ). (
log ) ( log . ) ( ) (
s =
s =

=
21
10/9/2012
xem xt:
M X H M X H
x p
M
e M X H
x p
M
e M X H
x p M
x p e
x p M
x p e
x p M
x p x p M x p x p
M x p x p M X H
M
i
M
i
i
M
i
i
i
M
i
i
M
i
i
i
M
i
i
i
M
i
i
M
i
i i
X
2 2
1 1
2 2
1
2 2
1
2
1
2
1
2
1
2
1
2
2 2
log ) ( 0 log ) (
) (
1
log log ) (
) (
1
. log log ) (
1
) ( .
1
). ( . log
) ( .
1
ln ). ( . log
) ( .
1
log ). ( ) ( . log ) ( log ). (
log ) ( log . ) ( log ) (
s s
|
.
|

\
|
s
|
.
|

\
|
s
|
|
.
|

\
|
s =
= =
=

= =
=
= =
= = =
chng minh
M X H
2
log ) ( 0 s s
du bng xy ra khi:
M
x p
x p M
i
i
1
) ( 1
) ( .
1
= =
22
10/9/2012
V d
Vi ngun nh phn (M=2),
p(1)=o v p(0)=1-o = |.
t (2), entropy nh phn:
H(X)= -o.logo -(1-o).log(1-o)



23
10/9/2012
l thuyt m ha ngun
Thng tin t 1 ngun to ra symbols khc nhau
din t bi entropy H(X)
tc ngun Thng tin (bit/s):
R
s
= rH(X) (bit/s)
H(X): entropy ngun (bits/symbol)
r: tc symbol(symbols/s)
gi nh ngun l u vo n mt knh truyn:
C: dung lng (bits/symbol)
S: tc symbol(symbols/s)
S.C: bits/s
24
10/9/2012
l thuyt m ha ngun(t.t. )
Thuyt Shannon(l thuyt m ha ko nhiu ):
Cho mt knh truyn v 1 ngun pht sinh Thng
tin tc t hn dung lng knh truyn, c th
m ha u ra ca ngun bng cch pht thng qua
knh truyn
Biu din ca M ha ngun bng V d :
ngun
nh phn
ri rc
m ha
ngun
knh
truyn
nh phn
ngun tc symbol
= r = 3.5 symbols/s
C = 1 bit/symbol
S = 2 symbols/s
SC = 2 bits/s
25
10/9/2012
V d ca M ha ngun
ngun nh phn ri rc: A(p=0.9), B(p=0.1)
ngun tc symbol (3.5) >dung lng knh
truyn(2) ngun symbols c th ko c pht
i trc tip
Kim tra thuyt Shannon:
H(X)=-0.1 log
2
0.1-0.9log
2
0.9=0.469bits/symbol
R
s
= rH(X) = 3.5(0.469)=1.642 bits/s < S.C = 2 bits/s
truyn dn c th dng M ha ngun lm
gim tc symbol trung bnh
26
10/9/2012
V d ca M ha ngun(t.t. )
T m c gn nhm n-symbol ca
ngun symbols
QUy lut:
T m ngn nht n nhm kh nng ln nht
T m di nht n nhm thp kh nng nht
Nhm n -symbol ca ngun symbols # m
rng th n ca ngun nguyn gc
27
10/9/2012
m rng th nht
tc symbol ca b m ha = tc symbol
ca ngun
ln hn ca knh truyn c th cha
ngun
symbol
P () t m [P()].[s m ha
Symbols]
A 0.9 0 0.9
B 0.1 1 0.1
L=1.0
28
10/9/2012
m rng th hai
i
i
i
l x p L
n
* ) (
2
1

=
=
ngun symbol P () t m [P()].[s ca m ha
Symbols]
AA 0.81 0 0.81
AB 0.09 10 0.18
BA 0.09 110 0.27
BB 0.01 111 0.03
L=1.29
L: trung bnh chiu di m
p(x
i
): kh nng symbol i
th
ca ngun m
rng
l
i
: chiu di ca t m tng ng vi i
th

symbol
nhm 2 ngun symbols cng 1 thi im :
29
10/9/2012
m rng th hai
symbol urce symbols/so code 645 . 0
2
29 . 1
= =
n
L
258 . 2 ) 645 . 0 ( 5 . 3 = =
n
L
r
tc symbol ti u ra b m ha :
vn ln hn 2 symbols/s ca knh truyn
Tip tc vi m rng th 3
m ha symbols/sec
>2
30
10/9/2012
m rng th ba
nhm 3 ngun symbols cng 1 thi im :
ngun
symbol
P () t m [P()].[s ca m ha Symbols]
AAA 0.729 0 0.729
AAB 0.081 100 0.243
ABA 0.081 101 0.243
BAA 0.081 110 0.243
ABB 0.009 11100 0.045
BAB 0.009 11101 0.045
BBA 0.009 11110 0.045
BBB 0.001 11111 0.005
L=1.598
31
10/9/2012
m rng th ba
533 . 0
3
598 . 1
= =
n
L
cond symbols/se code 864 . 1 ) 533 . 0 ( 5 . 3 = =
n
L
r
tc symbol ti u ra b m ha :
tc ny l chp nhn c bi knh truyn
m ha symbols/ngun
symbol
32
10/9/2012
hiu qu ca ngun m ha
hiu qu l cch o kh nng ca ngun m ha

=
= =
n
i
i i
l x p
L
L
L
eff
1
min min
) (
D
X H
L
2
min
log
) (
=
D L
X H
eff
2
log
) (
=
L
X H
ef f
) (
=
H(X): entropy ca ngun
D: s symbols trong
coding alphabet
Trong


or
Cho alphabet nh
phn
33
10/9/2012
Entropy v hiu qu ca ngun
m rng nh phn
Entropy m rng th n ca ngun ri rc
ko nh :
H (X
n
)=n*H (X)
hiu qu ca ngun m rng:

L
X H n
ef f
) ( .
=
34
10/9/2012
Biu din ca L/n
n
n
L
1.0
0.8
0.6
0.4
0.2
0
0
1 3 4 2
0.469 H(X)
L/n lun vt qu
entropy ngun v
hi t n entropy
ngun khi n ln
gim chiu di
trung bnh t m dn
n tng phc
tp ca m ha
35
10/9/2012
M ha Shannon-Fano [1]
Trnh t : 3 bc
1. lit k ngun symbols theo th t gim xc sut
2. phn chia tp thnh 2 tp con gn nht c th.
0s c gn n tp trn v 1s n tp di
3. tip tc phn chia tp con cho n khi ko th phn
chia c na
V d :
36
10/9/2012
V d M ha Shannon-Fano
U
i
p
i
1 2 3 4 5 T m
U
1
.34
U
2
.23
U
3
.19
U
4
.1
U
5
.07
U
6
.06
U
7
.01
0
0
1
1
1
1
1
1
0
0
0
0
0
1
1
1
1
1
1
1 1
1 1
00
01
10
110
1110
11110
11111
37
10/9/2012
M ha Shannon-Fano
41 . 2
7
1
= =

=
i
i
i
l p L
37 . 2 log ) (
2
7
1
= =

=
i
i
i
p p U H
98 . 0
41 . 2
37 . 2 ) (
= = =
L
U H
eff
m ha pht sinh l tin m ha do kh nng phn chia
khng dn n tin m ha c bit. nhiu tin m ha c
hiu qu ging nhau.
38
10/9/2012
M Huffman [1][2][3]
Trnh t : 3 bc
1. lit k ngun symbols theo th t gim xc sut.
2 ngun symbols xc sut thp nht c gn a 0
v a 1.
2. 2 ngun symbols kt hp thnh ngun symbol mi
vi xc sut tng ng tng ca 2 xc sut
nguyn gc. xc sut mi t trong danh sch
tng ng vi gi tr ca n.
3. lp li cho n khi kh nng cui cng ca symbol
kt hp mi l 1.0.
V d :
39
10/9/2012
V d ca M ha Huffman
.01 U
7
.06 U
6
.07 U
5
.1 U
4
.19 U
3
.23 U
2
.34 U
1
p
i
U
i
0
1
.07
0
1
.14
0
1
.24
0
1
.42
0
1
.58
0
1
1.0
01011 U
7

01010 U
6

0100 U
5

011 U
4

11 U
3

10 U
2

00 U
1

T m U
i
40
10/9/2012
M ha Huffman : khuyt im
khi ngun c nhiu symbols (u ra/thng
tin ), m ha tr thnh cng knhm
ha Hufman + chiu di m ha c nh .
vn c d tha v d tha l ln so vi
tp nh thng tin nhm nhiu thng tin
c lp
41
10/9/2012
M ha Huffman : khuyt im
V d 9.8 v 9.9 ([2],pp. 437-438)
nhm to d tha nh nhng s T m
tng ly tha, m ha tr thnh phc tp
hn v tr hon pht sinh .
42
10/9/2012
Entropy V d 2
ua nga vi 8 con nga, vi xc sut thng
, , 1/8, 1/16, 1/64, 1/64, 1/64, 1/64
Entropy H(X) = 2 bits
Bao nhiu bits cn thit?
(a) Ch s mi con nga log8 = 3 bits
(b) gn m ha ngn hn n con nga vi
kh nng cao hn :
0, 10, 110, 1110, 111100, 111101, 111110,
111111
chiu di m t trung bnh = 2 bits!
43
10/9/2012
Entropy
Cn t nht H(X) bits i din X
H(X) l Bin thp on chiu di miu t
theo yu cu
Entropy = s ko chc chn ca bin ngu
nhin
44
10/9/2012
Entropy kt hp v iu kin
entropy kt hp :
H(X,Y) =
x

y
p(x,y) log p(x,y)
m rng n gin ca entropy n 2 RVs
entropy iu kin :
H(Y|X) =
x
p(x) H(Y|X=x)
=
x

y
p(x,y) log p(y|x)
Entropy ca Y Nu X c bit?
D kim chng:
Nu X, Y c lp , H(Y|X) = H(Y)
Nu Y = X, H(Y|X) = 0
H(Y|X) = Thng tin thm gia X & Y
Thc t : H(X,Y) = H(X) + H(Y|X)
45
10/9/2012
Thng tin tng h
I(X;Y) = H(X) H(X|Y)
= suy gim entropy do bin khc
I(X;Y) =
x

y
p(x,y) log p(x,y)/{p(x)p(y)}
Bao nhiu Thng tin v Y cha trong X?
Nu X,Y c lp , I(X;Y) = 0
Nu X,Y ging nhau , I(X;Y) = H(X) = H(Y)
i xng v khng m
46
10/9/2012
Thng tin tng h
Quan h gia
entropy, Thng tin
kt hp v tng
h
47
10/9/2012
Thng tin tng h
I(X;Y) o lng s tng t gia X v Y
s dng rng ri trong x l nh /tn hiu
nh y khoa V d :
ng k nh da trn MI
ti sao ? MI ko nhy cm vi li v
bias
48
10/9/2012
Bi tp v nh 1
tnh ton H(X) ngun ri rc ko nh c 6
symbols vi xc sut :
P
A
=1/2, P
B
=1/4, P
C
=1/8, P
D
=P
E
=1/20,P
F
=1/40
tm lng Thng tin cha trong thng tin
ABABBA v FDDFDF v so snh vi lng
Thng tin mong i trong 1 thng ip 6-symbol.
49
10/9/2012
Bi tp v nh 2
Mt ngun d liu c 16 symbols xc sut bng
nhau, mi symbol 1ms . Symbols pht sinh trong
khi 15, cch nhau khong 5-ms. Tm tc
ngun symbol .
50
10/9/2012
Bi tp v nh 3
Tm m Shannon-Fano ca ngun trong Bi tp
v nh 1, v tnh hiu qu.
51
10/9/2012
Tham kho
[1]. R. E. Ziemer & W. H. Transter, Information
Theory and Coding, Principles of
Communications: Systems, Modulation, and
Noise, 5
th
edition. John Wiley, pp. 667-720,
2002.
[2]. A. Bruce Carlson, Communications
Systems, Mc Graw-Hill, 1986, ISBN 0-07-
100560-9
[3]. S. Haykin, Fundamental Limits in
Information Theory, Communication
Systems, 4
th
edition, John Wiley & Sons Inc,
pp. 567-625, 2001