M ha M thng k ti u

Khi nim m ha, cc thng s ca m ha

M thng k
Entropy
M Shannon-Fano
M Huffman
13/02/2014
Slice 1 Trng H Bch Khoa H Ni
M thng k Khi nim v Entropy
Entropy trong l thuyt thng tin l php o nh lng v
thng tin ca ngun tin.
Ngun tin c Entropy ln ni dung ngu nhin
Ngun tin c Entropy nh ni dung c c cu trc,
lp li.
Entropy c s dng trong vic m ha nn thng tin.
Nu phn b xc sut PDF ca ngun tin c bit trc,
gi tr Entropy cho bit s bit trung bnh cn thit m
ha ngun tin.
13/02/2014
Slice 2 Trng H Bch Khoa H Ni
M thng k Tnh gi tr Entropy


H(X) Entropy ca ngun tin
X Ngun tin vi cc k t x
b=2 - bit thng tin
V d:




H(X)=2.04
13/02/2014
Slice 3 Trng H Bch Khoa H Ni

X x
b
x p x p X H ) ( log ). ( ) (
symbol Tn sut p(x) -p(x).log2p(x)
a 5 0.45 0.52
b 2 0.18 0.45
r 2 0.18 0.45
c 1 0.09 0.31
d 1 0.09 0.31
11 2.04
M thng k Tnh cht ca Entropy


V d: Ngun tin abracadabra






H(X)=2.04
Ngun tin abracadabra c th m ha vi m c di trung bnh
2.04bit/k t. Bn tin m ha theo cch ny c gi l m ti u hay
m ha Entropy.
13/02/2014
Slice 4 Trng H Bch Khoa H Ni

X x
b
x p x p X H ) ( log ). ( ) (
symbol Tn sut p(x) -p(x).log2p(x)
a 5 0.45 0.52
b 2 0.18 0.45
r 2 0.18 0.45
c 1 0.09 0.31
d 1 0.09 0.31
11 2.04
M thng k Entropy ca ngun tin nh phn

Bn tin binary gm 2 k t A,B
P(A)=1-P(B)

Nhn xt:
- Gi tr Entropy cc i H=1 khi A v B c
xc sut nh nhau (0.5). Khi di
m trung bnh l 1 bit ti u.
- Trong cc trng hp cn li, H<1, cn
la chn m khc t hiu qu tt
hn (code efficiency)


13/02/2014
Slice 5 Trng H Bch Khoa H Ni
M thng k nh ngha v phn loi
Entropy cung cp thng tin v di t m cn thit cho vic m ha
ngun tin.
iu kin tin quyt ca m thng k l cn bit trc xc sut xut
hin ca cc k t (symbol) trong ngun tin.
B m ha thng k s gn cc t m (code word) c di ngn vo
cc k t c xc sut ln, v ngc li, gn t m c di ln cho cc
k t c xc sut nh => Gim kch thc ca ngun tin.
Cc thut ton ca m ha thng k
M Shannon-Fano
M Huffman

13/02/2014
Slice 6 Trng H Bch Khoa H Ni
M Shannon-Fano
Do Shannon v Fano c lp xy dng da trn l thuyt Entropy.
M Shannon-Fanon c xy dng nhm ti u ha di ca tng
t m (code word) tim cn vi gi tr -logp(x).
V d:




H(X)=2.1858

13/02/2014
Slice 7 Trng H Bch Khoa H Ni
symbol Tn sut p(x)
Lng tin ring
-log2p(x)
A 15 0.38 1.38
B 7 0.18 2.48
C 6 0.15 2.70
D 6 0.15 2.70
E 5 0.13 2.96
0 1
0 1 0
1
0 1
15+7=22
symbol Code word
A 00
B 01
C 10
D 110
E 111
6+6+5=17
M Huffman
M Huffman c xy dng da trn l thuyt Entropy
M Huffman xy dng cy nh phn v gn gi tr bit t di ln
(bottom-up) nhm ti u ha kch thc ca ton b bn tin.
V d:




H(X)=2.1858

13/02/2014
Slice 8 Trng H Bch Khoa H Ni
symbol Tn sut p(x)
Lng tin ring
-log2p(x)
A 15 0.38 1.38
B 7 0.18 2.48
C 6 0.15 2.70
D 6 0.15 2.70
E 5 0.13 2.96
0
symbol Code word
A 0
B 100
C 101
D 110
E 111
1
0 1
1 1 0 0
So snh gia m Shannon-Fano v Huffman
13/02/2014
Slice 9 Trng H Bch Khoa H Ni
M Shannon-Fano: cc t m c kch thc gn vi lng
tin ring ca k t (sai s 1)
M Huffman m bo kch thc ca bn tin m ha nh
nht




Kch thc bn tin


H(X)=2.1858

symbol
Shannon-Fano
Code word
Huffman
Code word
Tn sut
Lng tin ring
-log2p(x)
A 00 0 15 1.38
B 01 100 7 2.48
C 10 101 6 2.7
D 110 110 6 2.7
E 111 111 5 2.96
bit bit bit L
bit bit bit L
Huffman
Shannon
87 ) 5 6 6 7 ( 3 15 1
89 ) 5 6 ( 3 ) 6 7 15 ( 2


symbol bit bit R
symbol bit bit R
Huffman
Shannon
/ 23 . 2 39 / 87
/ 28 . 2 39 / 89


Vi du
13/02/2014
Slice 10 Trng H Bch Khoa H Ni