1.

o thng tin:

I ( xi ) f
p ( xi )
1
I ( xi ) log
log p ( xi )
p ( xi )
2. Lng tin ring

I ( xi ) log p ( xi ) (dvtt)

3. Lng tin ring ca ngun

I ( X ) p ( xi ).I ( xi )

p ( xi ). log p ( xi ) (dvtt/tin)
4. Entropi ca ngun
Ri rc

H ( xi ) I ( xi ) log p ( xi )

H ( X ) I ( X ) p ( xi ). log p ( xi )

Lin tc
H ( X ) w( x ) dx

9. Lng tin tng h trung bnh

Ngun ri rc

p ( xi / y j )

I ( X ; Y ) p ( xi , y j ) log

p ( xi )
p( xi , y j )

i, j

p( x , y
i

) log

p ( xi ). p ( y j )

p ( xi , y j ) log

p( y j / xi )

i, j

p( y j )

I ( X ;Y ) H ( X ) H ( X / Y )
H ( X ) H (Y ) H ( X , Y )
H (Y ) H (Y / X )
10. Tc lp tin ca ngun

R ( X ) n0 .H ( X )

Ngun ri rc
n0 - Tn s to tin ca ngun

R ( X ) F .H ( X )
i
R F . log( N )

5. Lng tin ng thi

Ri rc

Nu p(xi) = p

I ( xi , y i ) log P ( xi , y i )

I ( xi , y i ) I ( xi ) I ( y i ) I ( xi ; y i )

I ( X , Y ) H ( X , Y ) P ( xi , y i ) log P ( xi , y i )
i, j

Lin tc

w( x, y ) log w( x, y )dxdy

I ( X ,Y ) H ( X ,Y )

x, y

6. bt nh c iu kin
Ri rc

Ngun lin tc

R 2 Fmax H ( X )

Ngun c gi tr nh hu hn

X x xmin x xmax

1
d( )

w( x) w N 0
dx

R 2 Fmax log( x max x min )

Ngun c cng sut trung bnh hu hn

I ( xi / yi ) log P( xi / yi )

I ( X / Y ) H ( X / Y ) P ( xi , y i ) log P ( xi / y i )
i, j

I (Y / X ) H (Y / X ) P( xi , y i ) log P( y i / x i )
i, j

p ( xi / y i )
p ( xi )

I ( xi ; y i ) I ( xi ) I ( y i ) I ( xi , y i )

i, j

I ( xi ; y i ) H ( xi ) H ( xi / y i ) log

X {x} x

w( x) Ptb

R 2 Fmax . log

2 ePtb

Lin tc

H ( X / Y ) I ( X / Y ) w( x, y ) log w( x / y )dxdy
x, y

H (Y / X ) I (Y / X ) w( x, y ) log w( y / x ) dxdy
x, y

11. Thng lng ca knh

C n 0 .I ( X , Y )

Knh ri rc

C f .H ( X ) max

Knh lin tc
7. Quan h gia cc Entropi

H(X,Y) = H(X)+H(Y/X)
= H(X)+H(X/Y)
H(Y/X) = H(Y)
H(Y/X) = H(X)
Nu X,Y c lp thng k

8. Lng tin tng h

C 2f H (Y ) H ( N )

Thng l nhiu chun

H ( N ) log

2 eN

C 2f (log

2 ePy log

f log(1

2 eN )

Px
S
) f log(1 )
N
N

1. Cc cng thc xc sut

P(B | A) P(A, B).P(B)

P ( Ai | B ) P ( Ai , B ) / P ( B )

P ( B, Ai ).P ( Ai )
n

P( B, A ).P( A )
j

j 1

2. M ha ngun ri rc
M hnh (A, p(xi))

n0-S k hiu c nhm

X x1 ...x L

4. Gii hn Hamming v di t m chng nhiu

M pht hin sai

P ( X ) ( p ( x1 )... p ( x L ))

H ( X ) p ( x i ) log 2 p ( x i ) log 2 L

H ( X ) max
H ( X ) max

1
p ( x1 ) p ( x 2 ) ... p ( x L )
L
log 2 L

M ha vi t m c di c nh

di t m ti thiu

H(X )
H (X )

1
R
log 2 L
H ( X ) max
Hiu sut bng 1
L 2k

nh l m ha ngun 1:
X: Ngun ri rc khng nh, H(X) hu hn.
Vi 0 :

N
H(X )
J
J

R H (X )
J

Pe1

M ha vi t m c di thay i

Xy dng b m R min
L

R ni . p( x i ) min
i 1

Bt ng thc Kraft:
Nu b m c cc t m c di tng
ng l n1<n2<<nL iu kin cn v
mi b m c tnh Prefix:
L

ni

i 1

nh l m ha ngun 2:
C th xy dng c mt m hiu nh
phn c tnh Prefix v c di t m trung
bnh R tha mn bt ng thc:

H ( X ) R H ( X) 1

3. M ha Huffman
Chn n0

N 1E R

i
i
N1E C n .( m 1)

i 1

R mn mk
t

m n m k C ni .(m 1) i
i 1

N . N 1E
t

Hiu sut m ha

iu kin:

M sa sai
iu kin R

R log 2 L 1

Pe 0

2 n0 m

L n0 Z
m 1

m n m k m k . C ni (m 1) i
i 1

n k log m C ni (m 1) i
i 0

5. Gii hn Hamming v qung cch m

Pht hin sai knh c s sai t

d t 1

Sa sai hon ton knh c c s sai t

d 2t 1