6A HE= @= E B H =

4 @HEC K D 2=L  HF=L= (C =E ? 

@A

) JA HE= @= E B H @= E B H = '"& GKA B H A AIJ=J IJE?= FHAII ) @=@A @A E@=@A @A K ?=F= GK= J F H K F=HA?A = A=J HE

= = E K ?

H= ?AEJ I ? = IE@ EII

@=

=JA =F E?= = A =

JE?= GKA AIJK@= GK= JE ?= AI LE@= A J =@= A KEJ=I FKJ=?E = JA HE= @= ? HECE = @AIJAI KJH=I 2 H HA=I @= ? F ANE F AN FKJ=@ H K E?= B E F = A

-II= JA HE= JALA IAKI FE =HAI AIJ=>A A?E@ I F H + =K@A 5D= ) JA HE= @= E B H = F = A J B E @AIA L A =H KI=@= A =FH NE ? @ J= F = D A JA F=H= ?

@A @=@ I F=H= JH= I

=@= F=H= =F E?=

A@E@= @A A JH FE= @A 5D= C H L GKA > AJ @=@= FA 2 H ANA

A JA ECK=

BAHA?A = ANF E?= > AJ IK?E J=

= @A GKA = ? = @A ?

@A @AI?HAL


FH CH= = J ?KHJ

F @AHE= IAH @AI?HEJ

 GKA A JA F H B H E   FHE J  A

EJAH= @ > AJ @

H FH CH=

GKA F=HA?A = A=J HE

F @AHE= IAH @AI?HEJ


FHAIA JA JANJ A @ I -N?A A JI ELH I ? B 1 B H -IJAI ? =JE JA HE=

F EI FHA?EI= @= @AI?HE ? >=IA I F E ?= E IACK @ I ?

FHE J

B E A => H=@ 6DA HO ANA F

?=F JK

ELH

GKA =FHAIA J= I A IE

?AEJ I BK @= LE@ I FA K

A J=EI @= I =KJ HAI =KJ H @ FHAA

?AEJ I B H= K FK> E?=@ I

@AI?HEJ I = B

@AI =FHAIA J=@ I FA AI @AIA L F A 2=H= B=?E EJ=H = ? A J=@ I A A JH FE= N I

=?HAI?A J= @ ?K

FHAIA JA JANJ @EIF

JA 9E EFA@E= B H=

I @AI?HEJ I LA A DJJF

AIJA JANJ

= F = E D= @

MMM E> KIF >H

HF=L=

- JH FE=
? @A @A E EH K ?AEJ @A E B H A J= J @A =? H@ ? = = KEJ = F F=H= IAH ?=FJKH=@ F H K = E?=

F=H= GK= GKAH @EIJHE>KE = E JKEJEL= @ A GKA

@A FH >=>E E@=@AI GKA K =

F II LA = @=

= GK= JE@=@A @A - JH FE=

E =@=
A JH FE= GKA JA A@E@= @A E ?AHJA = @A K

KEJ=I FH FHEA@=@AI A@E@= @A E B H

GKA AIJ FA = AGK= @EIJHE>KE

@ALAHE= IAH

= L=HE LA = A=J HE= @=@= FHAIA JA

H=

pi log (1/pi )

pi E

@E?= = FH >=>E E@=@A @A ALA J

@A FH >=>E E@=@AI @A K

= L=HE LA = A=J HE= @EI?HAJ=

JANJ @

KI=

C >=IA = A@= D

=IIE AIJ=

= A JH FE=

ANFHAII= A

>EJI

) A JH FE=

1 >EJ pcara = pcoroa = 0.5 log (1/pcara ) = log (1/pcoroa ) = 1 H = pcara log (1/pcara ) + pcoroa log (1/pcoroa ) = 0.5 1 + 0.5 1 = 1
= ?A @A K JA GKA A JH FE= @AFA @A F HJ= J =>E E@=@AI K = BK HAI =IIK @= @EIJHE>KE ? = ?AEJ @E= F ALA J @E= F @A A @= L=HE LA = A=J HE= I IK=I FH > @ I L= E@ I F H A = HABAHA IA =FA =I K

KJH= ANF E?=

F II LA F=H= hi  ) A JH FE= @ ? H = = > AI A JHA HE

@A A JH FE= @= @EIJHE>KE @AH=@= @=I A JH FE=I @ I

@A FH >=>E E@=@AI H  ALA J I @AII= @EIJHE>KE ANA =IIE ? F

= @A GKA AIJ= A@= D ALA J

@=@= F H

hi = log (1/pi )
?=H= @A  >EJ FH >=>E E@=@A @ I L= K NE HAI @A F EI

@ I = ?AI @= = A JH FE= @

AIJ= = A JH FE= @ I ALA J @AH=@= ? A BK

0.5

F=H= ?=@= K

@ I ALA J I J= ALA J L= D >E

) .ECKH=  =FHAIA J= =I HA = FH >=>E E@=@A @A K I ALA J I  AII= ? @E K 

1 >EJ h ph A H
AH =@

GKA = L=HE LA = A=J HE= E ? KE =FA =I GK= @ L= H

H @A A JH FE= H  KJH GKA ? HHAIF

p=0
NE

p=1

E ?AHJA = F H

= E ?AHJA = @A =

= GK= @

p = 0.5 ALA

J I AGKEFH L LAEI

@= A JH FE=

.ECKH=   4A = K @ ALA J ALA J >E F HE

AI A JHA

h ph

A h

BK

) A JH FE= @

ALA J

@=@= F H

@AH=@= FA = IK= FH >=>E E@=@A p @A FH >=>E E@=@AI H 

h

p @A log (1/p) = A JH FE= @=@= F H p log (1/p) =

@=I FH >=>E E@=@AI = @=I A JH FE=I

A JH FE= @= @EIJHE>KE @ I ALA J I F 7 JA I

@=@= FA = I

@AH=@= FA =I IK=I FH >=>E E@=@AI ? ?AEJ 7 @A A JH FE= E =@ IEI

= AIJH=J CE= >=IJ= JA E JKEJEL= F=H= A JA @AH @= ? @E ?= AJ=@ @AH ? =I JE = @A F=H= HA?A>AH ?

=JH=L I @= =F E?= * + A , A HAIF ALA J FA = B

A I=CA I

@AJAH

= ? @E ?=@ H B E FH AGKEFH L LAEI

A JH=@=I I HEAI @A ALA J I ) ) * + A , = F=H= ?=@= H A @AI?HEJ I @=@= FA =

0 K 1 5A I ALA J I = 0.25 ?=@= = ? @E ?= JE @ALA ? JAH 2 >EJI 11 10 01 A 00 HAIFA?JEL= A JA ? B H K = @A A JH FE= @ ALA J h ) A JH FE= @=I A I=CA
I= @=I I HEAI @A A I=CA I p

B H @A ) 1 @ J= JE

K = @A A JH FE= @= @EIJHE>KE

@A FH >=>E E@=@AI H  J= = D JE @E F H ALA J @AI?HEJ =I FH >=>E E@=@AI = F=H= B H A @AI?HEJ L= = H @A

>

JA 5A

L=

2
*

>EJI GKA HAFHAIA J= + A , I A A JA = ? @E ?=

J=

I ALA J I

=FHAIA J=@ I ? HAIFA?JEL=

HAIFA?JEL=

I ALA J I @ALA ?

0.5 0.25 0.125 A 0.125 JAH 1 2 3 A 3 >EJI

K = @A A JH FE= GKA HAFHAIA J= @A ? @E ?= = K = @A A JH FE= @= JANJ

01 001
ALA J = D = @A

000

A JA ? JA K

FA = B H >EJI

h @E =

) A JH FE= @=I F H ALA J

A I=CA I @=@= FA = B H

@EIJHE>KE

@A FH >=>E E@=@AI H  @AI?HEJ A I=CA I = A JH FE=

1.75

JA GKA

AIIA ?

A@E@= @= GK= JE@=@A @A E B H

HAGKAHE@=

@E= F=H= @AI?HALAH = L=HE LA = A=J HE=

J=I I >HA A JH FE=

) GKE JI ) ? B=?E GK= @ E@=@A @EII @ C >=IA IANJI IAFJI GKA ANFHAII= A JH FE= A @EJI K =JI HAIFA?JEL= >EJI A JA KJH=I >=IAI F @AHE= IAH KI=@=I ? BAEJ= F H ! " # $ %  K A A = A JH FE= IAHE= ANFHAII= A JHEJI GK=HJI ) JH ?= @A >=IA F @A IAH = AII= ? I ? AH LA AH FH >=>E ) FH L=

Hb (X ) = (logb a) Ha (X ) LA 0 log 0 = 0 KI=@=

KIJE ?=@= F H ? AH JA @A = LAH .ECKH= 

= JA HE= @= E B H = LA = =@E =E HAI GKA )IIE IA FHA

A JA

JE KE@=@A K

x log (x)

JA @A =

x
AH

@A JAH

K@= = A JH FE= HAI @A A JH FE= H A

I L=

h I

K ECK=EI =

GKA

0 pi 1
K J=
=

GKA E

F E?= A

log (1/pi ) 0
@E?E
@A F

- JH FE= ?

A JH FE= ?
J= >

A E B H
K

JK=

) JA HE= @= E B H = A=J HE=I I ?E=@= = A=J HE= @A E B H JK= = ?=F=

?=F= F H ANA K J= @E?E

E@=H ?

F=H @A L=HE LAEI = =I K J= GKA >

@A GK= JE ?=H = @A K

= GK= JE@=@A E B H

I L=HE LAEI = A=J HE=I ? ? DA?E@= = A JH FE= ? GKA K )I HA =

A JA = A JH FE= ? =

H (X ; Y )

= GK= JE@=@A @A E B H

= L=HE LA = A=J HE= @=@

KJH= L=HE LA = GK= JE@=@A = = @E=CH=

H (X |Y ) A J=
J =?AH?= @= A@E@=I I

= L=HE LA = A=J HE= ? AI A JHA AII=I

KJH= E B H

I (X ; Y )

ANFHAII=I

@= .ECKH=

.ECKH=

 4A =

AI A JHA A JH FE= A E B H

JK=

"

) A JH FE= ? @A E = B H = HA= E?= ? = A=J HE= =@ I

K J= A JA K

@=@= F H L= F EI

H (X ; Y ) = x X ; Y F @A IAH ?

1 -II= y px;y log px;y IE@AH=@= K = L=HE LA

= @EIJHE>KE ,AII= B H

FH >=>E IJE?= @A ALA J I = = AGK= @= A JH FE= ?

xi

yi

?=JA

xi &yi

@= A JH FE= F=H= K K J=

= L=HE LA

= A=J HE= F @A IAH KI=@= F=H= GK= JE ?=

H (X ; Y ) =

H (X &Y ) =
AIJH=J CE= IE

px&y 1/px&y
@E?E = ?K F AI @A ?

) A JH FE= ?

@=@= F H

1 7 = H (X |Y ) = x y px;y log py |x H (X |Y ) = H (X ; Y ) H (Y ) A GKA I L= HAI

@= A JH FE= F=H= K A J= J @A >JAH = 7 GKA = I =

H (X ; Y )
D

H (Y )

F @A

IAH ?= ?K =@ I FA = AGK=

L=HE LA = A=J HE= = FH FHEA@=@A = = ) E B H E B H

H (X |Y ) CAH= A JA @EBAHA @A H (Y |X ) H (X ) H (X |Y ) = H (Y ) H (Y |X ) K = B H =
JA GKA AI JK= @=@= F H

JK= A JHA =I @EIJHE>KE

;y I (X ; Y ) = x y px;y log pxx p y AIJH=J CE= IE F AI @A ? ?K I (X ; Y ) = H (X ) + H (Y ) H (X ; Y ) A L= HAI H (X ) H (Y ) A H (X ; Y ) F @A IAH ?= ?K =@ I FA = AGK= @= A

JH FE= K J=

F=H= K 7

= L=HE LA = A=J HE= =E F A @E?E A J= ? FKJ=?E = = F=H= GK= JE ?= JK= ?=F= @A E@=H ? ,ME A =J => F H 9E =J =>?A JH= @= A JH FE= ? = A E B H =EI @A @K=I L=H =?AIIA =I H JE

A JH FE= ? =I A DJJF

E LAEI = A=J HE=I B E @AIA L MMM

LE@=I F=H=

=JDM H I ?

AAN?D= CA =KJD HI &#$##

,EIJ
) @EIJ

?E=
?E= @A K >=?  AE> AH J= > ?D= =@= @A A JH FE= HA =JEL= ) @EIJ K =

A@E@= @= @EIJ K >=?  AE> AH K = @=@AI

?E= A JHA @K=I @EIJHE>KE

AI @A FH >=>E E@=@A EH GKA = @EIJHE>KE

?E= @A

D(p||q ) =

pi log (pi /qi ) = p

pi log (1/qi ) pi log (1/pi )

@A FH >=>E E AII= AGK= JE > = @A

A@E@= @= E A ?E

?E= @=@= F H =IIK

IA @

GKA = LAH@=@AEH= @EIJHE>KE ALA J

pi

qi

E @E?= AI

=I FH >=>E E@=@AI @ @A FH >=>E E@=@A = @ , AI ? 5A =I ?AEJ

i @A K

= L=HE LA = A=J HE= @EI?HAJ= = ? @E ?= A JA J= B=?E EJ=

=I @EIJHE>KE A JA @E *

pAq

) =F E?=

A I=CA I IACKE @ A J + A A @A A F H

= AIJH=J CE= =FHAIA J=@= FHALE= @A @EIJ ?E= IEIJA

= ? @E ?=@ H B IIA F = A =@ IA B IIA JAHE=

F=H= JH=J=H

I ALA J I ) A JA ?

A I=CA I ? FA = B H > JA

AGKEFH L LAEI qi

= 0.25

?=@= = ? @E ?= B H

F=H= ?=@= ALA J I?HEJ

>EJI 11

10 01
ALA J

00

HAIFA?JEL=

K = @A A JH FE= @ L= H @A

h(q )

) A JH FE= AIFAH=@= @=I J= = D @E ?= @A AIFAH=@ JE

I=CA I H (q ) @=@= FA = B H

K = @A A JH FE= @= @EIJHE>KE = @EIJHE>KE

@A FH >=>E E@=@AI

J= @A

>EJI GKA HAFHAIA J=

ALA J

@AI?HEJ

A J= J

LAH@=@AEH= @ I ALA J I ) * + A , = F=H= A JA HAIFA?JEL=

I ALA J I A L @A >EJI =@E?E E @ A J

pi = 0.5 0.25 0.125 A 0.125 HAIFA?JEL= A JA A = ? @E LAHE= h(pi ) = 1 2 3 A 3 >EJI 1 01 001 A 000
GKA = A JH FE= @=I = =F E?= F=H= = @EIJHE>KE A I=CA I LAH@=@AEH= F=H= @EIJHE>KE A J= J @= ? @E ?=

H (p) = 1.75 q ? h(qi ) = 2

?E= E A C=IJ

>EJI F=H= ?=@= ALA J = @A >EJI F=H=

HAIK J= A

E A ?E

I ALA J I ) * + A , @A

log (pi /qi ) = 1 0

1A

1 >EJ

HAIFA?JEL= )I E A ?E A

A JA  ?E=I F

JA GKA E A ?E

?E= @A A

>EJ HAFHAIA J= A? A JA

E= @A

>EJ ?E=

@AH=@=I FA =I FH >=>E E@=@AI LAH@=@AEH=I @ I ALA J I HAIK J=

pi log (pi /qi ) = 0.5 0

@E= F H ALA J @AI?HEJ

0.125

0.125

>EJ HAIFA?JEL= =

) E A ?E

FA = ? @E ?=

F=H= @EIJHE>KE

F=H= @EIJHE>KE = CK I ANA F

LAH@=@AEH= @EBAHA ?=I I

@A

= A JHA

D(p||q ) = 0.25 >EJ AIIA ?=I H (p) A H (q ) F H AII= HA =

AI I A ? * JH=@=I ?

D(p||q )

>IAHL=@= B E ECK=

A ?

JH=@= =FA =I A AI ?

KJH=I HA = A GKA

KJH=I @EIJHE>KE @EIJHE>KE

I ALA J I ) A

+ A , J

AIFAH=@= @=@= F H

qi = 0.5

0.125 h(qi ) = 1 2 3 A 3 >EJI A H (q ) = 1.75 >EJI A @EIJHE>KE pi = 0.125 0.125 0.25 A 0.5 h(pi ) = 3 3 2 A 1 >EJI A H (p) = 1.75 >EJI AIIA ?=I = E A ?E ?E= F=H= I ALA J I IAHE= @A log (pi /qi ) = 2 1 1 A 2 >EJI A =I E A ?E ?E=I F @AH=@=I FA =I FH >=>E E@=@AI LAH@=@AEH=I IAHE= pi log (pi /qi ) = 0.25 0.125 0.25 A 1 >EJ HAIFA?JEL= A JA )IIE = E A ?E ?E= @E= F H ALA J @AI?HEJ FA = ? @E ?= F=H= @EIJHE>KE q F=H= @EIJHE>KE LAH@=@AEH= p IAHE= @A D (p||q ) = 0.875 >EJ
LAH@=@AEH= @=@= F H )IIE DA?E@= IA = LAH@=@AEH= @EIJHE>KE IAHE= F II LA 5A A J= J ? IJHKEH K K = @A >EJI = B IIA KI=@ @A K ? @EC ? ? @EC = L=HE LA @AI?HE F=H= K = A=J HE= B IIA ? @E= @A J= = D IAHE= IEIJA LAH @=@=

0.25 0.125

H (p)

= @EIJHE>KE

q
GKA ?

A?AII HE I @ @=@AEH= F AH

H (p) + D(p||q )
@E @ AI

@E= F=H= @AI?HALAH = L=HE LA = A=J HE= E =@= A JH FE= ?HK =@= ALA J >=IA=@ @A K = @EIJHE>KE

-II= GK= JE@=@A @A E B H

@A >EJI F=H= E@A JE ?=H K K AIGKA = @A ? @E ?= K J= I ?

p KI= H H (p; q ) =
=

= @EIJHE>KE J= I

pi log (1/qi ) = H (p)+ D(p||q )

= @= A JH FE= ?

JA GKA =

@A A JH FE= @EIJE J I A = I JA GKA @A H @ GKA JE = I

?HK =@=

?AEJ I A JHAJ= J pi

) .ECKH= ! =FHAIA J= @EBAHA JAI @EIJHE>KE J I = E A ?E @AIJ=I L= GKA E A GKA ? ?E= F IEIJA = @EIJ ?E= @A

AI FH >=>E IJE?=I @A JHE J= ALA

@AH=@= =II ?E=@= = ?=@= ALA J @AH=@= =E HAI @ E A ?E GKA ?E= F E?= @ AH

K >=?  AE> AH D (p||q ) ? HHAIF A

log (pi /qi )

HAI @A E A ?E ? HHA AIFAH=@= ? HHA

?E= F

= ALA J I = @EI AH GKA = I GKA

= @EIJHE>KE ? @E ?=

LAH@=@AEH= ? =EI A? F K? F ?E=

=E H FH >=>E E@=@A @ @AH=@= A GKA = ? @E ?= = @EIJHE>KE >=EN JA E

JHE>KE

ALA J I ? = @EIJHE>KE

ALA J I ?

ALA J I GKA = @EIJHE>KE IK= E A ?E @K E@= I L= AI = @A

LAH@=@AEH= ? BHAGKA JAI @AH=@= FA AC=JEL I

H FH >=>E E@=@A @ F HJ

AIFAH=@= F H IAHA ?E=  AC=JEL= GK= @ E @ ?E= AH L=

LAH@=@AEH= ?E= HA

pi

H @A @EIJ

=  AII= FH FHEA@=@A C=H= JA GKA .E = A JA I ALA J I ? =II ?E=@ I

HAI @A @EIJ ?E= ECK= =

K ?= IA = =I @EIJHE>KE

= FH >=>E E@=@A

AI AIFAH=@= A LAH@=@AEH= AIJ

= E A ?E

.ECKH= !  ,EIJ

?E=I @A

K >=?

AE> AH A JHA @EBAHA JAI @EIJHE>KE )I ?KHL=I = KIJ=@=I B H= AI

AI @A

FH >=>E E@=@A @A JHE J= ALA J I @EI?HAJ I =FA =I F=H= B=?E EJ=H = LEIK= E =

E IAHE@=I

@=I @EIJHE>KE

J=I I >HA @EIJ ?E=

)I ? E B H LA = LA AI

0 log (0/q ) = 0
LA AI I AI

p log (p/0) =

KI=@=I ?E=I

= JA HE= @=

AII=I ?

KIJE ?=@=I F H ? A

JE KE@=@A

I ?=I I A GKA A L FHAIA JAI JHE>KE FH ?A@E J=@=I F=H=

GKA IA @AIA = @EBAHA ?E=H A JHA =I @EIJ @EIJHE>KE E @ K

D(p||q 1) A D(p||q 2)
= CK I @ I ALA J I A JA =I @EI -IIA AI =FHAIA I ALA J I

q1
K

q2

GKA

=FHAIA J= E

= @EIJHE>KE A J ?K B E =F E?=@ I L=HE=L=

= F IIE>E E@=@A

@E ?=H IKJE = F=H=

AI @A

= FH >=>E E@=@A EHHEI HE= = I L= HAI A JHA I L=

HAI @A FH >=>E E@=@A @=I @EIJHE>KE

HECE =EI L=HE=L= A JHA 0 A 0.19 A I KI=@ I 0.01 A 0.14 ? = =F E?= @AIIA FH ?A@E A J D (p||q 1) = D (p||q 2) = 0.70 >EJI A D (p||q 3) = 1.83 >EJI IA = =F E?= D(p||q 1) = D(p||q 2) = D(p||q 3) =  I L= HAI @A D (p||q ) I IA FHA =E HAI GKA AH IA =I @EIJHE>KE AI p A q I @EBAHA JAI K ECK=EI = AH IA =I @EIJHE>KE AI p A q I ECK=EI = .ECKH= ! ?K I ? )FAI=H @ @EIJ A = @EIJ ?E= @A AI K @= CA GKA K >=? AE> AH IE FH FHE= JHE?= A I A JA K = ?E= A JHA =I @EIJHE>KE = .ECKH= ! A = LA I L= GKA A = I=JEIB= AI I

= @AIECK= @=@A JHE= CK =H HAFHAIA J=@=

AJHE= AK? E@E= =

-II= ?=H=?JAH IJE?= AIJ

HAI @=I FH >=>E E@=@AI I

F H I

HABAHA JAI = ALA J I @EBAHA JAI >EJI 1IJ = = @EIJ ? L= ?E= A

D(p||q 3) = 1.83
= @= @EIJ ) @EII

q2

CAH=

D(p||q 2) = D(q 2||q 3) = 0.70 >EJI A p A q3 F @A IAH >JE@= FA = = @EIJ ?E= q 2 = q 3 0.7 + 0.7 = 1.83 H @A D (p||q ) @EBAHA JA @ L= H @A D (q ||p)
?E= A JHA

4ABAH


?E=I
6D =I ) '' - A A JI B 1 B H =JE 6DA HO 9E AO

+ LAH 6 1 JAHI?EA ?A DJJF A

AM ; H HC ME E .E A HC ME E + /=KII -N= F ANE@=@A @A F A F C C H L > AJEL

ME EFA@E=

DJJF @= ?

FJ ME EFA@E= F ANE@=@A

@A

C H L