Attribution Non-Commercial (BY-NC)

17 views

Attribution Non-Commercial (BY-NC)

- Analysis of Mobile Traffic based on Fixed Line Tele-Traffic Models
- Using the Fisher Transform
- Week3 Test
- Logarithm Knowledge
- Calculating Change Base Logarithms in c++
- Redlich Kister Algebraic Representation of T
- Best SQL Pl SQL Material
- EXCEL Statistics
- S1 - Review Excerise 2
- Logarithms New June05
- Excel Tutorial
- Addmaths Project
- Logarithm
- S1 REV2 Exercises Solutions
- Log
- Testing for Normality Using SPSS Statistics
- Summary of Equations-Mj3
- Powerpoint
- 5A_Derivative_of_Logarithmic_Functions.pdf
- fm

You are on page 1of 29

Definition

Let

distribution function F(x) = Pr ( X x ). If F(x) is continuous, the r.v. is said to be continuous. Let f(x) = F(x) when the derivative is defined. If

2

Definition

The differential entropy h( X ) of a continuous r.v. X with a density function f(x) is defined as

h ( X ) = f ( x ) log f ( x ) dx

s

(1)

where S is the support of the r.v. Since h ( X ) depends only on f(x), sometimes the differential entropy is written as h( f ) rather then

h( X ) .

3

, 0x<a f (x ) = 0, otherwise a 1 1 h ( X ) = log dx = log a 0 a a Note: For a<1, log a < 0, and h ( X ) = log a < 0.

1 a

However, 2

h(X )

=2

log a

Let

X ~=

1 22

2 2 2

, then

h( ) = ln x2 2 = ( x ) ln 2 2 2 1 1 2 2 2 = E X + ln (2 ) 2 2 1 2 1 2 = + ln 2 2 2 2

[ ]

1 1 2 = + ln 2 2 2 1 1 2 = ln e + ln 2 2 2 1 = ln 2e 2 2

nats

1 2 h( ) = log 2e bits. 2

6

Theorem 1

Let

1 log f ( X 1 , X 2 ,L X n ) n E [ log f ( X )] = h ( X ) in probability

proof: The proof follows directly from the weak law of the large numbers.

7

Def: For >0 and any n, we define the typical set A(n ) w.r.t. f(x) as follows:

(n )

A

(A( ) ) > 1 for n sufficiently large 1. Pr ( ) ( ( )) 2. Vol(A ) 2 for all n ( ( )) ( ) 3. Vol(A ) (1 )2 for n sufficiently large

n n n h X + n n h X

9

properties:

Thm: The Set A is the smallest volume set with probability 1-, to the first order in the exponent. The volume of the smallest set that contains most of the Prob. Is approximately 2nh. This is an n-D volume, so the corresponding side length is (2nh)1/n=2h. The differential entropy is the logarithm of the equivalent side length of the smallest set that contains most of the Prob. low entropy implies that the rv is confined to a small effective volume and high entropy indicates that the rv is widely dispersed.

10

(n )

f(x)

Quantization of a continuous rv

Spose we divide the range of X into bins of length . Lets assume that the density is continuous within the bins.

11

By the mean value theorem, there is a value x i within each bin such that

f ( xi ) =

(i +1) i

f ( x )dx

X = xi , if i X < (i + 1)

Pi =

(i +1) i

f ( x )dx = f ( xi )

12

H (X

) =

Pi log Pi

= f ( xi ) log( f ( xi ) )

Since

f (x ) = f (x ) = 1

i

13

f ( xi ) log f ( xi ) f ( x ) log f ( x ) dx , as 0

This proves the following Thm: If the density f(x) of the rv X is Riemann integrable, then

H (X

) + log h ( f ) = h ( X ),

-n

as 0

14

h( X 1 , X 2 , L , X n ) = f ( x ) log f ( x )dx

n n

15

Let X 1 , X 2 , L , X n have a multivariate normal distribution with mean and covariance matrix K. Then 1 h( X 1 , X 2 , L , X n ) = h( n ( , K ) ) = log(2e) n K bits 2 where K denotes the determinant of K .

16

x pf : ( X 1 , X 2 ,L, X n ) ~ N n ( , K ) f ( ~ ) = Then

1 2

)K

n

1 2

exp

1 ~ ( x )T K 1 ( ~ ) x 2

)K

n

1 2

dx

17

= =

1 1 E (X j j )( X i i ) K 1 ij + ln(2 ) n K 2 ij 2 1 1 K ji K 1 ij + ln(2 ) n K 2 j i 2

]( )

( )

jj

1 = KK 1 2 j =

1 + ln(2 ) n K 2

1 1 I jj + ln(2 ) n K 2 2 j

18

D( f // g ) = I ( X ; Y ) =

f f log g

f ( x, y ) dxdy f ( x, y ) log f ( x) f ( y )

I ( X ;Y ) = h( X ) h( X | Y ) = h(Y ) h (Y | X ) = h( X ) + h(Y ) h( X , Y ) I ( X ;Y ) = D ( f ( x, y ) // f ( x ) f ( y ) )

19

Remark: The mutual information between two continuous r.vs is the limit of the mutual information between their quantized versions.

20

Properties of

h ( x ) , D(p

q) , I(X; Y)

D(f

g) 0

g g pf : - D(f g) = s f log log s f ( Jensen' s inequality ) f f = log s g log 1 = 0. I (X; Y) 0 h(X Y) h(X)

21

i =1

Theorems h( X + c) = h( x ) : translation does not change the differential entropy h(aX ) = h( x ) + log a

22

1 y pf : let Y = aX. Then , f Y ( y ) = f x ( ), and a a h(aX) = - f Y ( y ) log f Y ( y ) d y y 1 1 y = f x ( ) log ( f x ( ))dy a a a a = f x ( x ) log x ( x ) dx + log a = h ( x ) + log a Corollary : h(AX) = h(X) + log det(A)

23

Theorem : The multivariate normal distribution maximizes the entropy over all distributions with the same variance.

Let the random vector X R n have zero mean and convariance K = E XX (i.e. , K ij = EX i X j , 1 i , j n ) , Then

T

24

Pf:

x Let g (~) be any density satisfying g( ~ ) xi x j d~ = K ij for all i , j. x x Let k be the density of a (0, K) vector. Note that logk (~) is a quadratic form and xi x jk (~)d~ = K ij . x x x 0 D ( g k ) = g log( g = h( g ) + h(k )

k )

= h( g ) g log k = h( g ) k log k

25

where the substitution glogk follows from the fact that g and k yield the same moments of the quadraic form logk (x) the Gaussian distribution maximizes the entropy over all distributions with the same variance.

26

Let X be a random variable with differential entropy h(x) Let X be an estimate of X and let . E( X - X) 2 be the expected prediction error. Let h(x) be in nats

Theorem : For any r.v. X and estimator X ) 2 1 e 2h(x) E(X - X 2e with equality iff X is Gaussian and X is the mean of X

27

x

= E(X - E(X)) 2 [ the mean of X is the best estimator for X ] 1 2h(x) e (2) = var (X) 2e [Gaussian distribution has the maximum entropy for a given varance] 1 i.e. , h(x) ln 2e 2 2

28

We have equality, only in (1), only if x is the best estimator (i.e. , x is the mean of X) and equality in (2) only if X is Gaussian. Gorollary : Given side information Y and estimator X(Y) it follows that (Y))2 1 e 2h(XY) E(X - X 2e Fano' s inequality

29

- Analysis of Mobile Traffic based on Fixed Line Tele-Traffic ModelsUploaded byijcsis
- Using the Fisher TransformUploaded byKumar
- Week3 TestUploaded byjeremydb77
- Logarithm KnowledgeUploaded byAtul Singh
- Calculating Change Base Logarithms in c++Uploaded byAamir Khan
- Redlich Kister Algebraic Representation of TUploaded bygggggg82
- Best SQL Pl SQL MaterialUploaded bypubaskar
- EXCEL StatisticsUploaded byaung20102011
- S1 - Review Excerise 2Uploaded byAhmed Nurul
- Logarithms New June05Uploaded byAnand Srivastava
- Excel TutorialUploaded byMiranda Metria
- Addmaths ProjectUploaded byAn-Nisa Azmi
- LogarithmUploaded byKassandra
- S1 REV2 Exercises SolutionsUploaded bysesma1
- LogUploaded byMuhammad Asad Ali
- Testing for Normality Using SPSS StatisticsUploaded byprasadkulkarnigit
- Summary of Equations-Mj3Uploaded byShayon Deb
- PowerpointUploaded byNguyễnBìnhPhương
- 5A_Derivative_of_Logarithmic_Functions.pdfUploaded byJashandeep Singh Kochhar
- fmUploaded byjames3h3park
- 1Uploaded byabhijeet3913
- LinksUploaded byHenry Pablo
- Sample Stats Questions - Answers File.docUploaded byRakshya humagain
- boxUploaded byjosesoyo7911
- senza-nome-3Uploaded byStefano Duca
- 1 Function Enrichment PDF August 14 2009Uploaded byTan KIm HAn
- QA Sessional Paper 2Uploaded byDrRitu Malik
- bbaldi_ips_chapter01Uploaded bycrutili
- SPE-147227-MS.pdfUploaded byMicaela Flores Lanza
- Lec 5 - Hypothesis TestUploaded byBenjamin Koh

- Democracy Promotion Bad - Michigan7 2015 (1)Uploaded byDavis Hill
- kangnian he-resumeUploaded byapi-375067074
- Doc66055_38D0TDADHUploaded byChrise Raj
- A Methodology for Making a Three-coil Wireless Power Transfer System More Energy Efficient Than a Two-coil Counterpart for Extended Transfer DistanceUploaded byphithuc
- BPCL Case Analysis_Group 1.pdfUploaded byShaheen Sharma
- Utterance Sentence and PropositionUploaded byخزين الاحسان
- Radio Evolution Congress eBookUploaded byEduardo Vicente USP
- General Douglas MacArthur's Principles of Leadership @ LeadershipNowUploaded bykothanzawoo1979
- Bob2Uploaded bypaskalpe
- Positive Behavioral Supports for Young ChildrenUploaded byRommel Villaroman Esteves
- NY Immersion Sri Chakra PujaUploaded bysuresh_c_pattar673
- Elsaesser Fassbinder GermanyUploaded byEvaNardulli
- Repatriation of Expat Employees, Knowledge Transfer, Org LearningUploaded byTiffany Wiggins
- Sivavakkiyam-Yoga and philosophy of a Tamil Siddha SivavakkiyarUploaded byGeetha Anand
- Apologetics or a System of Christian Evidence (Conrad Emil Lindberg, 1917)Uploaded byWascawwy Wabbit
- Blatt, Levy. (2003).Attachment Theory, Psychoanalysis, Personality Development, And PsychopathologyUploaded bySzandor LaVey
- Connecting Bauer to Business_Unit02Uploaded byRaymondDuong
- class profile- casee brimblecombe 1 docx 1Uploaded byapi-355762287
- John Donne QuestionsUploaded byPrithvirajThakur
- 2002slaughter Strategic ForesightUploaded byFaisal Suliman
- Christian PhilosophyUploaded byCesarine Sy Bawasanta
- English Lesson PlanUploaded byOANA SAMSON
- Havard Style ReferencingUploaded bysnake.gesh7066
- [E.a.koetting] Ipsissimus(b Ok.cc)Uploaded byMarvin Buren
- HRM edited 15-12-2016Uploaded bykow kai lynn
- 10 Tips to Manage Your BossUploaded byأحمد عبدالمحسن
- eddycUploaded byNooruddin Sheik
- SOMS - Human Resource Management - Course OutlineUploaded bysandeep k krishnan
- what is experiential educationUploaded byapi-277592123
- Lolita HypnosisUploaded byCarlos Brito