- Cox - Statistics Paper
- Notes 19
- Draper Talk for Sf Asa
- Mthesisv2 Final Corrected
- extremum_estimators_mle.pdf
- CR Lower Bound Fisher Info
- W4213 - Estimation Theory
- Statistical Inference Exercises
- ajfisher
- MLE
- Fisher Information Tutorial
- Simulation Study of Parameter Estimation and Measurement Planning on Photovoltaics Degradation
- Application of Stochastic Lognormal Diffusion Model With
- The Epic Story of Maximum Likelihood
- Dr Ost Klaas Sen
- Comp Neuro Syl Lab Us 2018
- part1
- Guo´s Manuals SOA exam C.pdf
- Chapter 5
- JEE-2002-Vol-5-Issue-1
- Chapter 4
- 362assn7-solns
- 19180282 Theory and Practice of Risk Assessment Icra 5 (Springer, Pms 136, 2015)
- Exercise 13.22
- Sources of Error in the Estimation of Mosquito Infection Rates
- Sample Size and Power Estimates for a Confirmatory Factor Analytic Model in Exercise and Sport: A Monte Carlo Approach
- A review of software fault detection and correction process, models and techniques
- Using Regression Models to Analyze Randomized Trials
- Curs 1 SSL - Introduction
- Test Set 2 SI
- Dispatches from Pluto: Lost and Found in the Mississippi Delta
- Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future
- The Innovators: How a Group of Hackers, Geniuses, and Geeks Created the Digital Revolution
- The Unwinding: An Inner History of the New America
- Yes Please
- Sapiens: A Brief History of Humankind
- The Emperor of All Maladies: A Biography of Cancer
- A Heartbreaking Work Of Staggering Genius: A Memoir Based on a True Story
- Grand Pursuit: The Story of Economic Genius
- This Changes Everything: Capitalism vs. The Climate
- The Prize: The Epic Quest for Oil, Money & Power
- Devil in the Grove: Thurgood Marshall, the Groveland Boys, and the Dawn of a New America
- John Adams
- The World Is Flat 3.0: A Brief History of the Twenty-first Century
- Smart People Should Build Things: How to Restore Our Culture of Achievement, Build a Path for Entrepreneurs, and Create New Jobs in America
- The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers
- Rise of ISIS: A Threat We Can't Ignore
- The New Confessions of an Economic Hit Man
- Team of Rivals: The Political Genius of Abraham Lincoln
- Angela's Ashes: A Memoir
- Steve Jobs
- How To Win Friends and Influence People
- Bad Feminist: Essays
- You Too Can Have a Body Like Mine: A Novel
- The Incarnations: A Novel
- The Light Between Oceans: A Novel
- Extremely Loud and Incredibly Close: A Novel
- The Sympathizer: A Novel (Pulitzer Prize for Fiction)
- The Silver Linings Playbook: A Novel
- Leaving Berlin: A Novel
- Bel Canto
- The Master
- A Man Called Ove: A Novel
- Brooklyn: A Novel
- The Flamethrowers: A Novel
- The Blazing World: A Novel
- The Rosie Project: A Novel
- The First Bad Man: A Novel
- We Are Not Ourselves: A Novel
- The Love Affairs of Nathaniel P.: A Novel
- Life of Pi
- The Perks of Being a Wallflower
- Lovers at the Chameleon Club, Paris 1932: A Novel
- The Bonfire of the Vanities: A Novel
- The Cider House Rules
- A Prayer for Owen Meany: A Novel
- The Wallcreeper
- The Art of Racing in the Rain: A Novel
- Wolf Hall: A Novel
- Beautiful Ruins: A Novel
- The Kitchen House: A Novel
- Interpreter of Maladies
- Good in Bed

Estimator

Institute of Communications Engineering, EE, NCTU

1

**Unit 3: Maximum Likelihood Estimator
**

**Why do we use the maximum likelihood (MLE) estimator
**

Sau-Hsuan Wu

**Often there is no obvious way to find the MVU estimator
**

The MLE is approximately efficient, thus approximately the

MVU estimator

**We demonstrate this through an example
**

x[n] = A + w[n], n=0,1…,N-1, with w[n] ~ N(0,A), A > 0

The PDF is

Institute of Communications Engineering, EE, NCTU

2

**Unit 3: Maximum Likelihood Estimator
**

Sau-Hsuan Wu

**It is not obvious whether an efficient estimator exists
**

The CRLB is

Can we find the MVU estimator?

By the Neyman-Fisher factorization theorem, we attempt to

to factorize the PDF into

Institute of Communications Engineering, EE, NCTU

3

Unit 3: Maximum Likelihood Estimator

Sau-Hsuan Wu

**Assume T(x) is a complete sufficient statistic
**

We may try to find the MVU estimator if

Since

**not obvious how to choose g
**

Or we may try

Suppose Â= x[0]

**if Â is an unbiased estimator
**

?

Institute of Communications Engineering, EE, NCTU

4

NCTU 5 . EE.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu We now have exhausted our possible optimal approaches We try to find an estimator that maximizes the likelihood Differentiating the LLK and setting it to zero produces Solving for Â gives >0 Institute of Communications Engineering.

can we say anything about the MLE estimator? Is it biased? E(Â) = E But. EE. as N Institute of Communications Engineering. NCTU 6 .Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Now.

we have Institute of Communications Engineering. Â A as N What is the variance of Â as N ? We use the statistical linearization argument used in Unit 1 Since will be concentrated about E(x2)=A+A2 Let u = and with Â = g(u) Linearizing about u0 = E(u) = A+A2 . NCTU 7 . EE.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Therefore.

1 E N It follows that E(Â) = A The asymptotic variance becomes Since for a ~ N(. NCTU 8 . EE. 2) Sau-Hsuan Wu 2 2 x [ n ] A A n 0 N 1 Institute of Communications Engineering.Unit 3: Maximum Likelihood Estimator As a result.

Besides. NCTU 9 . Applying this formula gives Sau-Hsuan Wu CRLB Â is asymptotically efficient.Unit 3: Maximum Likelihood Estimator Thus. by the central limit theorem. EE. Then. when N is is Gaussian too Institute of Communications Engineering. Gaussian.

The first-order and second-order derivatives of the loglikelihood function are well defined (existence) The Fisher information is zero : then the MLE of the unknown parameter is asymptotically distributed according to where I() is the Fisher information evaluated at the true value of the unknown parameter Institute of Communications Engineering. ) of the data x satisfies some regularity conditions. EE.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Theorem If the PDF p(x. NCTU 10 .

…. n=0.N-1 Suppose A and f0 are given Institute of Communications Engineering.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Ex.1. The MLE of the sinusoidal phase x[n]=A cos(2 f0 n + ) + w[n]. NCTU 11 . EE.

Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu The MLE is found by minimizing Differentiating with respect to produces Setting it to zero yields If f0 is not equal to 0 or ½. NCTU 12 . otherwise cos(2f0n+) = cos() Institute of Communications Engineering. EE.

EE. NCTU Sau-Hsuan Wu 13 .Unit 3: Maximum Likelihood Estimator Then. we have Upon expanding this we obtain Finally. Institute of Communications Engineering.

NCTU 14 . EE. we have In this case. so the asymptotic variance is Sau-Hsuan Wu where = (A2/2)/2 is the SNR Institute of Communications Engineering.Unit 3: Maximum Likelihood Estimator According to the Theorem.

EE. the variance could be smaller than the CRLB Institute of Communications Engineering. NCTU 15 .08.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Suppose A=1.05 The effect of data length is simulated below When estimator is biased. = /4 and 2 = 0. f0 =0.

NCTU 16 . EE.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu When N=80 Only at high SNR is the CRLB attained Institute of Communications Engineering.

Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu The required data record length for the asymptotic results to apply also depends on the SNR SNR = 10dB SNR = -15dB Institute of Communications Engineering. NCTU 17 . EE.

the CRLB may be attained even for short data records if the SNR is high enough Rewrite If the data record is large and/or the sinusoidal power is large. the noise terms will be small. NCTU 18 .Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu For signal in noise problem. allowing the MLE to attain it’ s asymptotic distribution Institute of Communications Engineering. EE.

we may well estimate a function of For example. a signal is attenuated by a shadowing effect [n] s.2) Or.t. with w[n] = ln( [n]) ~ N(0.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu MLE for transformed parameters In many instances. NCTU 19 . EE. The MLE of : Institute of Communications Engineering. the received signal is modeled as [n] = [n] More convenient to estimate ln() = A from x = ln( [n]) because x[n] = A + w[n].

A) A 0 2 = Aˆ x 2 Institute of Communications Engineering. pT ( x. thus The maximum likelihood estimator is ˆarg max{ pT ( x.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Now. ). p( x. ). ). we consider another transformation of = A2 The transformation is not one-to-one : A . EE.)} arg max p( x. ) 1 2 0 2 2 = arg max p( x. NCTU 20 . p( x. ) = arg max p( x.

then ˆarg max pT ( x. NCTU 21 . EE. ) is parameterized by .) : g () Institute of Communications Engineering. is given by ˆ ˆg ( ) ˆis the MLE of obtained by maximizing p(x. ) where If g is not a one-to-one function.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Invariance property of the MLE The MLE of the parameter = g(). ) arg max max p( x. where the PDF p(x.

Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu MLE for Gaussian linear model Recall that the PDF of Gaussian vector parameters p ( x. EE. θ) 1 ln det T 1 ( x μθ ( )) C (θ)( x μθ ( )) k 2 k 2 k It can be shown that (see Appendix 3C for details) ln det C (θ) 1 C (θ) tr C (θ) k k Institute of Communications Engineering. NCTU 22 . θ) 1 N 2 1 2 1 exp ( x μθ ( ))T C 1 (θ)( x μθ ( )) 2 2 det C (θ) Thus C (θ) 1 ln p( x.

θ) ( Hθ)T 1 C ( x Hθ) 0 θ θ θ H C H H T C 1 x T 1 1 Institute of Communications Engineering. we have ln p( x.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu As a result. NCTU 23 . consider the case of x = H+ w whose PDF is p ( x. θ) 1 1 C (θ) μθ ( )T 1 tr C (θ) C (θ)( x μθ ( )) k 2 k k 1 1 C (θ) ( x μθ ( ))T ( x μθ ( )) 2 k Now. EE. θ) Thus 1 exp ( x Hθ)T C 1 ( x Hθ) 2 1 N 2 1 2 2 det C ln p( x.

1.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu The MLE of the sinusoidal parameter estimation x[n]=A cos(2 f0 n + ) + w[n]. f0 and The PDF is given by MLE is found by minimizing Institute of Communications Engineering. EE. n=0. NCTU 24 .N-1 Suppose A > 0 and 0 < f0 < ½ We want to find the MLE of A.….

1 c.2 s)T (x.2 s) = (x-H)T (x-H) where = [1. NCTU 25 . 2 2 A 1 2 Also let 2 arctan 1 c = [1 cos2f0 … cos2f0 (N-1)]T s = [0 sin2f0 … sin2f0 (N-1)]T Then J’ (1. EE. 2]T and H = [c s] Institute of Communications Engineering.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Notice that J is nonquadratic in A and . 2.1 c. thus we need to do some parameter transformations Let 1 = A cos and 2 = -A sin As a result. f0) = (x.

. we can find f0 through xTH (HTH)-1HTx Institute of Communications Engineering. EE.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu The cost function is exactly what encountered in Gaussian linear model with C=I. ˆ α H H HT x T 1 The cost function becomes T ˆ ˆ ˆ ˆ J '( . f ) x Hα x Hα 1 2 0 =x I H H H H x = x H H H H x T 1 T T T 1 T x H H H HT x T 1 T since I-H (HTH)-1HT is an idempotent matrix (A2=A) Since H = [c s]. NCTU 26 .

we have Similarly. NCTU 27 . EE. we also have cTc N/2 and sTs N/2 Institute of Communications Engineering.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Rewrite xTH (HTH)-1HTx into When f0 is not close to 0 or ½.

NCTU 2 0 28 .Unit 3: Maximum Likelihood Estimator Then. 2 fˆ = arg max 0 N f0 Sau-Hsuan Wu N 1 x[n]exp(j 2f n) n 0 Institute of Communications Engineering. xTH (HTH)-1HTx becomes Therefore. EE.

hence H 1 N 0 T 1 ˆ N c 2 T T ˆ α H H H x T x sˆ 2 0 N 2 N N 1 ˆ 2 x[n ]cos 2f 0n n 0 N 1 N ˆn x [ n ]sin 2 f 0 2 n 0 Institute of Communications Engineering. NCTU T cˆ x T s x 29 .Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu ˆ [cˆsˆ Given fˆ ] 0 . EE.

NCTU 30 . EE.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Applying the invariance property of the MLE. we obtain 2 2 2 ˆ ˆ ˆ A 1 2 N N 1 x[n]exp( j 2fˆn) n 0 0 N 1 ˆ x[n ]sin 2f 0n ˆ 2 0 ˆ arctan arctan Nn 1 ˆ x[n ]cos 2fˆn 1 0 n 0 Institute of Communications Engineering.

NCTU 31 . a [ p ] 2 u | A( f ) | The asymptotic log likelihood function can be shown to be 1 N N N 2 2 ln P( x.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Parameter estimation of an AR process x[n] + a[1]x[n-1] +…+ a[p]x[n-p] = u[n] p A( z ) 1 a[m]z m 1 m p A( f ) 1 a[m]exp( j 2fm ) m 1 2 Pxx ( f . EE. a. θ) θ a [1] a [2]...u2 ) ln 2 ln u2 2 | A ( f ) | I ( f )df 1 2 2 2u 2 2 Where N 1 1 I( f ) = x[n ]exp( j 2f 0n ) N n 0 is the periodogram of the data and Pxx(f ) is the PSD 2 u Institute of Communications Engineering.

t.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Differentiating w. which is a function of a through A( f ). we must minimize u . EE. u2 and setting the result to zero yields and thus Substituting this back into the asymptotic LLK results in 2 ˆ ˆ a To find .r. NCTU 32 . Institute of Communications Engineering.

NCTU 33 . for k=1. EE.f ) = A*( f ) and I( .…. we have Besides.f ) = I( f ) Institute of Communications Engineering. A( .p.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Define Since J(a) is quadratic in a.

This can be shown to be Institute of Communications Engineering.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Setting it equal to zero produces or But is just the inverse Fourier transform of the periodogram evaluated at k. EE. NCTU 34 .

Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu The set of equations to be solved for the approximate MLE of the AR filter parameters a becomes Or in a matrix form This can be solved with the Levison recursion Institute of Communications Engineering. NCTU 35 . EE.

NCTU 36 . k=1.p The zero derivative condition All terms in the sum are zero except for k=0 Institute of Communications Engineering. EE.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu We complete the discussion with an explicit form of the 2 MLE of u Since Note that by the optimality of aˆ [k ].….

the MLE is given by Institute of Communications Engineering.Unit 3: Maximum Likelihood Estimator By definition. aˆ [0] 1 . EE. It follows Finally. NCTU Sau-Hsuan Wu 37 .

Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Asymptotic PDF of MLE for a scalar parameter If the PDF p(x. The first-order and second-order derivatives of the loglikelihood function are well defined (existence) The regularity condition of then the MLE of the unknown parameter is asymptotically distributed according to where I() is the Fisher information evaluated at the true value of the unknown parameter Institute of Communications Engineering. NCTU 38 . ) of the data x satisfies some regularity conditions. EE.

we have with equality if and only if 1 = 2 Maximizing the LLK function is equivalently maximizing If 0 is the true value of . NCTU 39 . By the law of large numbers as N Institute of Communications Engineering.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Proof of the asymptotic property of the MLE First show the MLE is consistent (asymptotically unbiased) By the Kullback-Leibler information. EE.

Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Since by Kullback-Leibler information. NCTU 40 . EE. the MLE is consistent Institute of Communications Engineering. we have is maximized at = 0 By a continuity argument. we therefore have maximized at = 0 as N Thus.

and differentiable on the open interval (a. let gives the result Institute of Communications Engineering. b) such that Thus. b). b] where a < b. NCTU 41 .Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu We next show that the asymptotic PDF of the MLE By Taylor expansion at 0 and the mean value theorem where The mean value theorem: If f (x) is continuous in the closed interval [a. then there exists a c in (a. EE.

Beside.Unit 3: Maximum Likelihood Estimator But by the definition of the MLE. we have so This leads to Since . thus Sau-Hsuan Wu . EE. NCTU 42 . Institute of Communications Engineering. for IID PDF of x[n].

NCTU 43 .Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu On the other hand. for the numerator which is in fact a R. the variance is equal to Institute of Communications Engineering.V. EE. and by the independence of x[n]. with mean equal to by the regularity condition.

V.V. x. then “ xn/yn”has the same asymptotic PDF as the R.V. according to Slutsky’ s theorem: If a sequence of R. EE. and a sequence of R. yn converges to a constant c.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu Therefore. by the central limit theorem. i (0)) Now. NCTU 44 . “ x/c” As a result. and its denominator converges to i (0) Thus Institute of Communications Engineering. i (0)) asymptotically. since its numerator ~ N(0.V. the numerator converges to a Gaussian PDF ~ N(0. xn has the asymptotic PDF of the R.

EE.Unit 3: Maximum Likelihood Estimator Institute of Communications Engineering. NCTU Sau-Hsuan Wu 45 .

- Cox - Statistics PaperUploaded byMichael Pearson
- Notes 19Uploaded bySandeep Singh
- Draper Talk for Sf AsaUploaded byniksloter84
- Mthesisv2 Final CorrectedUploaded bySugumar Lakshmi Narayanan
- extremum_estimators_mle.pdfUploaded byVictor Haselmann Arakawa
- CR Lower Bound Fisher InfoUploaded byrigasti
- W4213 - Estimation TheoryUploaded byMiki Arsovski
- Statistical Inference ExercisesUploaded byognobogvoshno
- ajfisherUploaded byspotirak
- MLEUploaded bychatdem
- Fisher Information TutorialUploaded byManan
- Simulation Study of Parameter Estimation and Measurement Planning on Photovoltaics DegradationUploaded byWisse Prabawati
- Application of Stochastic Lognormal Diffusion Model WithUploaded byAlexander Decker
- The Epic Story of Maximum LikelihoodUploaded byoverlord303
- Dr Ost Klaas SenUploaded bymintajvshokolade
- Comp Neuro Syl Lab Us 2018Uploaded byAndrej Ilić
- part1Uploaded byDeepak Soman
- Guo´s Manuals SOA exam C.pdfUploaded byAustin Fritzke
- Chapter 5Uploaded byGharib Mahmoud
- JEE-2002-Vol-5-Issue-1Uploaded byAy Ch
- Chapter 4Uploaded byJose C Beraun Tapia
- 362assn7-solnsUploaded byKasih Pandu
- 19180282 Theory and Practice of Risk Assessment Icra 5 (Springer, Pms 136, 2015)Uploaded byinnoversitas
- Exercise 13.22Uploaded byLeonard Gonzalo Saavedra Astopilco
- Sources of Error in the Estimation of Mosquito Infection RatesUploaded byAni Ioana
- Sample Size and Power Estimates for a Confirmatory Factor Analytic Model in Exercise and Sport: A Monte Carlo ApproachUploaded bykakonauta
- A review of software fault detection and correction process, models and techniquesUploaded byDr Muhamma Imran Babar
- Using Regression Models to Analyze Randomized TrialsUploaded byKevyn Niu
- Curs 1 SSL - IntroductionUploaded byMihai Ilie
- Test Set 2 SIUploaded byReginaldRemo