# Unit 3 Maximum Likelihood

Estimator

Institute of Communications Engineering, EE, NCTU

1

Unit 3: Maximum Likelihood Estimator

Why do we use the maximum likelihood (MLE) estimator

Sau-Hsuan Wu

Often there is no obvious way to find the MVU estimator
The MLE is approximately efficient, thus approximately the
MVU estimator

We demonstrate this through an example

x[n] = A + w[n], n=0,1…,N-1, with w[n] ~ N(0,A), A > 0
The PDF is

Institute of Communications Engineering, EE, NCTU

2

Unit 3: Maximum Likelihood Estimator

Sau-Hsuan Wu

It is not obvious whether an efficient estimator exists
The CRLB is
Can we find the MVU estimator?
By the Neyman-Fisher factorization theorem, we attempt to
to factorize the PDF into

Institute of Communications Engineering, EE, NCTU

3

Unit 3: Maximum Likelihood Estimator

Sau-Hsuan Wu

Assume T(x) is a complete sufficient statistic
We may try to find the MVU estimator if

Since

not obvious how to choose g
Or we may try
Suppose Â= x[0]

if Â is an unbiased estimator
?

Institute of Communications Engineering, EE, NCTU

4

NCTU 5 . EE.Unit 3: Maximum Likelihood Estimator   Sau-Hsuan Wu We now have exhausted our possible optimal approaches We try to find an estimator that maximizes the likelihood  Differentiating the LLK  and setting it to zero produces  Solving for Â gives >0 Institute of Communications Engineering.

can we say anything about the MLE estimator?  Is it biased? E(Â) = E  But. EE. as N    Institute of Communications Engineering. NCTU 6 .Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu Now.

we have     Institute of Communications Engineering. Â  A as N    What is the variance of Â as N   ? We use the statistical linearization argument used in Unit 1 Since will be concentrated about E(x2)=A+A2 Let u = and  with Â = g(u) Linearizing about u0 = E(u) = A+A2 . NCTU 7 . EE.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu  Therefore.

1 E  N  It follows that E(Â) = A The asymptotic variance becomes  Since for a ~ N(. NCTU 8 . EE. 2)  Sau-Hsuan Wu  2 2 x [ n ]  A  A   n 0  N 1 Institute of Communications Engineering.Unit 3: Maximum Likelihood Estimator  As a result.

Besides. NCTU 9 .  Applying this formula gives Sau-Hsuan Wu CRLB   Â is asymptotically efficient.Unit 3: Maximum Likelihood Estimator  Thus. by the central limit theorem. EE. Then. when N    is is Gaussian too Institute of Communications Engineering. Gaussian.

   The first-order and second-order derivatives of the loglikelihood function are well defined (existence) The Fisher information is zero : then the MLE of the unknown parameter is asymptotically distributed according to where I() is the Fisher information evaluated at the true value of the unknown parameter Institute of Communications Engineering. ) of the data x satisfies some regularity conditions. EE.Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu Theorem If the PDF p(x. NCTU 10 .

…. n=0.N-1 Suppose A and f0 are given Institute of Communications Engineering.Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu Ex.1. The MLE of the sinusoidal phase   x[n]=A cos(2 f0 n + ) + w[n]. NCTU 11 . EE.

Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu  The MLE is found by minimizing  Differentiating with respect to produces  Setting it to zero yields  If f0 is not equal to 0 or ½. NCTU 12 . otherwise cos(2f0n+) = cos() Institute of Communications Engineering. EE.

EE. NCTU Sau-Hsuan Wu 13 .Unit 3: Maximum Likelihood Estimator  Then. we have  Upon expanding this we obtain  Finally. Institute of Communications Engineering.

NCTU 14 . EE. we have In this case.  so the asymptotic variance is  Sau-Hsuan Wu where = (A2/2)/2 is the SNR Institute of Communications Engineering.Unit 3: Maximum Likelihood Estimator  According to the Theorem.

EE. the variance could be smaller than the CRLB Institute of Communications Engineering. NCTU 15 .08.Unit 3: Maximum Likelihood Estimator   Sau-Hsuan Wu Suppose A=1.05 The effect of data length is simulated below  When estimator is biased. = /4 and 2 = 0. f0 =0.

NCTU 16 . EE.Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu When N=80 Only at high SNR is the CRLB attained Institute of Communications Engineering.

Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu The required data record length for the asymptotic results to apply also depends on the SNR SNR = 10dB SNR = -15dB Institute of Communications Engineering. NCTU 17 . EE.

the CRLB may be attained even for short data records if the SNR is high enough Rewrite If the data record is large and/or the sinusoidal power is large. the noise terms will be small. NCTU 18 .Unit 3: Maximum Likelihood Estimator    Sau-Hsuan Wu For signal in noise problem. allowing the MLE to attain it’ s asymptotic distribution Institute of Communications Engineering. EE.

we may well estimate a function of  For example. a signal is attenuated by a shadowing effect  [n] s.2)  Or.t. with w[n] = ln( [n]) ~ N(0.Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu MLE for transformed parameters    In many instances. NCTU 19 . EE.  The MLE of : Institute of Communications Engineering. the received signal is modeled as  [n] =  [n] More convenient to estimate ln() = A from x = ln( [n]) because x[n] = A + w[n].

A)   A   0   2 = Aˆ x 2 Institute of Communications Engineering. pT ( x. thus The maximum likelihood estimator is ˆarg max{ pT ( x.Unit 3: Maximum Likelihood Estimator    Sau-Hsuan Wu Now. ). p( x. ). ). we consider another transformation of = A2 The transformation is not one-to-one : A   . EE.)} arg max p( x.  )   1  2 0  2   2   = arg max p( x. NCTU 20 . p( x.  ) =  arg max p( x.

then   ˆarg max pT ( x. NCTU 21 . EE. ) is parameterized by .)    : g () Institute of Communications Engineering. is given by ˆ ˆg (  )  ˆis the MLE of obtained by maximizing p(x. ) where  If g is not a one-to-one function.Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu Invariance property of the MLE  The MLE of the parameter = g(). ) arg max max p( x. where the PDF p(x.

Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu MLE for Gaussian linear model  Recall that the PDF of Gaussian vector parameters p ( x. EE. θ) 1 ln det  T 1    ( x  μθ ( )) C (θ)( x μθ ( ))     k 2  k 2 k  It can be shown that (see Appendix 3C for details) ln det  C (θ)   1  C (θ)  tr  C (θ)   k   k   Institute of Communications Engineering. NCTU 22 . θ)  1 N 2 1 2 1  exp   ( x μθ ( ))T C 1 (θ)( x μθ ( ))  2  2 det C (θ)   Thus C (θ)  1  ln p( x.

θ) ( Hθ)T 1  C ( x Hθ) 0  θ  θ  θ H C H  H T C 1 x T 1 1 Institute of Communications Engineering. we have ln p( x.Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu As a result. NCTU 23 . consider the case of x = H+ w whose PDF is p ( x. θ) 1  1  C (θ)   μθ ( )T 1  tr  C (θ) C (θ)( x μθ ( ))   k 2   k  k 1 1  C (θ)  ( x μθ ( ))T ( x μθ ( )) 2  k  Now. EE. θ)   Thus 1  exp   ( x Hθ)T C 1 ( x Hθ)  2  1 N 2 1 2 2 det C  ln p( x.

1.Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu The MLE of the sinusoidal parameter estimation  x[n]=A cos(2 f0 n + ) + w[n]. f0 and  The PDF is given by  MLE is found by minimizing   Institute of Communications Engineering. EE. n=0. NCTU 24 .N-1  Suppose A > 0 and 0 < f0 < ½ We want to find the MLE of A.….

1 c.2 s)T (x.2 s) = (x-H)T (x-H) where = [1. NCTU 25 . 2 2 A  1 2 Also let 2  arctan   1  c = [1 cos2f0 … cos2f0 (N-1)]T s = [0 sin2f0 … sin2f0 (N-1)]T  Then J’ (1. EE. 2]T and H = [c s] Institute of Communications Engineering.Unit 3: Maximum Likelihood Estimator     Sau-Hsuan Wu Notice that J is nonquadratic in A and . 2.1 c. thus we need to do some parameter transformations Let 1 = A cos and 2 = -A sin  As a result. f0) = (x.

 . we can find f0 through xTH (HTH)-1HTx Institute of Communications Engineering. EE.Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu The cost function is exactly what encountered in Gaussian linear model with C=I. ˆ α H H  HT x T  1 The cost function becomes T ˆ ˆ ˆ ˆ J '( . f )  x  Hα  x Hα  1 2 0   =x  I H  H H  H x = x H  H H H x T  1 T T T 1 T x H  H H  HT x T 1 T since I-H (HTH)-1HT is an idempotent matrix (A2=A) Since H = [c s]. NCTU 26  .

we have  Similarly. NCTU 27 . EE. we also have cTc N/2 and sTs N/2 Institute of Communications Engineering.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu  Rewrite xTH (HTH)-1HTx into  When f0 is not close to 0 or ½.

NCTU 2 0 28 .Unit 3: Maximum Likelihood Estimator  Then. 2 fˆ = arg max 0 N f0 Sau-Hsuan Wu N 1 x[n]exp(j 2f n) n 0 Institute of Communications Engineering. xTH (HTH)-1HTx becomes  Therefore. EE.

hence H  1 N  0 T    1 ˆ   N c 2 T T ˆ α H H  H x   T x  sˆ 2 0 N    2   N N 1  ˆ 2 x[n ]cos 2f 0n  n 0   N 1 N ˆn  x [ n ]sin 2  f 0  2   n 0  Institute of Communications Engineering. NCTU T  cˆ x T  s x  29 .Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu ˆ [cˆsˆ Given fˆ ] 0 . EE.

NCTU 30 . EE.Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu Applying the invariance property of the MLE. we obtain 2 2 2 ˆ ˆ ˆ A   1  2  N N 1 x[n]exp( j 2fˆn) n 0 0  N 1  ˆ x[n ]sin 2f 0n    ˆ  2 0 ˆ  arctan  arctan Nn 1 ˆ  x[n ]cos 2fˆn  1   0   n 0  Institute of Communications Engineering.

NCTU 31 . a [ p ]  2 u   | A( f ) | The asymptotic log likelihood function can be shown to be 1 N N N 2 2 ln P( x.Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu Parameter estimation of an AR process  x[n] + a[1]x[n-1] +…+ a[p]x[n-p] = u[n] p    A( z ) 1 a[m]z m 1 m p  A( f ) 1 a[m]exp( j 2fm ) m 1  2 Pxx ( f . EE. a. θ)    θ  a [1] a [2]...u2 )  ln 2 ln u2  2  | A ( f ) | I ( f )df 1  2 2 2u 2 2 Where N 1 1 I( f ) = x[n ]exp( j 2f 0n )  N n 0 is the periodogram of the data and Pxx(f ) is the PSD 2 u Institute of Communications Engineering.

t.Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu Differentiating w. which is a function of a through A( f ). we must minimize u . EE. u2 and setting the result to zero yields and thus  Substituting this back into the asymptotic LLK results in  2 ˆ ˆ  a To find .r. NCTU 32 . Institute of Communications Engineering.

NCTU 33 . for k=1. EE.f ) = A*( f ) and I( .…. we have  Besides.f ) = I( f ) Institute of Communications Engineering. A( .p.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu  Define  Since J(a) is quadratic in a.

This can be shown to be Institute of Communications Engineering.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu  Setting it equal to zero produces  or  But is just the inverse Fourier transform of the periodogram evaluated at k. EE. NCTU 34 .

Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu  The set of equations to be solved for the approximate MLE of the AR filter parameters a becomes  Or in a matrix form  This can be solved with the Levison recursion Institute of Communications Engineering. NCTU 35 . EE.

NCTU 36 . k=1.p  The zero derivative condition  All terms in the sum are zero except for k=0 Institute of Communications Engineering. EE.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu  We complete the discussion with an explicit form of the 2 MLE of u Since  Note that by the optimality of aˆ [k ].….

the MLE is given by Institute of Communications Engineering.Unit 3: Maximum Likelihood Estimator  By definition. aˆ [0] 1 . EE. It follows  Finally. NCTU Sau-Hsuan Wu 37 .

Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu Asymptotic PDF of MLE for a scalar parameter If the PDF p(x.    The first-order and second-order derivatives of the loglikelihood function are well defined (existence) The regularity condition of then the MLE of the unknown parameter is asymptotically distributed according to where I() is the Fisher information evaluated at the true value of the unknown parameter Institute of Communications Engineering. NCTU 38 . ) of the data x satisfies some regularity conditions. EE.

we have  with equality if and only if 1 = 2 Maximizing the LLK function is equivalently maximizing  If 0 is the true value of . NCTU 39 . By the law of large numbers  as N   Institute of Communications Engineering.Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu Proof of the asymptotic property of the MLE  First show the MLE is consistent (asymptotically unbiased) By the Kullback-Leibler information. EE.

Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu Since by Kullback-Leibler information. NCTU 40 . EE. the MLE is consistent  Institute of Communications Engineering. we have  is maximized at = 0 By a continuity argument. we therefore have  maximized at = 0 as N   Thus.

and differentiable on the open interval (a. let gives the result Institute of Communications Engineering. b) such that Thus. b). b] where a < b. NCTU 41 .Unit 3: Maximum Likelihood Estimator     Sau-Hsuan Wu We next show that the asymptotic PDF of the MLE By Taylor expansion at 0 and the mean value theorem where The mean value theorem: If f (x) is continuous in the closed interval [a. then there exists a c in (a. EE.

Beside.Unit 3: Maximum Likelihood Estimator  But by the definition of the MLE. we have so  This leads to  Since . thus Sau-Hsuan Wu . EE. NCTU 42 . Institute of Communications Engineering. for IID PDF of x[n].

NCTU 43 .Unit 3: Maximum Likelihood Estimator  Sau-Hsuan Wu On the other hand. for the numerator  which is in fact a R. the variance is equal to Institute of Communications Engineering.V. EE. and by the independence of x[n]. with mean equal to  by the regularity condition.

V.V. x. then “ xn/yn”has the same asymptotic PDF as the R.V. according to Slutsky’ s theorem: If a sequence of R. EE. and a sequence of R. yn converges to a constant c.Unit 3: Maximum Likelihood Estimator Sau-Hsuan Wu  Therefore. by the central limit theorem. i (0)) Now. NCTU 44 . “ x/c” As a result. and its denominator converges to i (0) Thus   Institute of Communications Engineering. i (0)) asymptotically.  since its numerator ~ N(0.V. the numerator converges to a Gaussian PDF ~ N(0. xn has the asymptotic PDF of the R.

EE.Unit 3: Maximum Likelihood Estimator Institute of Communications Engineering. NCTU Sau-Hsuan Wu 45 .