Unit 1 Minimum Variance

Unbiased Estimator

Institute of Communications Engineering, ECE, NCTU

1

Unit 1: Minimum Variance Unbiased Estimator

What are estimations all about?



Sau-Hsuan Wu

We want to know (or estimate) an unknown quantity 
This quantity, , may not be obtained directly, or it is a
notion (quantity) derived from a set of observations
In any event, we need to retrieve the value of from a set of
observations, x = x[0],…, x[N-1], which are related to 
We hope to determine according to

How do we start?

Suppose we have some knowledge about the collected data
Examples:

x[n] = A1 + w[n],
x[n] = A2 + B2 n + w[n],

n=0,…,N-1
n=0,…,N-1

Institute of Communications Engineering, ECE, NCTU

2

Unit 1: Minimum Variance Unbiased Estimator

Sau-Hsuan Wu

Â1= 
x[n]/N
and ?

An estimator may be thought of as a rule that assigns a
value to for each realization of x
The estimate of is the value of for a given realization
of x
How do we assess the performance of estimation?

What are we concerned the most about the estimate?


How close will  be to A?
Are there better estimators than the sample mean for instance?
Ă= x[0] ?

What are the differences between Âa
ndĂ?

Institute of Communications Engineering, ECE, NCTU

3

the PDFs are parameterized by .  As an example:  We have a class of PDFs where each one is different due to a different value of . The parameter is assumed to be deterministic but unknown Given the PDF.e. ECE.Unit 1: Minimum Variance Unbiased Estimator   Sau-Hsuan Wu Can be have some performance measure? We model the data by its probability density function (PDF). i. assuming that the data are inherently random. NCTU 4 . we calculate some statistics of the estimates     E (Â ) = E (Ă) ? Var (Â) = Var (Ă) ? Institute of Communications Engineering.

but one that is also mathematically tractable Sometimes. then the data are described by the joint PDF Any estimator that yields estimates according to the prior knowledge of is termed a Bayesian estimator Institute of Communications Engineering.    To incorporate this prior knowledge.Unit 1: Minimum Variance Unbiased Estimator   Sau-Hsuan Wu Note: In an actual problem. ECE. we are not given a PDF but must choose one that is not only consistent with the problem constraints and any prior knowledge. we might want to constraint the estimator to produce values in a certain range. U] interval for instance Assign a PDF to . NCTU 5 . we can assume that  is no longer deterministic but a random variable having a uniform distribution over the [-U.

Unit 1: Minimum Variance Unbiased Estimator    Sau-Hsuan Wu We conclude for the time being that we are hoping to have an estimator that gives  An unbiased mean of the estimate:  A minimum variance for the estimate: Is an unbiased estimator always the optimal estimator? Consider a widely used optimality criterion: the minimum mean squared error (MSE) criterion Institute of Communications Engineering. ECE. NCTU 6 .

r. ECE. we have mse (Ă) = a2 2 / N + (a-1)2A2 Taking the derivative w.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu  As an example. to a and setting it to zero leads to aopt = A2 / (A2 +  2/ N ) The optimal value of a depends upon the unknown parameter A  The estimator is not realizable How do we resolve this dilemma?  Since. NCTU 7 . mse (Ă) = var(Ă) + b2. we set b=0 and search for the estimator that minimizes var(Ă)  minimum variance unbiased (MVU) estimator Institute of Communications Engineering. as an alternative.t. consider the modified estimator  We attempt to find the ‘ a’which yields the minimum MSE     Since E(Ă)=aA and var(Ă) = a2 2 / N.

Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu However.1) x[1] ~ N(. ECE. NCTU 8 . does a MVU estimator always exist for all ?  Example:    x[0] ~ N(.1) if 0 ~ N(.2) if 0 Both of the two estimators are unbiased   Therefore  None of the estimator has a variance uniformly less than or equal to 18/36 Institute of Communications Engineering.

ECE.Cramer-Rao Lower Bound Institute of Communications Engineering. NCTU 9 .

2).Unit 1: Minimum Variance Unbiased Estimator   How to measure the accuracy of estimation? Consider a single sample of observation x[0] = + w[0] with w[0]~N(0.    Sau-Hsuan Wu The unbiased estimator is  and the variance is The accuracy of estimation improves as 2 decreases We can see this from the likelihood function (LK) of  Institute of Communications Engineering. ECE. NCTU 10 .

ECE. the sharpness of the LK is measured by the curvature of the LLK.e. in a more rigorous definition. NCTU 11 . which is inversely proportional to i2 in this case Actually.  In general. the variance i2 in this case is the negative inverse of the second derivative of the logarithm of the likelihood function (LLK)  Therefore. the curvature depends on x[0] as well. i.Unit 1: Minimum Variance Unbiased Estimator   Sau-Hsuan Wu The accuracy of estimation increases with the “ sharpness” of the LK. thus a more appropriate measure of curvature is Institute of Communications Engineering.

) Institute of Communications Engineering.Unit 1: Minimum Variance Unbiased Estimator  Now. p(x . namely The variance of the estimation is given by  We want to find a lower bound for     Sau-Hsuan Wu How do we start?  The hint from the previous example is that the variance is inversely proportional to the average curvature of p(x . NCTU 12 . ) which is parameterized by  Consider all unbiased estimators . ECE. we put our problem in a more rigorous statement  Define a scalar parameter = g() Define a PDF of the observation.

NCTU 13 . the smaller the lower bound for the variance of estimation By the Cauchy-Schwarz inequality. we have Institute of Communications Engineering.Unit 1: Minimum Variance Unbiased Estimator  Therefore. ECE. we may mathematically associate the variance with the average curvature like this   Sau-Hsuan Wu The larger the curvature.

ECE.Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu Then. NCTU 14 . can we relate the square of the first-order derivative to the second-order derivative? i.e. the connection between and  Observe that if the integral and partial are interchangeable Institute of Communications Engineering.

we have supposed that  Substituting the results to the Cauchy-Schwarz inequality Institute of Communications Engineering.Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu Then. ECE. NCTU 15 .

NCTU 16 .Unit 1: Minimum Variance Unbiased Estimator  Therefore. ECE.  The equality holds if and only if  where c might be a function of but not of x If =g()= . then Sau-Hsuan Wu Institute of Communications Engineering.

t. on the both sides leads to  Taking the expectation on the both sides.r.Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu Taking the derivative w. for unbiased estimators we have  Thus. ECE. NCTU 17 . Institute of Communications Engineering.

Unit 1: Minimum Variance Unbiased Estimator

Sau-Hsuan Wu

Clearly, the Cramer-Rao bound (CRB)

is valid if the regularity condition holds

Does the regularity condition always hold?

Refer to Leibniz’
s rule

Institute of Communications Engineering, ECE, NCTU

18

Unit 1: Minimum Variance Unbiased Estimator

Sau-Hsuan Wu

The gradient, with respect to , of the log likelihood
function (LLK) is called the score

Under the regularity condition, the expected value of the
score is zero, namely

The Fisher information is the variance of the score

Institute of Communications Engineering, ECE, NCTU

19

Unit 1: Minimum Variance Unbiased Estimator

By definition,

If the regularity condition holds,

Sau-Hsuan Wu

then the information is additive for independent x and y

Therefore, it is referred to as the information of 

Institute of Communications Engineering, ECE, NCTU

20

NCTU 21 . d 0 The goal is to prove Cα ˆ  T 1  a C  I (θ)  Namely.  αˆ a 0 Institute of Communications Engineering. θ)  ln p ( x. θ)  I (θ)      p( x. θ)dx T θ  θ       1  I (θ) p. θ)      p ( x. θ)dx θ  θ      ln p ( x. s . ECE. θ)dx Cαˆ (α T ln p ( x.Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu Cramer Rao’ s lower bound for vector parameter   Let vector = g( ) and define ˆα)( α ˆα)T p( x. for arbitrary a. θ)  ln p ( x.

θ)dx a    T Cαˆ   ln p ( x. θ)   T ˆ a  α b dx a α b dx       θ  θ        T  aT    ˆ  α p( x. θ)dx   b T  θ     T 2 2   g ( θ )  T   a b   θT      Institute of Communications Engineering. θ)dx      θ     2 T    p( x. θ)  ln p ( x. ECE. θ)   p( x. θ)dx b     θ   θ      2 I (θ) T  T  ln p( x.Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu Cauchy-Schwarz inequality T ˆ ˆ aT  ( α  α )( α  α ) p( x. NCTU 22 . θ)  T  b  p( x. θ)   ˆα)   a (α b p( x.

since b is arbitrary. by Cauchy-Schwarz inequality.Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu Therefore. we can define T  g (θ) -1 b I (θ) a  θ  As a result 2 2 T g (θ)  T  g (θ) -1  g (θ)  T  b = a I (θ) a a T T θ    θ  θ    T  g ( θ )  g (θ) T T -1 -1 a Cαˆaa I (θI ) (θI ) (θ) a T  θ  θ Institute of Communications Engineering. we have 2  g (θ)  T T  T a b  a C ab I (θb ) ˆ α   T θ    Besides. ECE. NCTU 23 .

so is I-1().d 0 T  θ  θ Institute of Communications Engineering.Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu Since I() is positive definite. thus  g (θ) -1  g T (θ) CˆI (θ) p. ECE. NCTU 24 . We have 2 T T  g (θ) -1  g (θ)  T  g ( θ )  g (θ) T -1 I (θ) a a Cαˆaa I (θ) a a T T θ  θ   θ  θ   T T  g (θ) -1  g (θ)   a Cˆa a I (θ) a T θ  θ    T T positive semidefinite T   g ( θ )  g (θ)  T -1 a  CˆI (θ) a 0  T θ  θ     Since a is arbitrary. s.

θ) 1 ˆα)  I ( θ )  (α T  θ  θ c(θ)  Thus. θ )  g (θ) T -1 ˆ a ( αα) c(θ) b c(θ) I (θ) a T T  θ  θ  θ  For arbitrary a  g (θ) -1 ln p( x. for = g() =  ln p( x. NCTU 25 . ECE. θ) 1 ˆθ)   I (θ)(θ  θ c(θ) Institute of Communications Engineering. θ )  ln p ( x .Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu When g()= . we have Cθˆ-I -1 (θ) 0  Since the conditions for equality are T  ln p ( x .

NCTU 26 .r. θ)  1  I (θ) E  I (θ)  c(θ)=1  T θθ  c(θ)    Eventually.t. we have ˆθ)     Ii (θ)  ˆ  1  Ii (θ)  ˆ  I i (θ)(θ (θθ)  (θθ)  I i (θ)     T   θ  c(θ) 1 c(θ)  N c(θ)       c(θ)    Ii (θ)  ˆ  1  I i (θ)  ˆ 1  E ( θ  θ )  E ( θ  θ )  I ( θ )=  I i (θ)   x  x i 1 c(θ)  N c(θ)  c(θ)     c(θ)  Putting the row vectors into a matrix returns 2 ln p( x.   Denote the i-th row of I()/c( ) by Ii()/c(). θ) ˆθ) I (θ)(θ  θ Institute of Communications Engineering. ECE. we arrive at ln p( x.Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu Now take the derivative on the both sides w.

achieves the CRLB too Cˆ=ACθˆΑ T I (θ)  θ  θ Institute of Communications Engineering. i. ECE.e.Unit 1: Minimum Variance Unbiased Estimator  For a positive semidefinite matrix A  [A]ii 0   Sau-Hsuan Wu Since xTAx 0 x  eiTAei = [A]ii 0 Therefore. C  I (θ) ) θ ˆ θ Now. for a linear transformation = g() = A+ b ˆb  E (α ˆAθ ˆ α ) α and  g (θ) -1  g T (θ) . Cθˆ-I -1 (θ) 0 -1 ˆ)    C  I (θ)   var( i  θˆ   ii ii   1 ˆ Suppose E (θ and achieves the CRLB. NCTU 27 .

= [A B]T and H       ln p( x. x[N-1]]T .Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu We revisit the line fitting problem   1 0   x[n] = A + Bn + w[n].N-1   1 1  Let x = [x[0].…. n=0. ECE.…. NCTU 28 . θ) T T 2 1 N  1     H x H Hθ /    θ I (θ) H T H / 2 1 ln p( x. θ) T  ˆθ  I (θ)  H H  H T x θI (θ)  θ        θ  An estimator which is unbiased and attains the CRLB bound is said to be efficient in that it efficiently uses the data Institute of Communications Engineering.

now 2(2 N 1)2 2 122 var( Aˆ )  Sau-Hsuan Wu N ( N 1)  if N 2 and var( Bˆ ) N N ( N 2 1) Thus. NCTU 29 . ECE. the CRLB always increases as we estimate more parameters Institute of Communications Engineering.Unit 1: Minimum Variance Unbiased Estimator   Efficiency is maintained over a linear transformation Let us take a closer look at I() = HTH/2  N  2 I (θ)  N ( N 1)   22  N ( N 1) 2(2 N 1)2   N ( N 1)  22 . I 1 (θ)   N ( N 1)(2 N 1)   62  2  6   N ( N 1) 62   N ( N 1)   2 12   N ( N 2 1)  Compared with x[n] = A + w[n] in which var(Â) = 2/N.

I) Based on the previous result. we can use a whitening approach Let C-1 = DTD  DE{w wH} DT = I x’= D x = D(H+ w ) = H’+ w’with w’~ N(0. ECE. NCTU 1 30 .C) To derive the MVU.Unit 1: Minimum Variance Unbiased Estimator       Sau-Hsuan Wu Extension to linear model x = H+ w with w ~ N(0. we have 1 1 T T T T ˆ θ H ' H ' H ' x ' =  H D DH  H T DT Dx = H C H  H T C 1 x T and 1 1 Cθˆ H ' H ' =  H D DH  =  H C H T 1 T T 1 T 1 Institute of Communications Engineering.

why?  The variance depends on the gradient of g() Institute of Communications Engineering. ECE.Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu Ex. NCTU 31 . n=0. g (θ) 2 A A2   2  θ   T  Recall that N 2   I (θ)  0     4   0   g (θ) 1  g T (θ) 4 A2 2 A4 422 I (θ)  2 4   T N   θ  θ N N N 24   The variance of estimation increases with the SNR .N-1 Let = [A σ2]T and g() = 12/2 = A2 /σ2 =  Then. Estimation of the signal-to-noise (SNR) ratio    Given x[n] = A + w[n].….

Sufficient Statistics Institute of Communications Engineering. NCTU 32 . ECE.

x[N-1]} S3 = {x[n]}  x[n] is also a statistic Then. x[1]. what is the minimum set of sufficient statistics?  Given x[n] = T0 .Unit 1: Minimum Variance Unbiased Estimator   Is there a set of T(x) of x that is sufficient for estimation? What do we mean by the sufficiency of a set of T(x)?   We want to have a set of statistics T(x) such that given T(x). NCTU 33 .x(N-1)] is independent of A Suppose Â1=  x[n]/N. do we still need the individual data? Institute of Communications Engineering.…. x[N-1]}  each element is a statistic S2 = {x[0]+x[1]. x[2]. ECE.…. then are the followings sufficient?     Sau-Hsuan Wu S1 = {x[0].…. any x(n) of x = [x(0).

a value of A near A0 is more likely even given T0 For (b).Unit 1: Minimum Variance Unbiased Estimator     Sau-Hsuan Wu We say the conditional PDF : p( x | x[n] = T0 .g. p( x | x[n] = T0 . ECE. For (a). A) should not depend on A if statistic T0 is sufficient E. A) is a constant Institute of Communications Engineering. however. NCTU 34 .

we have  Institute of Communications Engineering. NCTU 35 . ECE.  Clearly. A) to show that x[n] = T0 is sufficient By Baye’ s rule  Since T(x) is a direct function of x.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu  Now. we need to determine p( x | x[n] = T0 .

we have Institute of Communications Engineering. NCTU 36 . ECE.Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu Thus.

NCTU 37 .Unit 1: Minimum Variance Unbiased Estimator  Since T(x) = x[n] ~ N(NA. ECE. N2)  Thus Sau-Hsuan Wu which does not depend on A Institute of Communications Engineering.

)=g(T(x).) into p(x. to identify potential sufficient statistics is difficult An efficient procedure of finding the sufficient statistics is to employ the Neyman-Fisher factorization theorem Observe that If we can factorize p(x.) h(x) where     g is a function that depends on x only through T(x) h is a function that depends only on x  T(x) is a sufficient statistic for  The converse is also true If T(x) is a sufficient statistic  p(x.)=g( T(x). ECE.) h(x) Institute of Communications Engineering.Unit 1: Minimum Variance Unbiased Estimator     Sau-Hsuan Wu In general. NCTU 38 .

ECE.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu  Recall p(x. we want to estimate 2 of y[n]=A+x[n]   Suppose A is given. NCTU 39 . then define x[n] = y[n]-A Clearly.A)  Now. T(x) = x2[n] is a sufficient statistic for 2 Institute of Communications Engineering.

p(x. NCTU 40 . )h(x)   Since Institute of Communications Engineering.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu  Proof of the Neyman-Fisher Factorization Theorem ()  By assumption.) = g(T(x). ECE.

ECE.Unit 1: Minimum Variance Unbiased Estimator  We have  Then  which does not depend on  Hence T(x) is a sufficient statistic Sau-Hsuan Wu Institute of Communications Engineering. NCTU 41 .

T0) =1 Therefore.T0) with w(x)  (T(x) . ECE.) = P(x| T(x)=T0) ( not a function of ) We can define p(x| T(x)=T0 . we can let Institute of Communications Engineering. NCTU 42 .Unit 1: Minimum Variance Unbiased Estimator  () Recall  And  Suppose T(x) is a sufficient statistic     Sau-Hsuan Wu Then p(x| T(x)=T0 .) = w(x)  (T(x) .

we have and  This holds for arbitrary T0. ECE. resulting in where Institute of Communications Engineering. NCTU 43 .Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu As a result.

NCTU 44 .Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu Ex.N-1  Suppose A and f0 are given  Expand the exponent Institute of Communications Engineering. we want to estimate the phase of a sinusoid x[n]=A cos(2 f0 n + ) + w[n].1. n=0. ECE.….

however g(T1(x). NCTU 45 . no single sufficient statistic exists.T2 (x).Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu In this case. ECE. ) h(x) Institute of Communications Engineering.

T2(x).Unit 1: Minimum Variance Unbiased Estimator      Sau-Hsuan Wu The r statistics T1(x). then is unbiased and    A valid estimator for (not dependent on ) Of lesser or equal variance than that of for all  If T(x) is complete. Tr(x) are jointly sufficient if p(x | T1(x). Tr(x) . Tr(x)} are sufficient statistics for  Now. T2(x).…. Tr(x) . ) does not depend on  If p(x . then is the MVU estimator Institute of Communications Engineering.….…. T2(x).…. ) h(x)  {T1(x). we know how to obtain the sufficient statistics How do we apply them to help obtain the MVU estimator? The Rao-Blackwell-Lehmann-Scheffe Theorem  If is an unbiased estimator of and T(x) is a sufficient statistic for . NCTU 46 . ECE. ) = g(T1(x). T2(x).

) is not a function of  after the integration w.t. ECE.Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu Proof  1> validity  by definition p(x|T(x) . x. NCTU 47 .r. the result is not a function of but T 2> unbiasedness By assumption Institute of Communications Engineering.

NCTU 48 . ECE.Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu 3> show  Recall that  If 0 0 is solely a function of T(x) Institute of Communications Engineering.

the MVU can be found by   Taking any unbiased and carrying out Alternatively. a statistic is complete if there is only one function. must be the MVU In summary.Unit 1: Minimum Variance Unbiased Estimator       Sau-Hsuan Wu Finally. say g. for all and any unbiased estimator Then. since there is only one function of T(x) that leads to an unbiased estimator  find the unique g(T(x)) that makes unbiased Institute of Communications Engineering. NCTU 49 . of the statistic that is unbiased is solely a function of T(x)  If T(x) is complete  is unique and unbiased Since T(x) is sufficient. is as good as with another T1(x) Besides. ECE.

Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu Completeness of a sufficient statistic  We know that for x[n] = A + w[n]. A Since T ~ N(NA. ECE. NCTU 50 . N2)   where v(T) = g(T(x)) -h(T(x)) Let = T /N and v’ ( ) = v(N  )   W( ) Institute of Communications Engineering. 2) T(x) = x[n] is sufficient and g(T(x)) = T(x)/N is unbiased Suppose. a second function h for which E{h(T(x))} = A  E{g(T(x)) -h(T(x))} = A –A =0. with w[n]~ N(0.

f Since W(f) is still Gaussian  V’ (f) = 0  v’ ( ) = 0. thus T(x) is complete Institute of Communications Engineering. ECE.   g(T(x)) = h(T(x)). NCTU 51 .Unit 1: Minimum Variance Unbiased Estimator   Sau-Hsuan Wu Which is a convolution of v’ ( ) and w( ) and =0 A  v’ ( )=0 Recall a signal v(t) is zero iff F(v(t)) = 0     F(v’ ( )*w( )) = V’ (f)W(f) = 0.

while w[0] ~ U[-1/2. is T(x) completely sufficient? Let v(T) = g(T(x)) -h(T(x))  However. so that   but Institute of Communications Engineering. x = x[0] = T.Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu Incomplete sufficient statistic  Now consider x[0] = A + w[0]. NCTU 52 . ECE. 1/2] T(x) = x[0] is sufficient. But.

NCTU 53 . ECE.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu  So that  A nonzero v(T) = sin (2T) will satisfy this condition Institute of Communications Engineering.

T Institute of Communications Engineering. we say a sufficient statistic is complete if is satisfied only by the zero function or by v(T) = 0. v(T) = g(T) –h(T) = sin (2T) Let g(T) = T = x[0] since E{x[0]} =A  h(T) = T . NCTU 54 .sin (2T)   = x[0] .Unit 1: Minimum Variance Unbiased Estimator       Sau-Hsuan Wu Hence.  not sure if  = x[0] is an MVU estimator To summarize. ECE.sin (2x[0]) is also an unbiased estimator of A. using statistic T = x[0]  x[0] is not complete.

3> Find a function g(T(x)) that yields an unbiased estimation of which is the MVU of  Alternatively. ECE. NCTU 55 .Unit 1: Minimum Variance Unbiased Estimator   Sau-Hsuan Wu As a summary. the RBLS method can be used to find the MVU even when an efficient estimator does not exist The procedure we learn by now to find an MVU estimator     1> Find a sufficient statistic T(x) for  by the Neyman-Fisher factorization theorem 2> Determine if T(x) is complete. where is an unbiased estimator Institute of Communications Engineering. if so.

) / } = 0 Is a sample mean the MVU estimator for this case?  Its variance is    Institute of Communications Engineering.….1.Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu Ex: Mean of uniform noise  x[n] = w[n] . NCTU 56 . w[n] ~ U[0. ] Want to find the MVU estimator for the mean = /2 The initial approach of using the CRLB to find an efficient estimator cannot even be tried for it does not satisfy the regularity condition : E{ln P(x. ECE. n=0.N-1.

we follow the procedure we learned so far  Define  Then  where =2 The PDF is or Institute of Communications Engineering. ECE.Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu Now. NCTU 57 .

ECE.Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu Alternatively so that    T(x) = max(x) We need to determine a function g to make T(x) unbiased To do so requires us to determine E{T(x)} Institute of Communications Engineering. NCTU 58 .

ECE.Unit 1: Minimum Variance Unbiased Estimator  The CDF of T(x)  The PDF follows as Sau-Hsuan Wu Institute of Communications Engineering. NCTU 59 .

Unit 1: Minimum Variance Unbiased Estimator  But d Pr{x[n] <  }/d is the PDF of x[n] or  Integrating. we obtain Sau-Hsuan Wu Institute of Communications Engineering. ECE. NCTU 60 .

NCTU 61 . ECE.Unit 1: Minimum Variance Unbiased Estimator  Which finally yields  We now have Sau-Hsuan Wu Institute of Communications Engineering.

we multiply T(x) by (N+1)/N  which is the MVU estimator whose corresponding variance is and  < 2/(12N) of the sample mean Institute of Communications Engineering. ECE.Unit 1: Minimum Variance Unbiased Estimator  Sau-Hsuan Wu To make it unbiased. NCTU 62 .

Tr(x)]T is said to be sufficient for the estimation of    If P(x| T(x) . ECE. ) h(x)  T(x) is of minimum dimension Institute of Communications Engineering. NCTU 63 . ) =g(T(x). a vector T(x) = [T1(x).Unit 1: Minimum Variance Unbiased Estimator    Sau-Hsuan Wu Extension to a vector parameter  We want to seek an unbiased vector estimator such that each element has the minimum variance Similarly. ) = P(x| T(x)) If P(x . T2(x)….

2 ]T is the unknown vector parameter The PDF is  Expanding the exponent.…. =[A. f0 . we obtain  Cannot reduce the PDF due to    Institute of Communications Engineering. n=0. NCTU 64 .Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu  Back to a similar example x[n]=A cos(2 f0 n) + w[n].N-1 Now.1. ECE.

NCTU 65 . 2 ]T  We are able to factorize the PDF which gives N 1   x[n ]cos(2f 0n )    T1 ( x )  n 0   T ( x )    N 1 T2 ( x )     2 x [n ]    n 0   Institute of Communications Engineering.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu  If f0 is known. then =[A. ECE.

we have N 1  x[n ]    T1 ( x )  n 0   T ( x )    N 1 T2 ( x )     2 x [n ]   n 0    Taking the expected values produces  NA   NA  E{T ( x )}   2 2 2    NE { x [ n ]} N (   A )    Institute of Communications Engineering. =[A. ECE. 2 ]T Set f0 = 0 for x[n]=A cos(2 f0 n) + w[n].N-1 Again.Unit 1: Minimum Variance Unbiased Estimator     Sau-Hsuan Wu Now apply the above result to our previous discussion x[n]=A + w[n]. n=0.1. NCTU 66 .….

ECE.Unit 1: Minimum Variance Unbiased Estimator   So T2(x) only helps estimate the second moment not the variance If we transform T (x) into 1   T ( x )  1   N   g (T ( x ))   2 1  1 1  N N T2 ( x ) N T1 ( x )       Sau-Hsuan Wu   N 1 2 2 x [ n ]  x   n 0  x Then. E{g(T(x))} gives E  x   A     E{g (T ( x ))}  1 N 1 2  2 2 2  2   A E x  E  x [n ] x        N n  0     Institute of Communications Engineering. NCTU 67 .

2/N)  Sau-Hsuan Wu 2  E x A  N 2 2  Since  Substituting this back into E{g(T(x))} yields A  A     E{g (T ( x ))}  2  2 2 2 2  2 2   A E x   A A  N          Therefore.Unit 1: Minimum Variance Unbiased Estimator x ~ N (A. multiply the second element by N/(N-1) yields an unbiased estimator of 2 1 T ( x)   1    N    N 1 2   g (T ( x ))  1      2 2 1 1    x [ n ]  Nx    T2 ( x ) N  T1 ( x )       n 0  N 1    N     N 1   x Institute of Communications Engineering. NCTU 68 . ECE.

NCTU 69 .Unit 1: Minimum Variance Unbiased Estimator  Since N 1 N 1 2 2 x [ n ]  x  x [ n ]  Nx     n 0  Sau-Hsuan Wu 2 n 0 Eventually. T(x) is complete Actually. 2 ]T ? We have shown that for Gaussian PDF. ECE. we have x x       ˆg (T ( x ))  1  N 1 N 1  1  2 2 2     x [ n ]  Nx x [ n ]  x         N  1  n  0   N 1 n 0         Is this the MVU of =[A. this is also true for the vector exponential family of PDFs ˆ efficient ? Is  Institute of Communications Engineering.

the CRLB is Sau-Hsuan Wu are independent with Institute of Communications Engineering. NCTU 70 .Unit 1: Minimum Variance Unbiased Estimator  In fact  and  Therefore  While. ECE.