- Chapter2_solutions.doc
- Sample Period Selection With Variable Rolling
- Measuring Robustness on Generalized Gaussian Distribution
- Principal Component Analysis Based Approach for Fault Diagnosis in Pneumatic Valve Using Damadics Benchmark Simulator
- Point Estimation Statistics
- Book - Shevlyakov -Robustness in Data Analysis.pdf
- 07279172
- An Examination of Blujme and Vasicek Betas
- ISI_MStat_06
- Epidemiologia de Expetativas Macroeconomicas
- M. Giles - Multilevel Monte Carlo Path Simulation
- Stanford Stats 200
- Coles and Davison - Statistical Modelling of Extreme Values - June 2008
- Flenniken Thesis
- IHT_AGUNG
- Sampling Distribution
- Problem Set 4
- Syllabus for Elements of Statistics
- Dummy Variables
- metrics11_endogeneity
- Eco Assesment
- R Code for Discussion 6
- Duhh
- parameter estimation of diffusion models
- Power Transformer No-Load Loss Prediction With FEM Modeling and Building Factor Optimization
- 7abrilCZL Announcent 7Apr2016 Deep Drilling Delivers High Grades Up to 47 Percent Zn FINAL (1)
- Chapter II - Overview of Supervised Leraning
- Yes Please
- The Unwinding: An Inner History of the New America
- Sapiens: A Brief History of Humankind
- Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future
- Dispatches from Pluto: Lost and Found in the Mississippi Delta
- The Innovators: How a Group of Hackers, Geniuses, and Geeks Created the Digital Revolution
- Devil in the Grove: Thurgood Marshall, the Groveland Boys, and the Dawn of a New America
- John Adams
- The Emperor of All Maladies: A Biography of Cancer
- Grand Pursuit: The Story of Economic Genius
- This Changes Everything: Capitalism vs. The Climate
- A Heartbreaking Work Of Staggering Genius: A Memoir Based on a True Story
- The Prize: The Epic Quest for Oil, Money & Power
- The New Confessions of an Economic Hit Man
- Team of Rivals: The Political Genius of Abraham Lincoln
- The World Is Flat 3.0: A Brief History of the Twenty-first Century
- The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers
- Smart People Should Build Things: How to Restore Our Culture of Achievement, Build a Path for Entrepreneurs, and Create New Jobs in America
- Rise of ISIS: A Threat We Can't Ignore
- Angela's Ashes: A Memoir
- Steve Jobs
- How To Win Friends and Influence People
- Bad Feminist: Essays
- The Light Between Oceans: A Novel
- The Sympathizer: A Novel (Pulitzer Prize for Fiction)
- Extremely Loud and Incredibly Close: A Novel
- Leaving Berlin: A Novel
- The Silver Linings Playbook: A Novel
- You Too Can Have a Body Like Mine: A Novel
- The Incarnations: A Novel
- The Love Affairs of Nathaniel P.: A Novel
- Life of Pi
- The Master
- Bel Canto
- A Man Called Ove: A Novel
- Brooklyn: A Novel
- The Flamethrowers: A Novel
- We Are Not Ourselves: A Novel
- The First Bad Man: A Novel
- The Rosie Project: A Novel
- The Blazing World: A Novel
- The Wallcreeper
- The Art of Racing in the Rain: A Novel
- Wolf Hall: A Novel
- The Kitchen House: A Novel
- Beautiful Ruins: A Novel
- Interpreter of Maladies
- The Bonfire of the Vanities: A Novel
- Lovers at the Chameleon Club, Paris 1932: A Novel
- The Perks of Being a Wallflower
- The Cider House Rules
- A Prayer for Owen Meany: A Novel
- Everything Is Illuminated

Unbiased Estimator

Institute of Communications Engineering, ECE, NCTU

1

**Unit 1: Minimum Variance Unbiased Estimator
**

**What are estimations all about?
**

Sau-Hsuan Wu

**We want to know (or estimate) an unknown quantity
**

This quantity, , may not be obtained directly, or it is a

notion (quantity) derived from a set of observations

In any event, we need to retrieve the value of from a set of

observations, x = x[0],…, x[N-1], which are related to

We hope to determine according to

How do we start?

**Suppose we have some knowledge about the collected data
**

Examples:

x[n] = A1 + w[n],

x[n] = A2 + B2 n + w[n],

n=0,…,N-1

n=0,…,N-1

Institute of Communications Engineering, ECE, NCTU

2

**Unit 1: Minimum Variance Unbiased Estimator
**

Sau-Hsuan Wu

Â1=

x[n]/N

and ?

**An estimator may be thought of as a rule that assigns a
**

value to for each realization of x

The estimate of is the value of for a given realization

of x

How do we assess the performance of estimation?

**What are we concerned the most about the estimate?
**

**How close will Â be to A?
**

Are there better estimators than the sample mean for instance?

Ă= x[0] ?

**What are the differences between Âa
**

ndĂ?

Institute of Communications Engineering, ECE, NCTU

3

the PDFs are parameterized by . As an example: We have a class of PDFs where each one is different due to a different value of . The parameter is assumed to be deterministic but unknown Given the PDF.e. ECE.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Can be have some performance measure? We model the data by its probability density function (PDF). i. assuming that the data are inherently random. NCTU 4 . we calculate some statistics of the estimates E (Â ) = E (Ă) ? Var (Â) = Var (Ă) ? Institute of Communications Engineering.

but one that is also mathematically tractable Sometimes. then the data are described by the joint PDF Any estimator that yields estimates according to the prior knowledge of is termed a Bayesian estimator Institute of Communications Engineering. To incorporate this prior knowledge.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Note: In an actual problem. ECE. we are not given a PDF but must choose one that is not only consistent with the problem constraints and any prior knowledge. we might want to constraint the estimator to produce values in a certain range. U] interval for instance Assign a PDF to . NCTU 5 . we can assume that is no longer deterministic but a random variable having a uniform distribution over the [-U.

Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu We conclude for the time being that we are hoping to have an estimator that gives An unbiased mean of the estimate: A minimum variance for the estimate: Is an unbiased estimator always the optimal estimator? Consider a widely used optimality criterion: the minimum mean squared error (MSE) criterion Institute of Communications Engineering. ECE. NCTU 6 .

r. ECE. we have mse (Ă) = a2 2 / N + (a-1)2A2 Taking the derivative w.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu As an example. to a and setting it to zero leads to aopt = A2 / (A2 + 2/ N ) The optimal value of a depends upon the unknown parameter A The estimator is not realizable How do we resolve this dilemma? Since. NCTU 7 . mse (Ă) = var(Ă) + b2. we set b=0 and search for the estimator that minimizes var(Ă) minimum variance unbiased (MVU) estimator Institute of Communications Engineering. as an alternative.t. consider the modified estimator We attempt to find the ‘ a’which yields the minimum MSE Since E(Ă)=aA and var(Ă) = a2 2 / N.

Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu However.1) x[1] ~ N(. ECE. NCTU 8 . does a MVU estimator always exist for all ? Example: x[0] ~ N(.1) if 0 ~ N(.2) if 0 Both of the two estimators are unbiased Therefore None of the estimator has a variance uniformly less than or equal to 18/36 Institute of Communications Engineering.

ECE.Cramer-Rao Lower Bound Institute of Communications Engineering. NCTU 9 .

2).Unit 1: Minimum Variance Unbiased Estimator How to measure the accuracy of estimation? Consider a single sample of observation x[0] = + w[0] with w[0]~N(0. Sau-Hsuan Wu The unbiased estimator is Â and the variance is The accuracy of estimation improves as 2 decreases We can see this from the likelihood function (LK) of Institute of Communications Engineering. ECE. NCTU 10 .

ECE. the sharpness of the LK is measured by the curvature of the LLK.e. in a more rigorous definition. NCTU 11 . which is inversely proportional to i2 in this case Actually. In general. the variance i2 in this case is the negative inverse of the second derivative of the logarithm of the likelihood function (LLK) Therefore. the curvature depends on x[0] as well. i.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu The accuracy of estimation increases with the “ sharpness” of the LK. thus a more appropriate measure of curvature is Institute of Communications Engineering.

) Institute of Communications Engineering.Unit 1: Minimum Variance Unbiased Estimator Now. p(x . namely The variance of the estimation is given by We want to find a lower bound for Sau-Hsuan Wu How do we start? The hint from the previous example is that the variance is inversely proportional to the average curvature of p(x . NCTU 12 . ) which is parameterized by Consider all unbiased estimators . ECE. we put our problem in a more rigorous statement Define a scalar parameter = g() Define a PDF of the observation.

NCTU 13 . the smaller the lower bound for the variance of estimation By the Cauchy-Schwarz inequality. we have Institute of Communications Engineering.Unit 1: Minimum Variance Unbiased Estimator Therefore. ECE. we may mathematically associate the variance with the average curvature like this Sau-Hsuan Wu The larger the curvature.

ECE.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Then. NCTU 14 . can we relate the square of the first-order derivative to the second-order derivative? i.e. the connection between and Observe that if the integral and partial are interchangeable Institute of Communications Engineering.

we have supposed that Substituting the results to the Cauchy-Schwarz inequality Institute of Communications Engineering.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Then. ECE. NCTU 15 .

NCTU 16 .Unit 1: Minimum Variance Unbiased Estimator Therefore. ECE. The equality holds if and only if where c might be a function of but not of x If =g()= . then Sau-Hsuan Wu Institute of Communications Engineering.

t. on the both sides leads to Taking the expectation on the both sides.r.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Taking the derivative w. for unbiased estimators we have Thus. ECE. NCTU 17 . Institute of Communications Engineering.

**Unit 1: Minimum Variance Unbiased Estimator
**

Sau-Hsuan Wu

Clearly, the Cramer-Rao bound (CRB)

is valid if the regularity condition holds

**Does the regularity condition always hold?
**

Refer to Leibniz’

s rule

Institute of Communications Engineering, ECE, NCTU

18

**Unit 1: Minimum Variance Unbiased Estimator
**

Sau-Hsuan Wu

**The gradient, with respect to , of the log likelihood
**

function (LLK) is called the score

**Under the regularity condition, the expected value of the
**

score is zero, namely

The Fisher information is the variance of the score

Institute of Communications Engineering, ECE, NCTU

19

**Unit 1: Minimum Variance Unbiased Estimator
**

By definition,

If the regularity condition holds,

Sau-Hsuan Wu

then the information is additive for independent x and y

Therefore, it is referred to as the information of

Institute of Communications Engineering, ECE, NCTU

20

NCTU 21 . d 0 The goal is to prove Cα ˆ T 1 a C I (θ) Namely. αˆ a 0 Institute of Communications Engineering. θ) ln p ( x. θ) I (θ) p( x. θ)dx T θ θ 1 I (θ) p. θ) p ( x. θ)dx θ θ ln p ( x. s . ECE. θ)dx Cαˆ (α T ln p ( x.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Cramer Rao’ s lower bound for vector parameter Let vector = g( ) and define ˆα)( α ˆα)T p( x. for arbitrary a. θ) ln p ( x.

θ)dx a T Cαˆ ln p ( x. θ) T ˆ a α b dx a α b dx θ θ T aT ˆ α p( x. θ)dx b T θ T 2 2 g ( θ ) T a b θT Institute of Communications Engineering. θ)dx θ 2 T p( x. θ) ln p ( x. ECE. θ) p( x. θ)dx b θ θ 2 I (θ) T T ln p( x.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Cauchy-Schwarz inequality T ˆ ˆ aT ( α α )( α α ) p( x. NCTU 22 . θ) T b p( x. θ) ˆα) a (α b p( x.

since b is arbitrary. by Cauchy-Schwarz inequality.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Therefore. we can define T g (θ) -1 b I (θ) a θ As a result 2 2 T g (θ) T g (θ) -1 g (θ) T b = a I (θ) a a T T θ θ θ T g ( θ ) g (θ) T T -1 -1 a Cαˆaa I (θI ) (θI ) (θ) a T θ θ Institute of Communications Engineering. we have 2 g (θ) T T T a b a C ab I (θb ) ˆ α T θ Besides. ECE. NCTU 23 .

so is I-1().d 0 T θ θ Institute of Communications Engineering.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Since I() is positive definite. thus g (θ) -1 g T (θ) CˆI (θ) p. ECE. NCTU 24 . We have 2 T T g (θ) -1 g (θ) T g ( θ ) g (θ) T -1 I (θ) a a Cαˆaa I (θ) a a T T θ θ θ θ T T g (θ) -1 g (θ) a Cˆa a I (θ) a T θ θ T T positive semidefinite T g ( θ ) g (θ) T -1 a CˆI (θ) a 0 T θ θ Since a is arbitrary. s.

θ) 1 ˆα) I ( θ ) (α T θ θ c(θ) Thus. θ ) g (θ) T -1 ˆ a ( αα) c(θ) b c(θ) I (θ) a T T θ θ θ For arbitrary a g (θ) -1 ln p( x. for = g() = ln p( x. NCTU 25 . ECE. θ) 1 ˆθ) I (θ)(θ θ c(θ) Institute of Communications Engineering. θ ) ln p ( x .Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu When g()= . we have Cθˆ-I -1 (θ) 0 Since the conditions for equality are T ln p ( x .

NCTU 26 .r. θ) 1 I (θ) E I (θ) c(θ)=1 T θθ c(θ) Eventually.t. we have ˆθ) Ii (θ) ˆ 1 Ii (θ) ˆ I i (θ)(θ (θθ) (θθ) I i (θ) T θ c(θ) 1 c(θ) N c(θ) c(θ) Ii (θ) ˆ 1 I i (θ) ˆ 1 E ( θ θ ) E ( θ θ ) I ( θ )= I i (θ) x x i 1 c(θ) N c(θ) c(θ) c(θ) Putting the row vectors into a matrix returns 2 ln p( x. Denote the i-th row of I()/c( ) by Ii()/c(). θ) ˆθ) I (θ)(θ θ Institute of Communications Engineering. ECE. we arrive at ln p( x.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Now take the derivative on the both sides w.

achieves the CRLB too Cˆ=ACθˆΑ T I (θ) θ θ Institute of Communications Engineering. i. ECE.e.Unit 1: Minimum Variance Unbiased Estimator For a positive semidefinite matrix A [A]ii 0 Sau-Hsuan Wu Since xTAx 0 x eiTAei = [A]ii 0 Therefore. C I (θ) ) θ ˆ θ Now. for a linear transformation = g() = A+ b ˆb E (α ˆAθ ˆ α ) α and g (θ) -1 g T (θ) . Cθˆ-I -1 (θ) 0 -1 ˆ) C I (θ) var( i θˆ ii ii 1 ˆ Suppose E (θ and achieves the CRLB. NCTU 27 .

= [A B]T and H ln p( x. x[N-1]]T .Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu We revisit the line fitting problem 1 0 x[n] = A + Bn + w[n].N-1 1 1 Let x = [x[0].…. n=0. ECE.…. NCTU 28 . θ) T T 2 1 N 1 H x H Hθ / θ I (θ) H T H / 2 1 ln p( x. θ) T ˆθ I (θ) H H H T x θI (θ) θ θ An estimator which is unbiased and attains the CRLB bound is said to be efficient in that it efficiently uses the data Institute of Communications Engineering.

now 2(2 N 1)2 2 122 var( Aˆ ) Sau-Hsuan Wu N ( N 1) if N 2 and var( Bˆ ) N N ( N 2 1) Thus. NCTU 29 . ECE. the CRLB always increases as we estimate more parameters Institute of Communications Engineering.Unit 1: Minimum Variance Unbiased Estimator Efficiency is maintained over a linear transformation Let us take a closer look at I() = HTH/2 N 2 I (θ) N ( N 1) 22 N ( N 1) 2(2 N 1)2 N ( N 1) 22 . I 1 (θ) N ( N 1)(2 N 1) 62 2 6 N ( N 1) 62 N ( N 1) 2 12 N ( N 2 1) Compared with x[n] = A + w[n] in which var(Â) = 2/N.

I) Based on the previous result. we can use a whitening approach Let C-1 = DTD DE{w wH} DT = I x’= D x = D(H+ w ) = H’+ w’with w’~ N(0. ECE. NCTU 1 30 .C) To derive the MVU.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Extension to linear model x = H+ w with w ~ N(0. we have 1 1 T T T T ˆ θ H ' H ' H ' x ' = H D DH H T DT Dx = H C H H T C 1 x T and 1 1 Cθˆ H ' H ' = H D DH = H C H T 1 T T 1 T 1 Institute of Communications Engineering.

why? The variance depends on the gradient of g() Institute of Communications Engineering. ECE.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Ex. NCTU 31 . n=0. g (θ) 2 A A2 2 θ T Recall that N 2 I (θ) 0 4 0 g (θ) 1 g T (θ) 4 A2 2 A4 422 I (θ) 2 4 T N θ θ N N N 24 The variance of estimation increases with the SNR .N-1 Let = [A σ2]T and g() = 12/2 = A2 /σ2 = Then. Estimation of the signal-to-noise (SNR) ratio Given x[n] = A + w[n].….

Sufficient Statistics Institute of Communications Engineering. NCTU 32 . ECE.

x[N-1]} S3 = {x[n]} x[n] is also a statistic Then. x[1]. what is the minimum set of sufficient statistics? Given x[n] = T0 .Unit 1: Minimum Variance Unbiased Estimator Is there a set of T(x) of x that is sufficient for estimation? What do we mean by the sufficiency of a set of T(x)? We want to have a set of statistics T(x) such that given T(x). NCTU 33 .x(N-1)] is independent of A Suppose Â1= x[n]/N. do we still need the individual data? Institute of Communications Engineering.…. x[N-1]} each element is a statistic S2 = {x[0]+x[1]. x[2]. ECE.…. then are the followings sufficient? Sau-Hsuan Wu S1 = {x[0].…. any x(n) of x = [x(0).

a value of A near A0 is more likely even given T0 For (b).Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu We say the conditional PDF : p( x | x[n] = T0 .g. p( x | x[n] = T0 . ECE. For (a). A) should not depend on A if statistic T0 is sufficient E. A) is a constant Institute of Communications Engineering. however. NCTU 34 .

we have Institute of Communications Engineering. NCTU 35 . ECE. Clearly. A) to show that x[n] = T0 is sufficient By Baye’ s rule Since T(x) is a direct function of x.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Now. we need to determine p( x | x[n] = T0 .

we have Institute of Communications Engineering. NCTU 36 . ECE.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Thus.

NCTU 37 .Unit 1: Minimum Variance Unbiased Estimator Since T(x) = x[n] ~ N(NA. ECE. N2) Thus Sau-Hsuan Wu which does not depend on A Institute of Communications Engineering.

)=g(T(x).) into p(x. to identify potential sufficient statistics is difficult An efficient procedure of finding the sufficient statistics is to employ the Neyman-Fisher factorization theorem Observe that If we can factorize p(x.) h(x) where g is a function that depends on x only through T(x) h is a function that depends only on x T(x) is a sufficient statistic for The converse is also true If T(x) is a sufficient statistic p(x.)=g( T(x). ECE.) h(x) Institute of Communications Engineering.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu In general. NCTU 38 .

ECE.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Recall p(x. we want to estimate 2 of y[n]=A+x[n] Suppose A is given. NCTU 39 . then define x[n] = y[n]-A Clearly.A) Now. T(x) = x2[n] is a sufficient statistic for 2 Institute of Communications Engineering.

p(x. NCTU 40 . )h(x) Since Institute of Communications Engineering.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Proof of the Neyman-Fisher Factorization Theorem () By assumption.) = g(T(x). ECE.

ECE.Unit 1: Minimum Variance Unbiased Estimator We have Then which does not depend on Hence T(x) is a sufficient statistic Sau-Hsuan Wu Institute of Communications Engineering. NCTU 41 .

T0) =1 Therefore.T0) with w(x) (T(x) . ECE.) = P(x| T(x)=T0) ( not a function of ) We can define p(x| T(x)=T0 . we can let Institute of Communications Engineering. NCTU 42 .Unit 1: Minimum Variance Unbiased Estimator () Recall And Suppose T(x) is a sufficient statistic Sau-Hsuan Wu Then p(x| T(x)=T0 .) = w(x) (T(x) .

we have and This holds for arbitrary T0. ECE. resulting in where Institute of Communications Engineering. NCTU 43 .Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu As a result.

NCTU 44 .Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Ex.N-1 Suppose A and f0 are given Expand the exponent Institute of Communications Engineering. we want to estimate the phase of a sinusoid x[n]=A cos(2 f0 n + ) + w[n].1. n=0. ECE.….

however g(T1(x). NCTU 45 . no single sufficient statistic exists.T2 (x).Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu In this case. ECE. ) h(x) Institute of Communications Engineering.

T2(x).Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu The r statistics T1(x). then is unbiased and A valid estimator for (not dependent on ) Of lesser or equal variance than that of for all If T(x) is complete. Tr(x) are jointly sufficient if p(x | T1(x). Tr(x) . Tr(x)} are sufficient statistics for Now. T2(x).…. Tr(x) . ) does not depend on If p(x . then is the MVU estimator Institute of Communications Engineering.….…. T2(x).…. ) h(x) {T1(x). we know how to obtain the sufficient statistics How do we apply them to help obtain the MVU estimator? The Rao-Blackwell-Lehmann-Scheffe Theorem If is an unbiased estimator of and T(x) is a sufficient statistic for . NCTU 46 . ECE. ) = g(T1(x). T2(x).

) is not a function of after the integration w.t. ECE.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Proof 1> validity by definition p(x|T(x) . x. NCTU 47 .r. the result is not a function of but T 2> unbiasedness By assumption Institute of Communications Engineering.

NCTU 48 . ECE.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu 3> show Recall that If 0 0 is solely a function of T(x) Institute of Communications Engineering.

the MVU can be found by Taking any unbiased and carrying out Alternatively. a statistic is complete if there is only one function. must be the MVU In summary.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Finally. say g. for all and any unbiased estimator Then. since there is only one function of T(x) that leads to an unbiased estimator find the unique g(T(x)) that makes unbiased Institute of Communications Engineering. NCTU 49 . of the statistic that is unbiased is solely a function of T(x) If T(x) is complete is unique and unbiased Since T(x) is sufficient. is as good as with another T1(x) Besides. ECE.

Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Completeness of a sufficient statistic We know that for x[n] = A + w[n]. A Since T ~ N(NA. ECE. NCTU 50 . N2) where v(T) = g(T(x)) -h(T(x)) Let = T /N and v’ ( ) = v(N ) W( ) Institute of Communications Engineering. 2) T(x) = x[n] is sufficient and g(T(x)) = T(x)/N is unbiased Suppose. a second function h for which E{h(T(x))} = A E{g(T(x)) -h(T(x))} = A –A =0. with w[n]~ N(0.

f Since W(f) is still Gaussian V’ (f) = 0 v’ ( ) = 0. thus T(x) is complete Institute of Communications Engineering. ECE. g(T(x)) = h(T(x)). NCTU 51 .Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Which is a convolution of v’ ( ) and w( ) and =0 A v’ ( )=0 Recall a signal v(t) is zero iff F(v(t)) = 0 F(v’ ( )*w( )) = V’ (f)W(f) = 0.

while w[0] ~ U[-1/2. is T(x) completely sufficient? Let v(T) = g(T(x)) -h(T(x)) However. so that but Institute of Communications Engineering. x = x[0] = T.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Incomplete sufficient statistic Now consider x[0] = A + w[0]. NCTU 52 . ECE. 1/2] T(x) = x[0] is sufficient. But.

NCTU 53 . ECE.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu So that A nonzero v(T) = sin (2T) will satisfy this condition Institute of Communications Engineering.

T Institute of Communications Engineering. we say a sufficient statistic is complete if is satisfied only by the zero function or by v(T) = 0. v(T) = g(T) –h(T) = sin (2T) Let g(T) = T = x[0] since E{x[0]} =A h(T) = T . NCTU 54 .sin (2T) Â = x[0] .Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Hence. not sure if Â = x[0] is an MVU estimator To summarize. ECE.sin (2x[0]) is also an unbiased estimator of A. using statistic T = x[0] x[0] is not complete.

3> Find a function g(T(x)) that yields an unbiased estimation of which is the MVU of Alternatively. ECE. NCTU 55 .Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu As a summary. the RBLS method can be used to find the MVU even when an efficient estimator does not exist The procedure we learn by now to find an MVU estimator 1> Find a sufficient statistic T(x) for by the Neyman-Fisher factorization theorem 2> Determine if T(x) is complete. where is an unbiased estimator Institute of Communications Engineering. if so.

) / } = 0 Is a sample mean the MVU estimator for this case? Its variance is Institute of Communications Engineering.….1.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Ex: Mean of uniform noise x[n] = w[n] . NCTU 56 . w[n] ~ U[0. ] Want to find the MVU estimator for the mean = /2 The initial approach of using the CRLB to find an efficient estimator cannot even be tried for it does not satisfy the regularity condition : E{ln P(x. ECE. n=0.N-1.

we follow the procedure we learned so far Define Then where =2 The PDF is or Institute of Communications Engineering. ECE.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Now. NCTU 57 .

ECE.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Alternatively so that T(x) = max(x) We need to determine a function g to make T(x) unbiased To do so requires us to determine E{T(x)} Institute of Communications Engineering. NCTU 58 .

ECE.Unit 1: Minimum Variance Unbiased Estimator The CDF of T(x) The PDF follows as Sau-Hsuan Wu Institute of Communications Engineering. NCTU 59 .

Unit 1: Minimum Variance Unbiased Estimator But d Pr{x[n] < }/d is the PDF of x[n] or Integrating. we obtain Sau-Hsuan Wu Institute of Communications Engineering. ECE. NCTU 60 .

NCTU 61 . ECE.Unit 1: Minimum Variance Unbiased Estimator Which finally yields We now have Sau-Hsuan Wu Institute of Communications Engineering.

we multiply T(x) by (N+1)/N which is the MVU estimator whose corresponding variance is and < 2/(12N) of the sample mean Institute of Communications Engineering. ECE.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu To make it unbiased. NCTU 62 .

Tr(x)]T is said to be sufficient for the estimation of If P(x| T(x) . ECE. ) h(x) T(x) is of minimum dimension Institute of Communications Engineering. NCTU 63 . ) =g(T(x). a vector T(x) = [T1(x).Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Extension to a vector parameter We want to seek an unbiased vector estimator such that each element has the minimum variance Similarly. ) = P(x| T(x)) If P(x . T2(x)….

2 ]T is the unknown vector parameter The PDF is Expanding the exponent.…. =[A. f0 . we obtain Cannot reduce the PDF due to Institute of Communications Engineering. n=0. NCTU 64 .Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Back to a similar example x[n]=A cos(2 f0 n) + w[n].N-1 Now.1. ECE.

NCTU 65 . 2 ]T We are able to factorize the PDF which gives N 1 x[n ]cos(2f 0n ) T1 ( x ) n 0 T ( x ) N 1 T2 ( x ) 2 x [n ] n 0 Institute of Communications Engineering.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu If f0 is known. then =[A. ECE.

we have N 1 x[n ] T1 ( x ) n 0 T ( x ) N 1 T2 ( x ) 2 x [n ] n 0 Taking the expected values produces NA NA E{T ( x )} 2 2 2 NE { x [ n ]} N ( A ) Institute of Communications Engineering. =[A. ECE. 2 ]T Set f0 = 0 for x[n]=A cos(2 f0 n) + w[n].N-1 Again.Unit 1: Minimum Variance Unbiased Estimator Sau-Hsuan Wu Now apply the above result to our previous discussion x[n]=A + w[n]. n=0.1. NCTU 66 .….

ECE.Unit 1: Minimum Variance Unbiased Estimator So T2(x) only helps estimate the second moment not the variance If we transform T (x) into 1 T ( x ) 1 N g (T ( x )) 2 1 1 1 N N T2 ( x ) N T1 ( x ) Sau-Hsuan Wu N 1 2 2 x [ n ] x n 0 x Then. E{g(T(x))} gives E x A E{g (T ( x ))} 1 N 1 2 2 2 2 2 A E x E x [n ] x N n 0 Institute of Communications Engineering. NCTU 67 .

2/N) Sau-Hsuan Wu 2 E x A N 2 2 Since Substituting this back into E{g(T(x))} yields A A E{g (T ( x ))} 2 2 2 2 2 2 2 A E x A A N Therefore.Unit 1: Minimum Variance Unbiased Estimator x ~ N (A. multiply the second element by N/(N-1) yields an unbiased estimator of 2 1 T ( x) 1 N N 1 2 g (T ( x )) 1 2 2 1 1 x [ n ] Nx T2 ( x ) N T1 ( x ) n 0 N 1 N N 1 x Institute of Communications Engineering. NCTU 68 . ECE.

NCTU 69 .Unit 1: Minimum Variance Unbiased Estimator Since N 1 N 1 2 2 x [ n ] x x [ n ] Nx n 0 Sau-Hsuan Wu 2 n 0 Eventually. T(x) is complete Actually. 2 ]T ? We have shown that for Gaussian PDF. ECE. we have x x ˆg (T ( x )) 1 N 1 N 1 1 2 2 2 x [ n ] Nx x [ n ] x N 1 n 0 N 1 n 0 Is this the MVU of =[A. this is also true for the vector exponential family of PDFs ˆ efficient ? Is Institute of Communications Engineering.

the CRLB is Sau-Hsuan Wu are independent with Institute of Communications Engineering. NCTU 70 .Unit 1: Minimum Variance Unbiased Estimator In fact Â and Therefore While. ECE.

- Chapter2_solutions.docUploaded byMiguel Gonzalez
- Sample Period Selection With Variable RollingUploaded byelz0rr0
- Measuring Robustness on Generalized Gaussian DistributionUploaded byAnonymous 7VPPkWS8O
- Principal Component Analysis Based Approach for Fault Diagnosis in Pneumatic Valve Using Damadics Benchmark SimulatorUploaded byInternational Journal of Research in Engineering and Technology
- Point Estimation StatisticsUploaded byanon-257547
- Book - Shevlyakov -Robustness in Data Analysis.pdfUploaded byDiego Queiroz
- 07279172Uploaded byrallabhandiSK
- An Examination of Blujme and Vasicek BetasUploaded byblindslayer
- ISI_MStat_06Uploaded byapi-26401608
- Epidemiologia de Expetativas MacroeconomicasUploaded byRamon Alejandro Quinteros
- M. Giles - Multilevel Monte Carlo Path SimulationUploaded byCarlos Sánchez
- Stanford Stats 200Uploaded byJung Yoon Song
- Coles and Davison - Statistical Modelling of Extreme Values - June 2008Uploaded byAnonymous BZ0l8Qylp
- Flenniken ThesisUploaded byPhạm Ngọc Hòa
- IHT_AGUNGUploaded byErenHunter
- Sampling DistributionUploaded byraazoo19
- Problem Set 4Uploaded bySana Pakainin Mo Ako
- Syllabus for Elements of StatisticsUploaded byAhmed Kadem Arab
- Dummy VariablesUploaded bysaeedawais47
- metrics11_endogeneityUploaded byJohn Lukas
- Eco AssesmentUploaded byamiraazhar
- R Code for Discussion 6Uploaded bymuggsy
- DuhhUploaded byMymie Bruzon Maandig
- parameter estimation of diffusion modelsUploaded bysupermanvix
- Power Transformer No-Load Loss Prediction With FEM Modeling and Building Factor OptimizationUploaded bybtd2011
- 7abrilCZL Announcent 7Apr2016 Deep Drilling Delivers High Grades Up to 47 Percent Zn FINAL (1)Uploaded byGustavo Dominguez
- Chapter II - Overview of Supervised LeraningUploaded byChirag