Classical Estimation Classical Estimation

Chapter 2
Estimator Performance Metrics Estimator Performance Metrics
Mean of the estimator (bias)
Variance of the estimator
Minimum Variance Unbiased
(MVU) Estimation
UNBIASED O h i ill i ld UNBIASED On the average estimator will yield
true value or
O O = O
|
all for )

( E
EXAMPLE: DC level in WGN (zero-mean)
|
value true
A - on values can take A
1 - 0,1,...N n WGN ] [ ] [
< <
= + = n W A n X
X[n] 1/N A

1 - N
0 n
=

=
( ) UNBIASED A all for X[n] E 1/N ) A

E(
1 - N
0 n
= =

=
A
Bias Bias
Example : same setup as previous example but p p p p
consider the estimator
2 / ] [ ) (
1
1
A A E n x A
N
= =
v

Bias of estimator :
2 / ] [ ) (
2
0
A A E n x
N
A
n
= =
=

Bias of estimator :
u u u =
.
) ( ) ( E b
Generally seek unbiased estimator (necessary but not
sufficient for good estimator)
MinimumVariance Estimation Minimum Variance Estimation
Mean Square Error (MSE) : Mean Square Error (MSE) :
] ) [( ) (
2
u u u =
. .
E MSE
Results in unrealizable estimator (for classical
but not Bayesian estimators more later )
] ) [( ) ( u u u E MSE
but not Bayesian estimators, more later.)
] )] ) ( ( )) ( [[( ) (
2
u u u u u + =
. . . .
E E E MSE
)] )

( ))(

( [( 2 ] ) ) ( [( ] )) ( [(
2 2
u u u u u u u u + + =
. . . .
E E E E E E E
u u )

( E is constant and comes out of the Expectation, hence last term is 0

Minimum Variance Unbiased Estimation
2
OF FUNCTION STRONG
) ( )

( b VAR
O =
O + O =
k ! t d ' hi h
on dependent

a produce ation will Differenti

OF FUNCTION STRONG
O O
O =
1 - N
X[n]
N
1
X X a A : Example
know! t don' which we
= =
v

0 n
? a" " best Find
[ ]
N
p
=

(allowing biased estimators)

2 2 2
2 2
) ( ) / ( ) (
/N a ) A Var( aA ) A E( = =
v
v v
o
2 2 2
) ( ) / a ( ) A MSE( A aA N + = o
Minimum Variance Unbiased Estimation
0 A)A - 2(aA /N 2
2
a
da
dMSE
o = + =
A! on depends a
2
2
2
opt
A
A
da
o
=
2
N
A
o
+
Generally, in classical estimation, we constrain the
bias to be zero (to get a realizable estimator) and
then minimi e its a iance ( hich is same as then minimize its variance (which is same as
minimizing MSE)

Minimum Variance Unbiased (MVU) Estimator

Existence of MVU Estimator

DOES MVU EXIST?
values all for variance
smallest have must

u
)

var(u
2

u
MVU =

u
Estimator)
MVU Uniformly ( . of
values all for variance
u
1
u
MVU =
3
u
u
exist. not will general, In
1

u
0 1) N( X[1]
,1) N( X[0]
: EXAMPLE
> ~
~
u u
u
No MVU !
)

var(u
2

u
3

u
are X[1] and X[0]
0 ,2) N(
0 ,1) N( X[1]
<
> ~
u u
u u
3
u
0
u
t Independen
[ ] [ ]
Example (Contd)
( ) ] 1 [ ] 0 [
2
1

1
+ = u X X
Consider
] 1 [
3
1
] 0 [
3
2

2
+ = u X X
Unbiased ? Yes !
1 4
]) 1 [ (
4
1
]) 0 [ (
4
1
)

(
1
+ = u X VAR X VAR VAR
1 ]) 0 [ (
]) 1 [ (
9
1
]) 0 [ (
9
4
)

(
2
=
+ = u
X VAR
X VAR X VAR VAR
0 2
0 1 ]) 1 [ (
1 ]) 0 [ (
< =
> =
u
u X VAR
X VAR
0 2 < u
Example (Contd)
)

(u VAR
36 / 27
36 / 24
2

u
36 / 24
36 / 20
36 / 18
1

u
36 / 18
u
BETTER

2
u BETTER

1
u
0? for

and 0 for

choose not Why

2 1
< > u u u u 0? for and 0 for choose not Why
2 1
< > u u u u
Example (Contd)
possible smallest are 0 for
)

VAR( and 0 for )

VAR( : that show also Can

2 1
u
u u u
<
>
ance lower vari with

other No
possible smallest are 0 for
u
u

<
Exercise : prove this result for the class of estimators
HOW DO WE FIND MVU?
(IF IT EXISTS)
APPROACHES:
1)Compute Cramer Rao lower Bound on Variance.
Find the Estimator that satisfies it (Chapter 3)

u
)

(u VAR
2
u
1

u
u
CRLB
When CRLB satisfied with equality for all
MVU Estimator
u

MVU Estimator

Note : it may happen that no estimator exists whose variance equals CRLB
yet MVU still exists (as in picture above)
HOW DO WE FIND MVU?
2)Apply Rao Blackwell Lehmann Scheffe Theorem 2)Apply Rao-Blackwell-Lehmann-Scheffe Theorem
Find sufficient statistic and make it Unbiased
MVU Estimator (Chapter 5)
3)Restrict Estimator to be Linear function of data
Best Linear Unbiased Estimator or MVU for All

Linear Estimators (Chapter 6)

MVU FOR VECTOR PARAMETER
parameters of vector 1 p
2
1

(
(
(

= u
u
u
Letting or )

E( if Unbiased
p p
2
(
(
(

p
u u
u

) (
) (
Y
X
E
Letting or ) E( if Unbiased
(

=
(

=
i i
Y E
X E
u u
( )

) ( Y
1 1
(
(
(

(
(
(

E
Y E
u u
( ) D UNBIASE
)

(
) (
2 2
=
(
(
(

=
(
(
(

=
p
p
E
E E u
u
u
u
u u

Minimum is 1,2,...p i )

VAR( means variance Minimum =

i
u