Estimation: chapter 2+3

Minimum variance unbiased estimation + the CRLB

Natasha Devroye
devroye@ece.uic.edu
http://www.ece.uic.edu/~devroye
Spring 2011

Estimation: a first example

Estimate the DC level, A, of a signal given noisy measurements x[0], x[1], ... x
[N-1] where

x[n] are samples of this!

Find a few estimators

Compare their performance

mean?
variance?
pdf?

Estimation: a first example

Estimators of the DC level, A

Estimation: definitions

Estimation: definitions

How would you pick a ``good estimator?

Vector versions....

Minimum variance unbiased estimation

Why?

So?

Minimum variance unbiased estimation

Give a counter-example! (b1pg.20)

The Cramer-Rao Lower Bound

the CRLB give a lower bound on the variance of ANY UNBIASED estimator
does NOT guarantee bound can be obtained
IF find an estimator whose variance = CRLB then its MVUE
otherwise can use Ch.5 tools (Rao-Blackwell-Lehmann-Scheffe Theorem and
Neyman-Fisher Factorization Theorem) to construct a better estimator from
any unbiased one - possibly the MVUE if conditions are met

The Cramer-Rao Lower Bound (CRLB)

Use?
Intuition?

The Cramer-Rao Lower Bound (CRLB)

CRLB examples

CRLB proof

CRLB T or F

What is I()?

Why information?
non-negative
additive for independent observations

Vector form of the CRLB

Vector form of the CRLB

Vector form of the CRLB examples

What can we conclude?

CRLB for transformations

Vector CRLB for transformations

Example of vector CRLB with transformation

CRLB for General Gaussian Case

When observations are Gaussian and one knows the dependence of the
mean and covariance matrix on the unknown parameters, we know the
closed form of the CRLB (or Fisher information matrix):

CRLB for Gaussians examples

Example: range estimation

678$9$:)('.$6;<&=)<&,($/>,?*.=
678$9$:)('.$6;<&=)<&,($/>,?*.=

-.&/0%12345607833
"+#)333333/9/:7.939;7.3#&<,($"=
-.&/0%12345607833
"+#)333333/9/:7.939;7.3#&<,($"=
>7?71;73>7@67?219/8333 "+#'( !!)
>7?71;73>7@67?219/8333 "+#'( !!)
A7&05.73-1%73B76&C8 !!
A7&05.73-1%73B76&C8 !!
!"#$%&'()*$+,-.*
Continuous time signal model!"#$%&'()*$+,-.*
& + # ) # " + # % ! ) " %+ # )
, $ # $ $ # $ "!
!
$
!#!
"
& +# ) #" +"#+*!# %) ! ! ) " %+# )
$
!!#!
"

"

" + # *! ! )

"+#)
"+#)

D&/E61%127E
FG1273H&5001&/
D&/E61%127E
FG1273H&5001&/
/%0$,1$!2"3

D4I &+#)
J3K%L
D4I &+#)
J3K%L

$"

#$45
/%0$,1$!2"3
#$45
(*

(*

"+#''( $! " )

"+#''( !!)

)'

!(%&'

, $ # $ $ # $" " ! !(%&'

)'

!"#$

!"#$

Taken from http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522.htm

Example: range estimation

/0123&45678067'1&-9:&,7210;&<'=3;
Discrete time
signal model
/0123&45678067'1&-9:&,7210;&<'=3;
#$%&'(&$)%*
#$%&'(&$)%*
"! " !"#
"! " !"#

+,-&'(&$)%*
+,-&'(&$)%*
"#.!
"#.!

*!
*!

%&'()*+,-*./+! 0+#"1!"2*3
%&'()*+,-*./+!
0+#"1!"2*3
#4$5+0+#6$
!7
#4$5+0+#6$!7

&

)"

&
#"1! #"! !"1!
#"1! #"! !"1!

)"

'4$ 5 $ & 6 $! # & ( 7 % #4$ 5

'4$ 5 $ & 6 $! # & ( 7 % #4$ 5

89+:;<=*+
89+:;<=*+
>&?22<&@+AB<2*
>&?22<&@+AB<2*
C&.+"! " !"#
C&.+"! " !"#

$ $ ED#D ! D % # #
$ $ ED#D ! D % # #

&4$F&(5G ;&2++ @B@HI*.B+2&'()*2+2=&.=<@J+&=+$(

&4$F&(5G ;&2++ @B@HI*.B+2&'()*2+2=&.=<@J+&=+$(

E + $ + $( # #
E + $ + $( # #
$( + $ + $ ( % + # #
$( + $ + $ ( % + # #
$( % + + $ + % # #
$( % + + $ + % # #

* #4$ 5
*'#4$ 5
'
'4$ 5 $'') & 6 $! # & ( 7 % #4$ 5
'4$ 5 $ )'& 6 $! # & ( 7 % #4$ 5
'(' #4$ 5
'( #4$ 5

!"#$

!"#$

Taken from http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522.htm

Example: range estimation

!"#$%&'()*+")*,#&-!./
%&'()**+,(-.)/0)10(2345(16-7+.(8&1(-9:/)+(;(<=%>
H+7:(9/I()/0(J66*(
/&/KL61&(.61M-

G)1B/F$ A -

.@
% "#

+ ,&E"D/ C (
! )) ,/ $ $ &&
'
" # ?*

.@

"$ % ! "#

" # "$

.@
"$ % ! "# +

" # "$

) ,& B# A
) ,#
*

NO*+&9.(2)+P7+7-QQQ

(
&
&
# # "$ "/ $ '

+ ,& B "$ " / $ A (

))
&&
,/ $
*
'

.@
! "# +

,& B# A
! )) ,#
" #? *

(
&
&
# # "$ '

R-6()**1&O9M).9&/>(/$'('$ "$
ST6/(0&(PT)/:6(&8(G)19)U+6-QQ
!"#$
Taken from http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522.htm

Example: range estimation

/'%0"&1+2$*'2$3%&4/5(&673%289

-../012.+03412.3+567826.2.0+449 +33,:;<2./02=>267?18,+49
*+,(-)# ' +

,&
&

# "! ' (! (& ' $

%
" %&
* !% & (& #

'# " &

"! ' (! ( & ' $

!%

&

%
" %&
& (& #

"! ' (! ( & ' $

$!
'# " &

!%

&

%
" %&
& (& #
$!
"!

*+,(-)# ' +
$!
'# " &

"
&
&
!% .&2( / ) ( ( ' %(
!

$!
0

$!
'# " &

$ ! ) ! ! & (& '%&

%

!"#$%"&'&()&*"'+,-".
0

@A2AB1:,10
C2D+,.1*+4

!10 .&2( / ) ( ( ' %(

0
&
!10 ) ( ( ' %&
&

&

D+,.1*+4

*+,! )

!10 .&2( / ) ( ( ' %(

0
&
!10 ) ( ( ' %&
&

&

*+,! 6.2EFGH2IJK22(LM'
!"#$

-2?>312:N2EHOFK

Taken from http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522.htm

Example: range estimation

&"#$%'()*+,"*+-#'.&/0'12-#*34
&"#$%'()*+,"*+-#'.&/0'12-#*34
%&'()*+,-&-*'.-/&*0-*/11'2-*/+*+,-*3456*7(*+,-*.-8/9:

%&'()*+,-&-*'.-/&*0-*/11'2-*/+*+,-*3456*7(*+,-*.-8/9:
#
2/1<#>$ ; "
<&-= ? ;
#?
&'(
!
%
!"#
2/1<#>$ ; "
<&-= ? ;
?
&'( ! %!"#

@7*)-+*+,-*3456*7(*+,-*!"#$%A*B&-*C+1/(&DE*7D*F/1G&H*1-&B8+:*
@7*)-+*+,-*3456*7(*+,-*!"#$%A*B&-*C+1/(&DE*7D*F/1G&H*1-&B8+:*
?

) *( &
$ *(+%
*(+%(> + ''
0'+,* (
?
#>$
*#) $*$( &
*(+%(> (+ '' % $$ *(+%#>$ 0'+,*
( *# $ %

J*)#$,"*?
( J*)#$,"*?

3456*'&*'(2-1&-89*F17F71+'7(/8*+7:
3456*'&*'(2-1&-89*F17F71+'7(/8*+7:
3456*'&*'(2-1&-89*F17F71+'7(/8*+7:
KK LM4*N-/&B13456*'&*'(2-1&-89*F17F71+'7(/8*+7:
LM4*N-/&B1LM4*N-/&B1KK KK4NL*6O*N-/&B1LM4*N-/&B14NL*6O*N-/&B1KK 4NL*6O*N-/&B14NL*6O*N-/&B1-

)? " I
2/1< (> ; "
<G ? ;
) ?%"?I
&'(
!
>
!"#
2/1< ( ; "
<G ? ;
?
&'( ! %!"#

L7*+,-*3456*+-88&*B&A
K 3,77&-*&')(/8*0'+,*8/1)-*%
L7*+,-*3456*+-88&*B&A !"#
K P(&B1-*+,/+*LM4*'&*8/1)K 3,77&-*&')(/8*0'+,*8/1)-*%!"#
K 6-++-1*7(*M-/1Q9"8/1)-*+/1)-+&
K P(&B1-*+,/+*LM4*'&*8/1)K O,'=,*'&*Q-++-1R**
K 6-++-1*7(*M-/1Q9"8/1)-*+/1)-+&
S T7BQ8-*+1/(&G'++-.*-(-1)9R
K O,'=,*'&*Q-++-1R**
S T7BQ8-*4NL*Q/(.0'.+,R
S T7BQ8-*+1/(&G'++-.*-(-1)9R
S T7BQ8-*4NL*Q/(.0'.+,R

!"#$

!"#$
Taken from http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522.htm