Professional Documents
Culture Documents
Mean Square Estimation: Pillai
Mean Square Estimation: Pillai
2 E{ |Y ( X ) |2 } E X [ EY { |Y ( X ) |2 X }]
z z
where the inner expectation is with respect to Y, and the outer one is
with respect to X .
Thus
2 E[ E{ | Y ( X ) |2 X }]
E{ | Y ( X ) |2 X } f X ( X )dx. (16-6)
To obtain the best estimator , we need to minimize 2 in (16-6)
with respect to . In (16-6), since f X ( X ) 0, E{ | Y ( X ) |2 X } 0,
and the variable appears only in the integrand term, minimization
of the mean square error 2 in (16-6) with respect to is
equivalent to minimization of E{ | Y ( X ) |2 X } with respect to3 .
PILLAI
Since X is fixed at some value, ( X ) is no longer random,
and hence minimization of E{ | Y ( X ) |2 X } is equivalent to
E{ | Y ( X ) |2 X } 0. (16-7)
This gives
E{| Y ( X ) | X } 0
or
E{Y | X } E{ ( X ) | X } 0. (16-8)
But
E{ ( X ) | X } ( X ), (16-9)
4
PILLAI
in (16-8) we get the desired estimator to be
Yˆ ( X ) E{Y | X } E{Y | X 1 , X 2 ,, X n }. (16-10)
Thus the conditional mean of Y given X 1 , X 2 , , X n represents the best
estimator for Y that minimizes the mean square error.
The minimum value of the mean square error is given by
min
2
E{ | Y E (Y | X ) |2 } E[ E{ | Y E (Y | X ) |2 X }]
var(Y X )
E{var(Y | X )} 0. (16-11)
Thus
f X , Y ( x, y ) kxy 2y
f Y X ( y | x) ; 0 x y 1.
f X ( x) kx (1 x ) / 2 1 x
2 2
(16-13)
Hence the best MMSE estimator is given by 6
PILLAI
1
ˆ
Y ( X ) E{Y | X } x y f Y | X ( y | x)dy
1 1
x y 2y
1 x 2
dy 2
1 x 2 x dy
y 2
1
2 y 3
2 1 x 3 2 (1 x x 2 )
. (16-14)
31 x x 31 x
2 2
3 1 x 2
E{| | } E
2
2 E 0.
ak ak ak (16-19)
But
8
PILLAI
n n
(Y ai X i ) ( ai X i )
Y
i 1
i 1
Xk. (16-20)
ak ak ak ak
E{eh( X )} 0, (16-22)
implying that
e Y E{Y | X } h( X ).
This follows since
E{eh( X )} E{(Y E[Y | X ])h( X )}
E{Yh( X )} E{E[Y | X ]h( X )}
E{Yh( X )} E{E[Yh( X ) | X ]}
E{Yh( X )} E{Yh( X )} 0. 10
PILLAI
Thus in the nonlinear version of the orthogonality rule the error is
orthogonal to any functional form of the data.
The orthogonality principle in (16-20) can be used to obtain
the unknowns a1 , a2 , , an in the linear case.
For example suppose n = 2, and we need to estimate Y in
terms of X 1 and X 2 linearly. Thus
Yˆl a1 X 1 a2 X 2
From (16-20), the orthogonality rule gives
E{ X1*} E{(Y a1 X 1 a2 X 2 ) X 1*} 0
E{ X*2 } E{(Y a1 X 1 a2 X 2 ) X 2*} 0
Thus
E{| X 1 |2 }a1 E{ X 2 X 1*}a2 E{YX 1*}
E{ X 1 X 2*}a1 E{| X 2 |2 }a2 E{YX 2*}
or 11
PILLAI
E{| X 1 |2 } E{ X 2 X 1*} a1 E{YX 1*}
(16-23)
E{ X X *} 2 *
E{| X 2 | } a2 E{YX 2 }
1 2
n2 min E{| |2 }
a1 , a2 ,, an
n
min E{ } min E{ (Y ai X i )*}
*
a1 , a2 ,, an a1 , a2 ,, an
i 1
n
min E{ Y } min
a1 , a2 ,, an
*
a1 , a2 ,, an
i
a E{ X *
l }. (16-24)
i 1
But using (16-21), the second term in (16-24) is zero, since the error is
orthogonal to the data X i , where a1 , a2 , , an are chosen to be
optimum. Thus the minimum value of the mean square error is given
by 12
PILLAI
n
E{ Y } E{(Y ai X i )Y *}
2
n
*
i 1
n
E{| Y | } ai E{ X iY *}
2
(16-25)
i 1
Thus the best linear estimator is also the best possible overall estimator
in the Gaussian case.
Next we turn our attention to prediction problems using linear
estimators.
Linear Prediction
Suppose X 1 , X 2 , , X n are known and X n 1 is unknown.
Thus Y X n 1 , and this represents a one-step prediction problem.
If the unknown is X n k , then it represents a k-step ahead prediction
problem. Returning back to the one-step predictor, let Xˆ n 1
represent the best linear predictor. Then
16
PILLAI
n
Xˆ n 1 = ai X i ,
(16-35)
i 1
a1 X 1 a2 X 2 an X n X n 1
n 1
ai X i , an 1 1, (16-36)
i 1
* * * 1 2
rn rn 1 r1 r0 n
Let 19
PILLAI
r0 r1 r2 rn
*
r1 r0 r1 rn 1
Tn .
(16-44)
r* r* r1* r0
n n 1
Notice that Tn is Hermitian Toeplitz and positive definite. Using
(16-44), the unknowns in (16-43) can be represented as
a1 0
a2 0 Last
a 0
2
column
Tn n
3 1
of (16-45)
0
a T 1
n n
1 2
n
Let
20
PILLAI
Tn11 Tn12 Tn1, n 1
1 T 21
T 22
T 2 , n 1
Tn n n n
. (16-46)
T n 1,1 T n 1, 2 T n 1, n 1
n n n
n . (16-47)
an
T n 1,n 1
1 n
Thus
1
2
n n 1, n 1
0,
T n (16-48) 21
PILLAI
and
a1 Tn1,n 1
1 Tn
2 , n 1
a2
T n 1,n 1 . (16-49)
n
an T n 1,n 1
n
Eq. (16-49) represents the best linear predictor coefficients, and they
can be evaluated from the last column of Tn in (16-45). Using these,
The best one-step ahead predictor in (16-35) taken the form
1 n
Xˆ n 1 n 1,n 1 (Tni ,n 1 ) X i . (16-50)
Tn i 1
and from (16-48), the minimum mean square error is given by the
1
(n +1, n +1) entry of Tn .
From (16-36), since the one-step linear prediction error
n X n 1 an X n an 1 X n 1 a1 X 1 , (16-51) 22
PILLAI
we can represent (16-51) formally as follows
X n 1 1 an z 1 an 1 z 2 a1 z n n
Thus, let
An ( z ) 1 an z 1 an 1 z 2 a1 z n , (16-52)
them from the above figure, we also have the representation
1
n X n 1.
An ( z )
The filter
1 1
H ( z) (16-53)
An ( z ) 1 an z 1 an 1 z 2 a1 z n
represents an AR(n) filter, and this shows that linear prediction leads
to an auto regressive (AR) model. 23
PILLAI
The polynomial An (z ) in (16-52)-(16-53) can be simplified using
(16-43)-(16-44). To see this, we rewrite An (z ) as
An ( z ) a1 z n a2 z ( n 1) an 1 z 2 an z 1 1
a1 0
a
2 0
[ z n , z ( n 1) , , z 1 ,1] [ z n , z ( n 1) , , z 1 ,1] Tn1
an 0
1 2
n
(16-54)
A B
A D CA1 B . (16-56)
C D
In particular if D 0, we get
1 ( 1) n
A B
CA B .
A C 0 (16-57)
0
C [ z n , z ( n 1) , , z 1 ,1], A Tn , B
n2
25
PILLAI
we get
0
r0 r1 r2 rn
0
r1* r0 r1 rn 1
(1) n Tn 2
An ( z ) n
. (16-58)
| Tn | 0 | Tn | * *
rn 1 rn 2 r0 r1
n2
n ( n 1) 1
n 1 z z z 1
z ,, z ,1 0
or
n21 n2 (1 | sn 1 |2 ) n2 , (16-64)
Power Spectrum of a regular stochastic Process
1 2
1 k
33
PILLAI