MidSem2 (MARCH 15, 2018): ECE 431 SIGNAL DETECTION AND ESTIMATION FORMULA SHEET (Page 1/2)

ESTIMATION THEORY BEST LINEAR UNBIASED ESTIMATORS

CRAMER RAO LOWER BOUND Finding the BLUE
Scalar Parameter 𝐶 −1 𝑠 𝑠 𝑇 𝐶 −1 𝑥
𝜕 𝑙𝑛 𝑝(𝒙;𝜃)
1. 𝑎𝑜𝑝𝑡 = 𝑇 −1 ;𝜃̂ = 𝑎𝑜𝑝𝑡 𝑥 = 𝑇 −1
𝑇
𝑠 𝐶 𝑠 𝑠 𝐶 𝑠
1. 𝐸[ ] = 0 𝑓𝑜𝑟 𝑎𝑙𝑙 𝜃 1
𝜕𝜃 2. 𝑣𝑎𝑟(𝜃̂) = 𝑎𝑜𝑝𝑡
𝑇
𝐶 𝑎𝑜𝑝𝑡 =
1 𝑠 𝑇 𝐶 −1 𝑠
2. 𝑣𝑎𝑟(𝜃̂) ≥ 𝜕 𝑙𝑛 𝑝(𝒙;𝜃)
−𝐸[ ]
𝜕𝜃
𝜕 𝑙𝑛 𝑝(𝒙;𝜃) Gauss Markov Theorem
3. = 𝐼(𝜃)(𝑔(𝒙) − 𝜃)
𝜕𝜃 3. 𝒙𝑁𝑥1 = 𝐻𝑁𝑥𝑝 𝜃𝑝𝑥1 + 𝑤𝑁𝑥1 :General Linear Model
𝜕 𝑙𝑛 𝑝(𝒙;𝜃) 2 𝜕2 𝑙𝑛 𝑝(𝒙;𝜃)
4. 𝐸 [( ) ] = −𝐸 [ ] so that
𝜕𝜃 𝜕𝜃 2
1 4. With 𝑤[𝑛]~ℵ(0, 𝑪), BLUE of 𝜽 is given as:
5. 𝑣𝑎𝑟(𝜃̂) ≥ 𝜕 𝑙𝑛 𝑝(𝒙;𝜃) 2 ̂ = (𝑯𝑻 𝑪−𝟏 𝑯)−𝟏 𝑯𝑻 𝑪−𝟏 𝒙
−𝐸[( ) ] 𝜽
𝜕𝜃
−𝟏
5. 𝑣𝑎𝑟(𝜃̂𝑖 ) = [(𝑯𝑻 𝑪−𝟏 𝑯) ]𝒊𝒊
LINEAR MODELS −𝟏
6. 𝑪𝜽̂ = (𝑯𝑻 𝑪−𝟏 𝑯)
General
MAXIMUM LIKELIHOOD ESTIMATION
𝜕𝑏𝑇 𝜃 𝜕𝜃 𝑇 𝐴 𝜃
1. 𝜕𝜃
=𝑏 and
𝜕𝜃
= 2𝐴𝜃 MLE
2. 𝜃̂ = (𝐻 𝑇 𝐻)−1 𝐻 𝑇 𝑥 and 𝐼(𝜃) =
𝐻𝑇𝐻 1. MLE-that value of 𝜃 which maximizes the likelihood function
𝜎2 𝑝(𝒙; 𝜃) for fixed x.
3. 𝐶𝜃̂ = 𝐼 −1 (𝜃) = 𝜎 2 (𝐻 𝑇 𝐻)−1 𝜕 ln 𝑝(𝒙;𝜃)
2. Find MLE 𝜃̂ such that |𝜃̂=𝜃 = 0
4. 𝜃̂~ℵ(𝜽, 𝝈𝟐 (𝐻 𝑇 𝐻)−1 ) 𝜕𝜃
5. DFT relationships for i,j=1,2, …, M<N/2
2𝜋𝑖𝑛 2𝜋𝑗𝑛 𝑁 2𝜋𝑖𝑛 2𝜋𝑗𝑛 𝑁 3. An estimator 𝜃̂ is consistent if, given any ∈> 0,
∑𝑁−1 𝑁−1
𝑛=0 cos( 𝑁 ) cos( 𝑁 ) = 2 𝛿𝑖𝑗 ;∑𝑛=0 sin( 𝑁 ) sin( 𝑁 ) = 2 𝛿𝑖𝑗
lim Pr{| 𝜃̂ − 𝜃|>∈} = 0
Extension to Linear Models 𝑁→∞
𝜎2
6. (a) 𝑤~ℵ(0, 𝐶) (b) 𝐶 −1 = 𝐷𝑇 𝐷 4. Tchebycheff inequality: 𝑃(|𝑋 − 𝑚𝑋 | ≥∈) ≤ 𝑋2
∈
′
7. 𝑥 = 𝐻𝜽 + 𝒘𝑥 = 𝐷𝑥 = 𝐷𝐻𝜃 + 𝐷𝑤 5. Asymptotic Property of MLE: 𝜃̂ 𝑎~ 𝑁(𝜃, 𝐼 −1 (𝜃))
or 𝑥′ = 𝐻′𝜽 + 𝒘′𝒘′ = 𝑫𝒘 ~ ℵ(𝟎, 𝑰) 6. Statistical Linear approximation for large N
8. MVU Estimator of 𝜽 is: 𝑔(𝑢) ≈ 𝑔(𝑢0 ) + 𝑔′ (𝑢0 )(𝑢 − 𝑢0 )
𝜃̂ = (𝐻 ′ 𝑇 𝐻 ′ )−1 𝐻 ′ 𝑇 𝑥 ′ = (𝐻 𝑇 𝐷𝑇 𝐷𝐻)−1 𝐻 𝑇 𝐷𝑇 𝑥
𝜃̂ = (𝐻 𝑇 𝐻)−1 𝐻 𝑇 𝐶 −1 𝑥 MLE for Transformed Parameters
9. 𝐶𝜃̂ = (𝐻 𝐻 ) ⇒ 𝐶𝜃̂ = (𝐻 𝑇 𝐶 −1 𝐻)−1 ⇒ 𝐶𝜃̂ = 𝜎 2 𝐼
′ 𝑇 ′ −1
7. Invariance Property of MLE:
a. Consider a parameter 𝛼 = 𝑔(𝜃).
GENERAL MIN VAR UNBIASED ESTIMATION b. MLE is given as: 𝛼̂ = 𝑔(𝜃̂) where 𝜃̂ is MLE of 𝜃.
Sufficient Statistics c. The MLE of 𝜃̂ is obtained by maximizing 𝑝(𝒙; 𝜃)
𝜎2 1
1. MVU Estimator (Min Var = ) : 𝐴̂ = ∑𝑁−1 𝑛=0 𝑥[𝑛] 𝑝̅ 𝑇 (x;α)= max 𝑝(𝒙; 𝜃)
𝑁 𝑁 {𝜃:𝛼=𝑔(𝜃)}
1 1
2. 𝑝(𝒙; 𝐴) = 𝑵 exp{− ∑𝑁−1 2
𝑛=0 (𝑥[𝑛] − 𝐴) }
(2𝞹𝝈𝟐 ) 𝟐 2𝜎 2

3. If ∑𝑁−1
𝑛=0 𝑥[𝑛] is sufficient condition, then find LEAST SQUARES
𝑝(𝒙; 𝑇(𝒙) = 𝑇0 ; 𝐴) The Least Squares Approach
𝑝(𝒙|𝑇(𝒙) = 𝑇0 ; 𝐴) =
𝑝(𝑇(𝒙) = 𝑇0 ; 𝐴) 𝜃
𝑝(𝒙; 𝐴)𝛿(𝑇(𝒙)−𝑇0 )
4. Nr in Joint PDF: 𝑝(𝒙|𝑇(𝒙) = 𝑇0 ; 𝐴) = 𝑠[𝑛] -
𝑝(𝑇(𝒙)=𝑇0 ;𝐴)
1 1 Signal Model
5. 𝑝(𝑇(𝒙) = 𝑇0 ; 𝐴) = 2
exp{− 2
(𝑇0 − 𝑁𝐴)2 } Error=ϵ[n]
√2𝜋𝑁𝜎 2𝑁𝜎
6. 𝑇(𝒙)~ℵ(𝑁𝐴, 𝑁𝜎 2 ) + x[n]
Finding Sufficient Statistics 1. 𝐽(𝜃) = ∑𝑁−1 2
𝑛=0 (𝑥[𝑛] − 𝑠[𝑛]) with no probabilistic assumptions
7. Neyman-Fisher Factorization(S)-If we can factor the PDF 𝑝(𝒙; 𝜃) as: about data 𝑥[𝑛].
𝑝(𝒙; 𝜃) = 𝑔(𝑇(𝒙), 𝜃)ℎ(𝒙), then 𝑇(𝒙) is sufficient statistic for 𝜃. Linear Least Squares
𝑁−1
8. 𝑝(𝒙; 𝜃) = 𝑔(𝑇1 (𝒙), 𝑇2 (𝒙), … , 𝑇𝑟 (𝒙), 𝜃)ℎ(𝒙) where the ‘𝑟’ statistics ∑ 𝑥[𝑛]ℎ[𝑛]
𝑇1 (𝒙), 𝑇2 (𝒙), … , 𝑇𝑟 (𝒙)are jointly sufficient if conditional PDF does
2. 𝑠[𝑛] = 𝜃ℎ[𝑛] → 𝜃̂ = 𝑛=0
∑𝑁−1 ℎ 2 [𝑛] 𝑛=0
not depend on 𝜃: 𝑝(𝒙|𝑇1 (𝒙), 𝑇2 (𝒙), … , 𝑇𝑟 (𝒙); 𝜃) ∑𝑁−1
𝑛=0 (𝑥[𝑛]ℎ[𝑛])
2

9. For [𝑥 𝑦]𝑇 a Gaussian random vector with mean vector 𝜇 =

3. 𝐽𝑚𝑖𝑛 = ∑𝑁−1
𝑛=0 (𝑥[𝑛])
2
− 𝑁−1
∑𝑛=0 (ℎ[𝑛])2
[𝐸(𝑥)𝐸(𝑦)]𝑇 and Covariance matrix: 4. ̂=
Vector LS :𝜽 (𝑯𝑻 𝑯)−𝟏
𝑯𝑻 𝒙,
𝑣𝑎𝑟(𝑥) 𝑐𝑜𝑣(𝑥, 𝑦) 5. 𝑇
𝐽𝑚𝑖𝑛 = 𝒙 (𝒙 − 𝑯𝜽̂)
𝐶=( ), it is shown that:
𝑐𝑜𝑣(𝑦, 𝑥) 𝑣𝑎𝑟(𝑦) ̂ = (𝑯𝑻 𝑾𝑯)−𝟏 𝑯𝑻 𝑾𝒙,
𝑐𝑜𝑣(𝑥,𝑦)
6. Weighted LS :𝜽
𝐸(𝑥|𝑦) = 𝐸(𝑥) + (𝑦 − 𝐸(𝑦)). 7. 𝐽𝑚𝑖𝑛 = 𝒙𝑇 (𝑾 − 𝑾𝑯(𝑯𝑻 𝑾𝑯)−𝟏 𝑯𝑻 𝑾)𝒙
𝑣𝑎𝑟(𝑦)
10. Rao-Blackwell-Lehmann-Scheffe: Sequential Least Squares
𝑥[𝑛]
∑𝑁−1
𝜃̂ = 𝐸 (𝜃̌ |𝑇(𝒙)) = ∫ 𝜃̂𝑝 (𝜃̌ |𝑇(𝒙)) 𝑑𝜃̌ = 𝑔(𝑇(𝒙)) 8. 𝐴̂[𝑁 − 1] =
𝑛=0 𝜎2
1
𝑛
∑𝑁−1
𝑛=0 𝜎2
𝑛
1
9. 𝑣𝑎𝑟(𝐴̂[𝑁 − 1]) = 1
∑𝑁−1
𝑛=0 𝜎2
𝑛
1
𝜎2 𝑣𝑎𝑟(𝐴̂[𝑁−1])
10. 𝐾[𝑁] = 𝑛
∑𝑁−1
1 =
𝑣𝑎𝑟(𝐴̂[𝑁−1])+𝜎𝑁2
𝑛=0 𝜎2
𝑛
MidSem2 (MARCH 15, 2018): ECE 431 SIGNAL DETECTION AND ESTIMATION FORMULA SHEET (Page 2/2)
Ordered Recursive Least Squares GENERAL BAYESIAN ESTIMATORS
11. 𝜃̂𝑘 = (𝐻𝑘𝑇 𝐻𝑘 )−1 𝐻𝑘𝑇 𝑋 MMSE
2
12. 𝐽𝑚𝑖𝑛𝑘 = (𝑋 − 𝐻𝑘 𝜃̂𝑘 )𝑇 (𝑋 − 𝐻𝑘 𝜃̂𝑘 ) 1. 𝐵𝑚𝑠𝑒 = 𝐸 [(𝜃 − 𝜃 ) ]. Let = 𝜃 − 𝜃̂ : Error.
̂
13. 𝐻𝑘+1 = [𝐻𝑘 ℎ𝑘+1 ] = [𝑁𝑥𝑘 𝑁𝑥1] 2. Deterministic function C(𝜖): Cost Function.
14. Updating: 3. Average cost or E[C(𝜖)] is called as Bayes Risk(R) : ℛ = E[C(𝜖)].
−1
(𝐻 𝐻 )𝑇
𝐻𝑘𝑇 ℎ𝑘+1 ℎ𝑘+1 𝑇 𝑃𝑘⊥ 𝑋
𝜃̂𝑘 − 𝑘 𝑘 4. MMSE estimator minimizes the Bayesian MSE given by: ℛ(𝜃̂) =
ℎ𝑘+1 𝑇 𝑃𝑘⊥ ℎ𝑘+1
a. 𝜃̂𝑘+1 = [ ] 2
E [C(𝜃 − 𝜃̂) ] = Bmse (𝜃̂) while the MMSE estimator is given by:
ℎ𝑘+1 𝑇 𝑃𝑘⊥ 𝑥
ℎ𝑘+1 𝑇 𝑃𝑘⊥ ℎ𝑘+1 ̂ = 𝐸[𝜽|𝒙].
𝜽
𝑃𝑘⊥ = 𝐼 − 𝐻𝑘 (𝐻𝑘𝑇 𝐻𝑘 )−1 𝐻𝑘𝑇 MAVE and MAP
5. MAVE:𝐶(𝜃 − 𝜃̂ ) = |𝜃 − 𝜃̂|
b. 𝐷𝑘 = (𝐻𝑘𝑇 𝐻𝑘 )−1 6. Optimal Estimator 𝜃̂ = 𝑚𝑒𝑑𝑖𝑎𝑛 𝑜𝑓 𝑝(𝜽|𝒙)
𝐷𝑘 𝐻𝑘𝑇 ℎ𝑘+1 ℎ𝑘+1
𝑇
𝐻𝑘 𝐷𝑘 −𝐷𝑘 𝐻𝑘𝑇 ℎ𝑘+1
𝐷𝑘 + 0 ∈< 𝛿
𝑇 𝑃⊥ ℎ
ℎ𝑘+1 𝑇 𝑃⊥ ℎ
ℎ𝑘+1 7. MAP: C(𝜖)={ . Here 𝛿 > 0.
c. 𝐷𝑘+1 = [ 𝑘+1 𝑘+1 𝑘+1 𝑘+1
] 1 ∈> 𝛿
𝐻𝑘𝑇 ℎ𝑘+1 𝐷𝑘 1 8. Optimal Estimator: 𝜃̂ = 𝑎𝑟𝑔 max 𝑝(𝑥|𝜃)𝑝(𝜃)
𝑇 𝑃⊥ ℎ
ℎ𝑘+1 𝑇 𝑃⊥ ℎ
ℎ𝑘+1 𝜃
𝑘+1 𝑘+1 𝑘+1 𝑘+1
Where, 𝑃𝑘⊥ = 𝐼 − 𝐻𝑘 𝐷𝑘 𝐻𝑘𝑇 PDF of the error
(ℎ𝑇 𝑃⊥ 𝑋)2 9. For MMSE:
d. 𝐽min 𝑘+1 = 𝐽min 𝑘 – 𝑇𝑘+1 ⊥𝑘 a. 𝜽̂ = 𝐸(𝜽|𝒙)
ℎ 𝑃 ℎ 𝑘+1 𝑘 𝑘+1
b. Error: 𝜖 = 𝜃 − 𝐸(𝜽|𝒙)
THE BAYESIAN PHILOSOPHY c. Mean of the error is: 𝐸𝑥,𝜃 (𝜖)=0
Prior Knowledge Estimation d. Variance of the error for the MMSE estimator is: var(𝜖) =
2 ̂)
𝐵𝑚𝑠𝑒 (𝜽
1. 𝑚𝑠𝑒(𝐴̂) = ∫(𝜃̂ − 𝜃) 𝑝(𝒙; 𝜃)𝑑𝑥
2 e. 𝜖~ℵ(0, 𝐵𝑚𝑠𝑒 (𝜽̂ ))
2. 𝐵𝑚𝑠𝑒 (𝐴̂) = ∫ ∫(𝜃̂ − 𝜃) 𝑝(𝒙, 𝜃)𝑑𝑥𝑑𝜃
= ∫ 𝑣𝑎𝑟(𝜽|𝑥)𝑝(𝒙)𝑑𝑥
3. 𝜃̂ = 𝐸(𝜃|𝑥) = ∫ 𝜃𝑝(𝜃|𝒙) 𝑑𝜃
𝑝(𝑥 |𝜽)𝑝(𝜽)
4. 𝑝(𝜃|𝒙) =
∫ 𝑝(𝑥 |𝜽)𝑝(𝜽)𝑑𝜃
Bivariate Gaussian
𝑥 − 𝐸(𝑥)
5. 𝑿=[ ]
𝑦 − 𝐸(𝑦)
1 1 𝑇 −1
6. 𝑝(𝑿) = 1 exp [− 𝑿 𝐶
2
𝑿]
2𝜋|𝐶|2
𝑐𝑜𝑣(𝑥,𝑦)
7. 𝐸(𝑦|𝑥) = 𝐸(𝑦) +
𝑣𝑎𝑟(𝑥)
(𝑥 − 𝐸(𝑥)
𝑐𝑜𝑣 2 (𝑥, 𝑦)
𝑣𝑎𝑟(𝑦|𝑥) = 𝑣𝑎𝑟(𝑦) −
𝑣𝑎𝑟(𝑥)

Multi-variate Gaussian (x : kx1, y : 1x1)

𝐶𝑥𝑥 𝐶𝑥𝑦 𝑥 − 𝐸(𝑥)
8. 𝑪=[
𝐶𝑦𝑥 𝐶𝑦𝑦
]𝑿 = [ ]
𝑦 − 𝐸(𝑦)
1 1
9. 𝑝(𝑥, 𝑦) = 𝑘+𝑙 1 exp [− 𝑿𝑇 𝐶 −1 𝑿]
(2𝜋) 2 |𝐶|2
2

10. 𝐸(𝒚|𝒙) = 𝐸(𝒚) + 𝑪𝑦𝑥 𝑪−1𝑥𝑥 (𝒙 − 𝐸(𝒙)

11. 𝐶𝑦|𝑥 = 𝑪𝑦𝑦 − 𝑪𝑦𝑥 𝑪−1 𝑪
𝑥𝑥 𝑥𝑦
Bayesian Linear Model (x,𝜽 𝑗𝑜𝑖𝑛𝑡𝑙𝑦 𝑔𝑎𝑢𝑠𝑠𝑖𝑎𝑛)
12. 𝐸(𝜽|𝒙) = 𝝁𝜽 + 𝑪𝜽 𝑯𝑻 (𝑪𝒘 + 𝐻𝑪𝜽 𝑯𝑻 )−1 (𝒙 − 𝑯𝝁𝜽 )
−1 𝑻 −1
13. 𝐸(𝜽|𝒙) = 𝝁𝜽 + (𝑪−𝟏 𝑻 −𝟏
𝜽 + 𝑯 𝑪𝒘 𝑯) 𝑯 𝑪𝑤 (𝒙 − 𝑯𝝁𝜽 )
𝑻 −𝟏 −1
14. 𝐶𝜃|𝑥 = 𝑪𝜽 − 𝑪𝜽 𝑯𝑻 (𝑪𝒘 + 𝐻𝑪𝜽 𝑯𝑻 )−1 𝐻𝑪𝜽 = (𝑪−𝟏
𝜽 + 𝑯 𝑪𝒘 )
Nuisance Parameters
15. Parameters to be estimated: 𝜽
16. Additional nuisance parameters: α
17. Posterior PDF = 𝑝(𝜽, 𝜶|𝒙)
18. Posterior PDF of 𝜽 only:
𝑝(𝑥|𝜽)𝒑(𝜽)
𝑝(𝜽|𝒙) = ∫ 𝑝(𝜽, 𝜶|𝒙) 𝑑𝜶 =
∫ 𝑝(𝑥|𝜽)𝒑(𝜽)
where 𝑝(𝑥|𝜽) = ∫ 𝑝(𝒙|𝜽, 𝜶)𝑝(𝜶|𝜽)𝑑𝜶
19. If nuisance parameters are indep. of desired parameters:
𝑝(𝑥|𝜽) = ∫ 𝑝(𝒙|𝜽, 𝜶)𝑝(𝜶)𝑑𝜶
Woodbury’s Identity
𝐷−1 𝑈𝑈 𝑇 𝐷−1
20. (𝐷 + 𝑈𝑈𝑇 )−1 = 𝐷 −1 − 1+𝑈 𝑇 𝐷𝑈