Detection: chapter 3

Statistical detection theory I

Natasha Devroye
devroye@ece.uic.edu
http://www.ece.uic.edu/~devroye
Spring 2010

Example of detection

Example of estimation

Goals

infer value of unknown state of nature based on noisy observations

Mathematically, optimally
Noise
Nature

phenomenon
experiment
the ``truth

model of hypothesis
H

Transmission /
measurement

model of observation
or transmission
process

Processing

decision rule
estimation function

mapping from
observation space
to decisions/
estimates

Detection example 1: digital communications

Noise

Source

Encoder

Channel

Decoder

10001010100010

Detect?

Detection example 2: Radar communication

Send
Receive
Hypothesis

Detect?

Hypothesis

Destination

Further examples
Sonar: enemy submarine!
Image processing: detect and aircraft from infrared images
Biomedicine: cardiac arryhthmia from heartbeat sound wave
Control: detect occurrence of abrupt change in system to be controlled
Seismology: detect presence of oil deposit

Difference between detection and estimation?

Detection:

Discrete set of hypotheses

Right or wrong

Estimation:

Continuos set of hypotheses

Almost always wrong - minimize error instead

Estimation example 1: communications

Pulse amplitude modulation (PAM)

Analog source

Sampler

Receiver?

Estimation example 2: Radar

Send
Receive
Hypothesis

Estimate?
Hypothesis

Transmitter

Our methods
Will treat everything generally, with a unified mathematical representation
Bias towards Gaussian noise
Examples mainly drawn from communications / radar

Aside: Classical vs. Bayesian

Classial
Hypotheses/parameters are fixed, non-random

Bayesian
Hypotheses/parameters are treated as random variables with
assumed priors (or a priori distributions)

Other useful references:

Course
Textbook:
Fundamentals
of Statistical
Signal Processing,
Volume
Theory, by
Harry
L. Van
Trees, Detection,
Estimation,
and Modulation
Theory, Part
I, II, 1:
III,Estimation
IV
M.Poor,
Kay, Introduction
Prentice Hall,
(possibly)
H.Steven
Vincent
to 1993
Signaland
Detection
andFundamentals
Estimation of Statistical Signal Processing,
Volume
2: Detection
Theory,
by Steven
M. Kay,
Prentice
Hall 1998.
Louis
L. Scharf
and Cedric
Demeure,
Statistical
Signal
Processing:
Detection, Estimation, and Time
Series
OtherAnalysis
useful references:
Carl
Helstrom,
Elements
of SignalEstimation,
Detection and Estimation.
out ofPart
print,
Harry
L. Van Trees,
Detection,
ModulationIt's
Theory,
I, so
II, here's
III, IVmy pdf copy.
H. Vincent Poor, Introduction to Signal Detection and Estimation
Notes:
follow
courseDemeure,
textbooksStatistical
fairly closely,
using
a mixtureDetection,
of slides (highlighting
the Time
Louis IL.will
Scharf
andthe
Cedric
Signal
Processing:
Estimation, and
main
points
and with nice illustrations) and more in-depth blackboard derivations/proofs in class. I
Series
Analysis
will
a pdf version
of the
they become
here, but
be my
given
Carlpost
Helstrom,
Elements
of slides
SignalasDetection
and ready
Estimation.
It'sthe
outderivations
of print, sowill
here's
pdfincopy.
class only.
Fundamentals
of Statistical
Signal
Volume 1:fairly
Estimation
Theory,
M.of
Kay,
Prentice
Hall, 1993the
Notes:
I will follow
theProcessing,
course textbooks
closely,
usingbya Steven
mixture
slides
(highlighting
Topics:
Estimation
Theory:
main points
and with
nice illustrations) and more in-depth blackboard derivations/proofs in class. I
General
Minimum
Variance
Unbiased
Ch.2,
5 here, but the derivations will be given in
will post
a pdf version
of the
slides asEstimation,
they become
ready
Cramer-Rao
Lower
Bound,
Ch.3
class only.
Linear Models+Unbiased Estimators, Ch.4, 6
Maximum
LikelihoodTheory:
Estimation, Ch.7
Topics: Estimation
Least
squares
estimation,
Ch.8Unbiased Estimation, Ch.2, 5
General Minimum Variance
Bayesian
Estimation,
Ch.10-12
Cramer-Rao Lower Bound, Ch.3
Detection
Theory:
Linear Models+Unbiased
Estimators, Ch.4, 6
Statistical
Detection
Theory,
Ch.3 Ch.7
Maximum
Likelihood
Estimation,
Fundamentals
of Statistical
Signal
Processing, Volume 2: Detection Theory, by Steven M. Kay, Prentice Hall 1998.
Deterministic
Signals,
Ch.4
Least squares estimation, Ch.8
Random Signals, Ch.5
Bayesian Estimation, Ch.10-12
Statistical Detection Theory 2, Ch.6
Detection Theory:
Non-parametric and robust detection
Statistical Detection Theory, Ch.3
Deterministic Signals, Ch.4
Grading: Weekly homeworks (15%), Exam 1 = max(Exam1, Exam 2, Final) (20%), Exam 2 =
Random Signals, Ch.5
max(Exam 2, Final) (20%), Project (15%), Final exam (30%).
Statistical Detection Theory 2, Ch.6
Non-parametric and robust detection

Course outline

1 of 3

Grading: Weekly homeworks (15%), Exam 1 = max(Exam1, Exam 2, Final) (20%), Exam 2 =
1/11/10 8:50 PM
max(Exam 2, Final) (20%), Project (15%), Final exam (30%).

1 of 3

1/11/10 8:50 PM

Estimation: General Minimum Variance Unbiased Estimation

Bias: (expected value of estimator - true value of data)

MVUE:

Estimation: Cramer-Rao lower bound

Lower bound on variance of ANY unbiased estimator!
Usage:
assert whether an estimator is MVUE
benchmark against which to measure the performance of
an unbiased estimator
feasibility studies
Depends on?

Nature

Noise
Transmission /
measurement

Estimation: linear models

Whats a linear model and why is it useful?

What can be said?

Best Linear Unbiased Estimators (BLUE)

Processing

Estimation: Maximum Likelihood Estimation

Alternative to MVUE which is hard to find in general
Easy to compute - very widely used and practical
What is the MLE?

Properties?

Estimation: Least Squares

Alternative estimator with no general optimality properties, but nice and
intuitive and no probabilistic assumptions on data are made - only need a
signal model

Advantages?
Disadvantages?

Estimation: Bayesian Estimation

Parameter to be estimated is assumed to be random, according to some prior
distribution which models our knowledge of it
Bayesian Minimum Mean Squared Error (MMSE):

Applications to Gaussian noise / linear model

Estimation: Bayesian Estimation

General risk functions - arbitrary cost functions

Maximum a posteriori (MAP) estimation

Linear MMSE: constrain estimator to be linear - very practical

Other useful references:

Course
Textbook:
Fundamentals
of Statistical
Signal Processing,
Volume
Theory, by
Harry
L. Van
Trees, Detection,
Estimation,
and Modulation
Theory, Part
I, II, 1:
III,Estimation
IV
M.Poor,
Kay, Introduction
Prentice Hall,
(possibly)
H.Steven
Vincent
to 1993
Signaland
Detection
andFundamentals
Estimation of Statistical Signal Processing,
Volume
2: Detection
Theory,
by Steven
M. Kay,
Prentice
Hall 1998.
Louis
L. Scharf
and Cedric
Demeure,
Statistical
Signal
Processing:
Detection, Estimation, and Time
Series
OtherAnalysis
useful references:
Carl
Helstrom,
Elements
of SignalEstimation,
Detection and Estimation.
out ofPart
print,
Harry
L. Van Trees,
Detection,
ModulationIt's
Theory,
I, so
II, here's
III, IVmy pdf copy.
H. Vincent Poor, Introduction to Signal Detection and Estimation
Notes:
follow
courseDemeure,
textbooksStatistical
fairly closely,
using
a mixtureDetection,
of slides (highlighting
the Time
Louis IL.will
Scharf
andthe
Cedric
Signal
Processing:
Estimation, and
main
points
and with nice illustrations) and more in-depth blackboard derivations/proofs in class. I
Series
Analysis
will
a pdf version
of the
they become
here, but
be my
given
Carlpost
Helstrom,
Elements
of slides
SignalasDetection
and ready
Estimation.
It'sthe
outderivations
of print, sowill
here's
pdfincopy.
class only.
Fundamentals
of Statistical
Signal
Volume 1:fairly
Estimation
Theory,
M.of
Kay,
Prentice
Hall, 1993the
Notes:
I will follow
theProcessing,
course textbooks
closely,
usingbya Steven
mixture
slides
(highlighting
Topics:
Estimation
Theory:
main points
and with
nice illustrations) and more in-depth blackboard derivations/proofs in class. I
General
Minimum
Variance
Unbiased
Ch.2,
5 here, but the derivations will be given in
will post
a pdf version
of the
slides asEstimation,
they become
ready
Cramer-Rao
Lower
Bound,
Ch.3
class only.
Linear Models+Unbiased Estimators, Ch.4, 6
Maximum
LikelihoodTheory:
Estimation, Ch.7
Topics: Estimation
Least
squares
estimation,
Ch.8Unbiased Estimation, Ch.2, 5
General Minimum Variance
Bayesian
Estimation,
Ch.10-12
Cramer-Rao Lower Bound, Ch.3
Detection
Theory:
Linear Models+Unbiased
Estimators, Ch.4, 6
Statistical
Detection
Theory,
Ch.3 Ch.7
Maximum
Likelihood
Estimation,
Fundamentals
of Statistical
Signal
Processing, Volume 2: Detection Theory, by Steven M. Kay, Prentice Hall 1998.
Deterministic
Signals,
Ch.4
Least squares estimation, Ch.8
Random Signals, Ch.5
Bayesian Estimation, Ch.10-12
Statistical Detection Theory 2, Ch.6
Detection Theory:
Non-parametric and robust detection
Statistical Detection Theory, Ch.3
Deterministic Signals, Ch.4
Grading: Weekly homeworks (15%), Exam 1 = max(Exam1, Exam 2, Final) (20%), Exam 2 =
Random Signals, Ch.5
max(Exam 2, Final) (20%), Project (15%), Final exam (30%).
Statistical Detection Theory 2, Ch.6
Non-parametric and robust detection

Course outline

1 of 3

Grading: Weekly homeworks (15%), Exam 1 = max(Exam1, Exam 2, Final) (20%), Exam 2 =
1/11/10 8:50 PM
max(Exam 2, Final) (20%), Project (15%), Final exam (30%).

1 of 3

1/11/10 8:50 PM

Detection: Statistical Detection Theory

Binary hypothesis testing

Hypothesis

Hypothesis

Detection: Statistical Detection Theory

Binary hypothesis testing

Detection: Statistical Detection Theory

Detection: Deterministic Signals

How to detect known signals in noise?

The famous matched filter!

T(x)
x[n]

X
s[n]

Generalized matched filter

> 2 hypotheses

Detection: Random Signals

What if s[n] is random?

Key idea behind estimator-correlator:

Linear model simplifies things again...

Detection: Statistical Decision Theory II

model for the pdfs under 2 hypotheses are unknown

Uniformly most powerful test

Generalized likelihood ratio test
Bayesian approach
Wald test
Rao test

Detection Theory

Binary hypothesis testing

Binary hypothesis testing

Neyman-Pearson hypothesis testing

Example 1

Useful problem 2.1

Example 2

Example 3

Deflection coefficient

Receiver Operating Characteristics (ROC)

[different terminology - can you map it to ours?]

Bayesian risk

Bayesian risk

Bayesian risk example

Multiple hypothesis testing

Multiple hypothesis testing

Multiple hypothesis testing example