VE564 Summer 2023

Lecture 5: Minimax Estimation

Prof. H. Qiao
UM-SJTU Joint Institute
June 20, 2023
Outline

Basics
Minimax and Bayesian
Example

Extensions
Improper Prior
Nonparametric
Restricted Bayes Estimator
Basics
Recap

To estimate g pθq, we use the loss function L:

Lpθ, dq ě 0, @θ, d

and Lpθ, g pθqq “ 0, @θ. The risk function for estimator δpX q of g pθq is

Rpθ, δq “ Eθ pLpθ, δqq

We focus on the risk function which is more general than the variance.

Minimax and Bayesian Example

 2
Minimax Estimator

Definition
An estimator δ M of θ, which minimizes the maximum risk:

inf sup Rpθ, δq “ sup Rpθ, δ M q

δ θ θ

is called a minimax estimator.

Different from MVU estimator, it is always difficult to find the minimax

estimator δ M for a class of problems.

2
Chap 4., TPE
Minimax and Bayesian Example
 2
Minimax Estimator

Though the minimax estimation is not Bayesian, as will be seen later,

Bayes estimators provide a tool for solving minimax problems. There is a
synthesis of the Bayesian and frequentist approaches.

The Bayesian approach provides a means of constructing an estimator

that has optimal frequentist properties.

The Bayesian paradigm is well suited for the construction of possibly

optimal estimators, but is less well suited for their evaluation. The
frequentist paradigm is complementary as it is suited for risk evaluation
but less suited for construction.

2
Chap 4., TPE
Minimax and Bayesian Example
Minimax Estimation from a Bayesian Point of View

Let Λ denote a general prior distribution on the parameter space Θ. The

Bayes risk of the Bayes solution δΛ is given by
ż
rΛ “ r pΛ, δΛ q “ Rpθ, δΛ qdΛpθq

A prior distribution on Θ is least favorable if rΛ ě rΛ1 for all prior

distributions Λ1 .

Minimax and Bayesian Example

Minimax Estimation from a Bayesian Point of View

3
Condition for a Bayes estimator to be minimax
Suppose that Λ is a distribution on Θ such that
ż
r pΛ, δΛ q “ Rpθ, δΛ qdΛpθq “ sup Rpθ, δΛ q
θ

Then:

• δΛ is minimax
• If δΛ is the unique Bayes solution with respect to Λ, it is the unique
minimax procedure
• Λ is least favorable

3
Theorem 1.4, Chap. 4, TPE
Minimax and Bayesian Example
Minimax Estimation from a Bayesian Point of View

Corollary 1
If a Bayes solution δΛ has constant risk, then it is minimax.

Minimax and Bayesian Example

Minimax Estimation from a Bayesian Point of View

Corollary 2
Let ωΛ be the set of parameter points at which the risk function of δΛ
takes on its maximum, that is

ωΛ “ tθ : Rpθ, δΛ q “ sup Rpθ1 , δΛ qu

θ1

Then, δΛ is minimax if and only if

ΛpωΛ q “ 1

Minimax and Bayesian Example

Example

Suppose that X has the binomial distribution bpp, nq and we want to

estimate p with squared error loss.

We first check whether the unbiased estimator X {n is minimax. Its risk

function pp1 ´ pq{n has a unique maximum at p “ 1{2. If X {n is
minimax, we should be able to find a prior distribution Λ that assigns
probability 1 to p “ 1{2. But the corresponding Bayes estimator is
δpX q “ 1{2, not X {n. Thus, X {n is not minimax.

We now try a Beta prior distribution Λ “ Bpa, bq, the Bayes estimator
can be shown to be
a`x
δΛ pxq “
a`b`x
with risk
1 (
npp1 ´ pq ` rap1 ´ pq ´ bps2
pa ` b ` nq2
Minimax and Bayesian Example
Example
?
n
If we set a “ b “ 2 , the risk is constant. That is, the estimator
?
x ` 12 n
δpxq “ ?
n` n

is a minimax estimator.

The minimax estimator δ is biased while X {n is unbiased. The risk rn of

δ and Rn of X {n are

1 1
rn “ ? , Rn “ pp1 ´ pq{n
4 p1 ` nq2

it can be shown that

sup Rn ppq{rn “ Rn p1{2q{rn Ñ 1, nÑ8

p

So when n is large, X {n is better.

Minimax and Bayesian Example
Example

If we replace the loss function with

pd ´ pq2
Lpp, dq “
pp1 ´ pq

The estimator X {n becomes a constant risk estimator with uniform prior

on p0, 1q, and hence a minimax estimator.

Minimax and Bayesian Example

Extensions
Minimax Estimator - Improper Prior

In some cases, the existence of a least favorable distribution is not

guaranteed. For example, the problem of estimation the mean θ of a
normal distribution with known variance. Since all possible values θ play
a completely symmetrical role that none is easier to estimate than any
other. It is reasonable to conjecture the least favorable distribution is
”uniform” on the real line.

Such distribution is called the Jeffreys prior and not proper. There are
two ways to handle this situation

• With a improper prior distribution, the posterior distribution may be

proper. And thus, we can compute the generalized Bayes estimator.
• We can approximate the improper prior distribution with a sequence
of proper distributions.

Improper Prior Nonparametric Restricted Bayes Estimator

Minimax Estimator - Improper Prior

A sequence of prior distribution tΛn u is least favorable if for every prior

distribution Λ we have

rΛ ď r “ lim rΛn
nÑ8

where
ż
rΛn “ Rpθ, δn qdΛn pθq

is the Bayes risk under Λn

Improper Prior Nonparametric Restricted Bayes Estimator

Minimax Estimator - Improper Prior

A Weaker Guarantee
Suppose that tΛn u is a sequence of prior distributions with Bayes risks
rn and that δ is an estimator for which

sup Rpθ, δq “ r
θ

Then,

• δ is minimax and
• the sequence tΛn u is least favorable.

• Even if the Bayes estimators δn are unique, it is not possible to

conclude that δ is the unique minimax estimator.
• It is in general difficult to check the condition, but it becomes easy
when the Λn are conjugate priors.

Improper Prior Nonparametric Restricted Bayes Estimator

Nonparametric Minimax Estimator

Let X be a random quantity with distribution F , and let g pF q be a

functional defined over a set F1 of distributions F . Suppose that δ is a
minimax estimator of g pF q when F is restricted to some subset F0 of F1 .
Then, if

sup RpF , δq “ sup RpF , δq

F PF0 F PF1

δ is minimax also when F is permitted to vary over F1 .

Improper Prior Nonparametric Restricted Bayes Estimator

Example: Nonparametric mean

Let X1 , ¨ ¨ ¨ , Xn be i.i.d with distribution F of finite mean θ to be

estimated with respect to the squared error loss. If the maximum risk of
every estimator of θ is infinite, the minimax problem is meaningless. To
avoid this difficulty, we consider two possible restrictions on F :

(a) Bounded variance

VarF pXi q ď M ă 8

(b) Bounded range

´8 ă a ă Xi ă b ă 8

Improper Prior Nonparametric Restricted Bayes Estimator

Example: Nonparametric mean

Under restriction (a), we want to first show X̄ is minimax for F0 of

normal distributions with finite variance. To show this, we try to find a
sequence of Bayes estimators δn with least favorable distributions. We
consider the conjugate normal distribution N pµ, b 2 q on the parameter of
mean, the Bayes estimator is given by

nx̄{σ 2 ` µ{b 2
δΛb “
n{σ 2 ` 1{b 2

with risk
1
rΛb “
n{σ 2 ` 1{b 2
σ2
As b Ñ 8, rΛb Ò n and X̄ is minimax for any finite σ 2 .

Improper Prior Nonparametric Restricted Bayes Estimator

Example: Nonparametric mean

Now consider the larger family F1 of all distributions F satisfying (a)

that F0 Ă F1 . The following condition holds with δ “ X̄ :

sup RpF , δq “ sup RpF , δq

F PF0 F PF1

Then, X̄ is the minimax estimator for F1 .

*Note that X̄ has bounded variance M{n regardless of the distributions.

Improper Prior Nonparametric Restricted Bayes Estimator

Example: Nonparametric mean

Now we consider the bounded range restriction (b). Without losing

generality, we choose a “ 0 and b “ 1, and let F1 denote the class of
distributions F with F p1q ´ F p0q “ 1. We shall show that the least
favorable distributions put all their mass on the end points 0 and 1, and
the estimator
?
n 1 1
δpX1 , X2 , ¨ ¨ ¨ , Xn q “ ? X̄ ` ?
1` n 21` n

is minimax over F1 .

Let F0 denote the set of distributions F according to which

PpXi “ 0q “ 1 ´ p, PpXi “ 1q “ p, 0ăpă1

We have already shown that δ is minimax on F0 with p “ EpXi q.

Improper Prior Nonparametric Restricted Bayes Estimator

Example: Nonparametric mean

The risk of δ on F1 is
„ ? 2
n 1
RpF , δq “ E ? X̄ ` ? ´θ
1` n 2p1 ` nq
„ 
1 1
RpF , δq “ ? 2 VarF pX q ` p ´ θq2
p1 ` nq 2

As 0 ď X ď 1, we know X 2 ď X and

VarF pX q “ EpX 2 q ´ θ2 ď θ ´ θ2

Then
1
RpF , δq ď ?
4p1 ` nq2

where the right hand side is the constant risk of δ over F0 .

Improper Prior Nonparametric Restricted Bayes Estimator

Restricted Bayes Estimator

The minimax estimators can be quite unsatisfactory over a large part of

the parameter space as we consider the least favorable distribution.

On the other hand, with some other prior distribution, the Bayes
estimator have have very high maximum risk. Then, we would like to
study the restricted Bayes estimator.

A restricted Bayes estimator is obtained by minimizing the Bayes risk

subject to the constraint

Rpθ, δq ď Mp1 ` q, @θ

where M is the maximum risk of the minimax estimator.

Improper Prior Nonparametric Restricted Bayes Estimator

Restricted Bayes Estimator

Binomial restricted Bayes estimator: suppose we believe p to be near

zero and we select Bp1, bq with fairly high value of b as the prior. The
Bayes estimator is
X `1
δpX q “
n`b`1
with risk
npp1 ´ pq ` rp1 ´ pq ´ bps2
Rpp, δq “ Epδ ´ pq2 “
rn ` b ` 1s2

At p “ 1, the risk is rb{pn ` b ` 1qs2 , which for fixed n and large b can
be arbitrarily close to 1. The constant risk of the minimax estimator is
?
1{4p1 ` nq2 . However, for fixed b, we have
?
4p1 ` nq2 sup Rpp, δq Ñ 1 as n Ñ 8

Then the estimator δ will be a restricted Bayes estimator for large enough
sample size n.
Improper Prior Nonparametric Restricted Bayes Estimator
Restricted Bayes Estimator

Normal mean: Let X1 , ¨ ¨ ¨ , Xn be i.i.d as N pθ, 1q. We consider the

family of prior distributions N p0, b 2 q. The Bayes estimator is given by

nb 2
δb “ x̄
1 ` nb 2
The maximum risk of δb is unbounded (not the Bayes risk!) for all n
while the minimax risk is 1{n. Then no such Bayes estimator can be
restricted Bayes.

As a comprise, the following estimator can be used

$
&x̄ ` M If x̄ ă ´M{p1 ´ cq
’
’
δM “ c x̄ If |x̄| ă M{p1 ´ cq
’
’
%x̄ ´ M If x̄ ą M{p1 ´ cq

The maximum risk tends toward 1{n as M Ñ 0 while the Bayes estimator
as M Ñ 8
Improper Prior Nonparametric Restricted Bayes Estimator