IPS

Introduction to Probability & Statistics

Lecture - 14

Prof. Pritam Ranjan

 IPS

Estimation - Big picture

• Given: (data) {X1 , X2 , ..., Xn } - carefully choose the
sampling technique [we assume random sample (iid)]

• Assume: Xi ∼ f (θ) - requires proper diagnostics / testing

for identifying f (·)

• Estimate the parameter - θ

(Parameter / statistic / estimate / estimator)

Pritam Ranjan / OM&QT 1

 IPS

Estimation - example
• Example 1: Let X1 , X2 , ..., Xn be a random sample from Ber (p). Find an
estimator of p (here θ = p).

• Example 2: Let X1 , X2 , ..., Xn be a random sample from Poi(λ). Find an

estimator of λ (here θ = λ).

• Example 3: Let X1 , X2 , ..., Xn be a random sample from Exp(λ). Find an

estimator of λ (here θ = λ).

• Example 4: Let X1 , X2 , ..., Xn be a random sample from N(µ, σ 2 ). Find

an estimator of σ (here θ = σ).

Pritam Ranjan / OM&QT 2

 IPS

Parameter estimation

• Point estimation for θ

– one number (θ̂)
– varies with random samples

• Interval estimation
– an interval (θ̂L , θ̂U ) where θ may lie
– more robust w.r.t. small variation in the random sample

Pritam Ranjan / OM&QT 3

 IPS

Point Estimation

Pritam Ranjan / OM&QT 4

 IPS

Point estimator

• Finding an estimator is easy

• Finding a “good” estimator is NOT easy

Pritam Ranjan / OM&QT 5

 IPS

Point estimation - methods

• Method of moments

• Maximum likelihood ←−

• Quasi-maximum likelihood or pseudo-maximum likelihood

• Bayesian method: Maximum a posteriori - (mode of the posterior)

• Bayesian method: MCMC - (posterior mean)

• Generalized estimating equation (GEE)

• Variational methods

..
.

Pritam Ranjan / OM&QT 6

 IPS

Point estimation - MLE

[Idea: Given the model, what is the most likely value of θ that can generate
this data?]
1. Write out the likelihood
2. Treat it as a function of the parameter of interest (θ)
3. Maximize L(θ) or log(L(θ)) w.r.t. θ

Pritam Ranjan / OM&QT 7

 IPS

Point estimator - MLE for p

• Example 1: Let X1 , X2 , ..., Xn be a random sample from Ber (p). Find

the MLE of p (here θ = p).

Pritam Ranjan / OM&QT 8

 IPS

Point estimator - MLE for λ

• Example 2: Let X1 , X2 , ..., Xn be a random sample from Poi(λ). Find the

MLE of λ (here θ = λ).

Pritam Ranjan / OM&QT 9

 IPS

Point estimator - MLE for λ

• Example 3: Let Y1 , Y2 , ..., Yn be a random sample from Exp(λ). Find

the MLE of λ (here θ = λ).

Pritam Ranjan / OM&QT 10

 IPS

Point estimator - MLE for µ

• Example 4: Let X1 , X2 , ..., Xn be a random sample from N(µ, σ 2 ). Find

the MLE of µ (here θ = µ).

Pritam Ranjan / OM&QT 11

 IPS

Point estimator - MLE for σ 2

• Example 5: Let X1 , X2 , ..., Xn be a random sample from N(µ, σ 2 ). Find

the MLE of σ 2 (here θ = σ 2 , assume µ known / unknown).

Pritam Ranjan / OM&QT 12

 IPS

Point estimator - MLE for σ

• Example 6: Let X1 , X2 , ..., Xn be a random sample from N(µ, σ 2 ). Find

the MLE of σ (here θ = σ, assume µ known / unknown).

Pritam Ranjan / OM&QT 13

 IPS

Example

• Suppose X1 , X2 , ..., X13 represent the number of IPS sessions skipped

by a random set of 13 PGP students. Suppose X follows binomial
distribution with n = 4 and unknown p. Estimate the expected number
of absence by a student.

Pritam Ranjan / OM&QT 14

 IPS

Application - HW*

• Blue Dart courier claims that the service time to ship a package from
Delhi to Indore is 48 hours, but occasionally it takes more than that. The
actual service time (in hours) X to ship a package from Delhi to Indore
follows the following distribution
� � �
1 (u−θ)
exp − 48 u>θ
fX (u) = 48
0 otherwise.

Assuming you have a data on a survey of 20 random shipment

(X1 , X2 , ..., X20 in hours) from Delhi to Indore (via Blue Dart), find the
maximum likelihood estimator of θ.

Pritam Ranjan / OM&QT 15

 IPS

Application
• Suppose Dell wants to introduce a new digital camera. For a pilot
survey they fixed a price comparable with Sony, but gave out at a 70%
discount rate with its new laptop.

— Dell wants to use its customers’ input in deciding a suitable price for
this camera.

— They conducted a survey and collected 15 responses from Delhi and

20 inputs from Kolkata. City-wise mean and standard deviations (in
thousand rupees) of the assessed prices are as follows:

Mean Standard deviation Sample size

Delhi 20.23 5.43 15
Kolkata 18.01 3.29 20

– What should be a reasonable price for this camera?

Pritam Ranjan / OM&QT 16

 IPS

Application
The daily return of CITI bank obtained from NYSE are to be modelled, where
rt = log(Pt /Pt−1 ).

6
CITI daily returns
Normal density
0.5

4
Quantiles of CITI returns

2
0.4

0
Density

0.3

−2
0.2

−4
0.1

−6
−8
0.0

−5 0 5 −3 −2 −1 0 1 2 3

CITI daily Returns Theoretical normal quantiles

What is a good estimator for the risk/volatility: σ?

Pritam Ranjan / OM&QT 17

 IPS

Goodness of an estimator

Are these estimators GOOD?

Why?

Pritam Ranjan / OM&QT 18

 IPS

Point estimator - properties

• Unbiasedness -

• Minimum variance - most reliable / least volatile

• Efficiency = Unbiased + minimum variance

• Consistency

• Asymptotic normality

• A statistic should also have features like ”minimal”, ”sufficient” and if

possible ”complete” 1

1
Sample must be unbiased, representative and random
Pritam Ranjan / OM&QT 19
 IPS

Efficient estimator

Pritam Ranjan / OM&QT 20

 IPS

Point estimator - unbiasedness

Bias = E[θ̂] − θ

Unbiased =⇒ [Bias=0]

Pritam Ranjan / OM&QT 21

 IPS

Point estimator - minimum variance

Objective: To find an estimator T ∗ (X n ) = θ̂ such that

Var (T ∗ (X n )) ≤ Var (T (X n ))

for any other estimator T (X n ) of θ.

That is, find an estimator that is least uncertain.

Question: Is it possible to find such an estimator?

Answer: No. Restrict in a smaller class (e.g., unbiased estimators).

Pritam Ranjan / OM&QT 22

 IPS

Point estimator - unbiasedness

• Example 1: Let X1 , X2 , ..., Xn be a random sample from Ber (p). Find an

unbiased estimator of p (here θ = p).

Pritam Ranjan / OM&QT 23

 IPS

Point estimator - unbiasedness

• Example 2: Let X1 , X2 , ..., Xn be a random sample from N(µ, σ 2 ). Find

an unbiased estimator of µ.

Pritam Ranjan / OM&QT 24

 IPS

Point estimator - unbiasedness

• Example 3: Let X1 , X2 , ..., Xn be a random sample from N(µ, σ 2 ). Find

an unbiased estimator of σ 2 (here θ = σ 2 and µ known).

Pritam Ranjan / OM&QT 25

 IPS

Point estimator - unbiasedness

• Example 4: Let X1 , X2 , ..., Xn be a random sample from N(µ, σ 2 ). Find

an unbiased estimator of σ 2 (here θ = σ 2 and µ unknown).

Pritam Ranjan / OM&QT 26

 IPS

Point estimator - HW*

• Example 5: Let X1 , X2 , ..., Xn be a random sample from Unif (0, θ). Find
an unbiased estimator of θ.

Pritam Ranjan / OM&QT 27