8.

Point Estimation

Statistics and Econometrics (ECO00037)

Lecture 5: Point and Interval Estimation

8.1 Parameters, Estimators and Estimates

I

Lecturer: Takashi Yamagata (Room A/EC/018)

E-mail: takashi.yamagata@york.ac.uk
Oce Hour: Monday 9.30-11.30

Reading:
Topic 8: Newbold Ch. 8.1; Freund Ch.10.2,10.3 (except Cramer-Rao
inequality), 10.4
Topic 9: Newbold Ch. 8.2, 8.3, 8.4, 8.6, 8.7; Freund Ch.11.2, 11.3,
11.4, 11.5

I
I

Autumn 2014

Suppose that X1 , X2 , ..., Xn are independently and

identically distributed, with common pdf f (X ; ), where
the parameter is interior of parameter space .
The form of pdf is known but the parameter(s), , is
unknown.
An estimator for , n , is a statistic of samples to
estimate
eg. X N ( , 2 ), = and n = X n
An estimator is a random variable with a sampling
distribution
An estimate is realisation of an estimator (a xed
value)
n is used for both estimator and estimate.

1 / 25

8. Point Estimation

2 / 25

8. Point Estimation

8.2 Properties of Estimators

8.2 Properties of Estimators

Statistical properties of estimators can be used to decide

which estimator is most appropriate in a given situation. We
consider unbiasedness, minimum variance, consistency, relative
eciency.
8.2.1 Unbiasedness
I A statistic n is an unbiased estimator of the parameter
if and only if E ( n ) = .

8.2.2 Relative Eciency

I Let n and n be unbiased estimators of a parameter . If
var ( ) < var ( ), we say that is relatively more ecient
than .

f(b)

0.2

X1 , X2 , ..., Xn are iid random variables drawn from

X
N ( , 2 ). Then, n = X n is unbiased estimator for
= .
E ( n ) = E (X n ) =

b^

0.3

Example

Proof.

0.4

1
n

n
i =1 E (Xi )

b~

0.1

-5

=
3 / 25

-4

-3

-2

-1

Distribution of b=

and b=
4 / 25

8. Point Estimation

8. Point Estimation

8.2 Properties of Estimators

I

8.3 Large Sample Properties of Estimators

8.2.3 Best Linear Unbiased Estimator (BLUE): If n

is a linear unbiased estimator ( n = ni=1 ai Xi ) and no
other linear unbiased estimator has a smaller variance,
then n is BLUE.
8.2.4 Mean Square Error (MSE) of n can be shown as
[Example]
E (n

)2 = var ( n ) + bias ( n )

Pr ( Y

where
var ( n ) = E

bias ( n ) = E ( n
I

E n

Sometimes we cannot obtain nite sample distributions of

estimators and instead look at asymptotic/large sample
properties.
8.3.1 Chebyshevs Inequality: Let Y be a random
variable with E (Y 2 ) < . Then

The estimator which has the smallest value of MSE may

be preferred to other estimators.

var ( )
2

MSE ( )

5 / 25

lim Pr (

) = 0 and lim Pr (
n

)=0

+c

8.3.4 Sultskys Theorem: If n

g ( . ) is a continuous function, then

> 0.

for all

> 0.
6 / 25

n =

and

NB: n
(1 n ) = 1 ( = 0), but
E (1 n ) = 1 in general!
8.3.5 Convergence in Distribution:
n( n

(a similar discussion applies to )

8.2.3 Consistency: n is a consistent estimator of the
parameter , if and only if
n

for all

g ( n ) = g ( )

for all > 0, since 2 n 2

0 and 2 + c n 2 as
n
. It is said that has a probability limit equal to
, or
(
) = 0, or = .
n

and

8.3 Large Sample Properties of Estimators

8.3.2 Probability Limits: In the above examples

n

> 0.

8. Point Estimation

8.3 Large Sample Properties of Estimators

I

for all

Now consider a biased estimator of , ,

E ( ) = + cn and var ( ) = 2 n. Then,
Pr (

8. Point Estimation

Consider an unbiased estimator of , , E ( ) =

var ( ) = 2 n. Then,
Pr (

E Y2

= 0.

Y as n

, where Y

N (0, V ).

n( n
) is asymptotically normally distributed
For the same example in 8.3.1,
n Xn

7 / 25

Z as n

, where Z

N (0, 1).
8 / 25

9. Interval Estimation

9.1 Condence Intervals: Small Sample,

known

A (point) estimator is a random variable, subject to sampling

variability (i.e. dierent samples will yield dierent estimates).
Interval estimator take account for this.
I 9.1.1 Mean:
I

Suppose X1 , X2 , ..., Xn are iid normal random variables,

Xi
N ( , 2 ), and is known.
Dene X n = n1 ni=1 Xi and SE X n = n. We
know X n N ( , 2 n ), then
Xn
SE (X n )

Pr

za2

Pr

za2 SE X n

I
I

za2 = 1
Xn

za2 SE X n

Observe that LB and UB of Pr [LB

UB ] are
random variables, where LB = X n za2 SE X n ,
UB = X n + za2 SE X n
Consider = 0.10, so that Pr [LB
UB ] = 0.90.
This means the probability of falling into a random
condence interval (LB, UB ) is 0.90
(lb, ub) is called a 90% condence interval for
Suppose we obtained ten sets of n random samples:
x1,1 , x1,2 , ..., x1,n ; x2,1 , x2,2 , ..., x2,n ; ...; x10,1 , x10,2 , ..., x10,n
We can obtain ten dierent condence intervals,
corresponding to these ten sets of data, which may look
like:

=1

Pr X n za2 SE X n
1
Pr X n + za2 SE X n
1

X n + za2 SE X n
Xn

za2 SE X n

=
9 / 25

9. Interval Estimation
9.1 Condence Intervals: Small Sample,
Example

9. Interval Estimation
9.1 Condence Intervals: Small Sample, known

known

A random sample of 16 observations from a normal population

with 2 = 4 had sample mean 10. Find a 90% condence
interval for the population mean, .
I
I

I
I

xq =
, =
q
Xn N (
,
Xq n z0.05
n

10 / 25

9.1.2 Dierence between Two Means:

I

,n=
, =
), so a 90% condence interval is

n1 and n2 samples are randomly drawn from

X1 N ( 1 , 21 ) and X2 N ( 2 , 22 ) respectively, and
form sample means, X 1 and X 2 , accordingly. 21 , 22 are
known. We can show [Example]
X1

z0.05 =
[See Standard Normal Table]
Answer is (9.1775, 10.8225)

X2

2,

2
1

n1

Therefore the 95% condence interval for

X1

11 / 25

X2

1.96

2
1
n1

2
2
n2

2
2

n2
1

is

12 / 25

9. Interval Estimation

9. Interval Estimation

9.1 Condence Intervals: Small Sample, known

I

9.2 Condence Intervals: Small Sample,

9.1.3 Two Means, Matched Pairs:

I

Dq
I

n matched pairs, (x1 , y1 ), (x2 , y2 ), ...., (xn , yn ), are

randomly drawn from bivariate normal distribution such
that (X , Y ) with mean ( x , y ) and var (X Y ).
var (X Y ) is known. Form sample mean
Dq = n 1 ni=1 Di where Di = Xi Yi .
We can show

the 100 (1
Dq

y,

var (X
n

9.2.1 Mean:
I

I
I

Suppose X1 , X2 , ..., Xn are iid normal random variables,

Xi
N ( , 2 ), and is unknown.
Dene X n = n1 ni=1 Xi and SE X n = S n.
As we know

(X n

Y)

) % condence interval for

unknown

is

The 100 (1
X n ta2,n

)SE X n

tn

) % condence interval for

1 SE X n .

will be

var (X Y )
n

13 / 25

9. Interval Estimation

9. Interval Estimation

9.2 Condence Intervals: Small Sample,

Example

9.2 Condence Intervals: Small Sample, unknown

unknown

I
I

xq =
,s=
,n=
(X n
) (S n) tn
interval is Xq n t0.05,n 1

9.2.2 Dierence between Two Means:

I

A random sample of 16 observations from a normal population

had sample mean 10 and standard deviation 2. Find a 90%
condence interval for the population mean, .
I

14 / 25

, =
1 , so a 90% condence

S
n

t0.05,n 1 =
[See t-distribution Table]
Answer is (9.1235, 10.8765) [compare to the previous
example]

15 / 25

n1 and n2 samples are randomly drawn from

X1 N ( 1 , 2 ) and X2 N ( 2 , 2 ) respectively, and
form sample means, X 1 and X 2 , accordingly. 2 is
unknown
NB: assumed 21 = 22 = 2 , homoskedasticity,
otherwise not straightforward solution
(X 1 X 2 ) ( 1 2 )
As
tn 1 + n 2 2
SE (X 1 X 2 )
where SE X 1
with S 2 =
j = 1, 2
the 100 (1
X1 X2

X2 =

S2
n1

(n1 1 )S 12 +(n 2 1 )S 22
,
(n 1 1 )+(n 2 1 )

+ Sn2

Sj2 =

nj
i =1

(X i

nj 1

) % condence interval for

t 2,n1 +n2 2 SE X 1 X 2

Xj)

,
2

is
16 / 25

9. Interval Estimation

9. Interval Estimation

9.2 Condence Intervals: Small Sample, unknown

I

9.3 Condence Intervals: Large Sample, unknown

9.2.3 Two Means, Matched Pairs:

I

I
I
I

n matched pairs, (x1 , y1 ), (x2 , y2 ), ...., (xn , yn ), are

randomly drawn from bivariate normal distribution,
(X , Y ).
Suppose our interest is in the condence interval of
mean of D = X Y , D .
Form sample mean Dq = n 1 ni=1 Di where
Di = Xi Yi .
Dq
D
As
tn 1
q

I
I

SD2

n
i =1 ( D i

where SE (Dq ) =
with
=
n 1
the 100 (1
) % condence interval for
Dq t 2,n 1 SE (Dq )

2
Dq )

Suppose X1 , X2 , ..., Xn are iid random variables with

mean and nite variance 2 . n 30.
Dene X n = n1 ni=1 Xi and SE X n = S n.
By Central Limit Theorem (CLT)
d

SE (D )

S D2
n

9.3.1 Mean:

(X n
)SE X n
Z as n
, where
Z N (0, 1).
The 100 (1
) % condence interval for will be
X n za2 SE X n .

is

17 / 25

9. Interval Estimation

9. Interval Estimation
9.3 Condence Intervals: Large Sample, unknown

9.3 Condence Intervals: Large Sample, unknown

Example

I
I
I

9.3.2 Dierence between Two Means:

I

A random sample of 36 observations had sample mean 10 and

standard deviation 2. Find a 90% condence interval for the
population mean, .
I

18 / 25

xq =
,s=
,n=
, =
(X n
) (S n) N (0, 1), approximately, so a 90%
S
condence interval is Xq n za2
n
z0.05 =
Answer is (9.4517, 10.5483)

I
I
I

19 / 25

n1 and n2 samples are randomly drawn from

X1 i.i.d.( 1 , 21 ) and X2 i .i .d.( 2 , 22 ) respectively,
and form sample means, X 1 and X 2 , accordingly.
min (n1 , n2 ) 30.
(X 1 X 2 ) ( 1 2 ) d
By CLT
Z as min (n1 , n2 )
SE (X 1 X 2 )
where SE X 1
Sj2

with
=
the 100 (1
X1 X2

X2 =

nj
i =1

(X i

Xj)

S 12
n1

S 22
n2

, j = 1, 2
nj 1
) % condence interval for
z 2 SE X 1 X 2

is

20 / 25

9. Interval Estimation

9. Interval Estimation

9.3 Condence Intervals: Large Sample, unknown

9.2 Condence Intervals: Large Sample, unknown

I

9.3.3 Two Means, Matched Pairs:

I

I
I
I

n 30 matched pairs, (x1 , y1 ), (x2 , y2 ), ...., (xn , yn ), are

randomly drawn from bivariate normal distribution,
(X , Y ).
Suppose our interest is in the condence interval of
mean of D = X Y , D .
Form sample mean Dq = n 1 ni=1 Di where
Di = Xi Yi .
Dq
D d
By CLT
Z as n
q

P
(1
n

SE (D )

S2

(D

i
2
D
i =1
where SE (Dq ) =
n with SD =
n 1
the 100 (1
) % condence interval for
q
q
D z 2 SE (D )

2
Dq )

9.3.4 Proportion of Successes:

By Central Limit Theorem we found the binomial random
variable X is distributed as N (n , n (1
))
approximately.
Now dene the sample proportion of successes as
P = X n. From the above result,
)

N (0, 1) approximately

as P is consistent estimator of
P

is

P (1 P )
n

, we argue that

N (0, 1) approximately

21 / 25

9. Interval Estimation

9. Interval Estimation

9.3 Condence Intervals: Large Sample, unknown

I

9.3 Condence Intervals: Large Sample, unknown

9.3.4 Proportion of Successes:

It follows that the 100 (1
) % condence interval for
is
P (1 P )
P z 2
n
9.3.5 Dierences between Proportions of Successes:
I

I
I

Two independent populations,

X1 Binomial (x1 ; n1 , 1 ) X2 Binomial (x2 ; n2 ,
Dene P1 = X1 n1 and P2 = X2 n2 .
Analogous to 9.3.2, the 100 (1
) % condence
interval for 1
is
2
P1

P2

22 / 25

9.3.3 Proportion of Successes:

Example
In a survey, a random sample of 100 purchasers of toilet rolls,
20 indicated cheapness as the major reason for brand selection.
Find a 90% condence interval for the population proportion.
I

2 ).

P1 (1 P1 ) P2 (1 P2 )
+
n1
n2

23 / 25

p=

,n=

P
P (1 P )n

, p (1 p )n =
, =
N (0, 1), approximately, so a 90%

condence interval is P z
z0.05 =
Answer is (0.1342,0.2658)

P (1 P )
n

24 / 25

Useful Sampling Distributions for Condence Interval

and Hypothesis Testing for
Estimating E (X ) = by Xq = n 1 ni=1 X i
Xi
i .i .d .N ( , 2 )
n small, 2 known
n small, 2 unknown

1 n D ,
q
Estimating E (D ) = D = X
Y by D = n
i =1 i
where D i = X i Y i with Matched Pair X i , Y i
Di
i .i .d .N ( D , 2D )
n small, 2D known
n small, 2D unknown

i .i .d .( , 2 )
n large

Xi

Di

i .i .d .( D , 2D )
n large

q Yq
Estimating E (X i Y i ) = X
Y by X
nX
1 nY
q
where Xq = nX 1 i =
1 & Y = nY
i =1 Y i , X i & Y i are independently drawn
Xi
i .i .d .N ( X , 2X ) Y i
i .i .d .N ( Y , 2Y )
Xi
i .i .d .( X , 2X ) Y i
i .i .d .( Y , 2Y )
min (n X , nY ) small,
min (nX , nY ) small,
min (n X , nY ) large
2 & 2 known
2 & 2 unknown
X
Y
X
Y
Assume 2X = 2Y = 2
Estimating

by P = Xn where X
n small
N/A

Binomial (x ; n, )
n small
N/A

n large or n

& n (1

)>4

X
Y
Estimating X
Y by P X = n X and PY = n Y
where X
Binomial (x ; nX , X ) & Y
Binomial (y ; nY , Y ), X & Y are independent
min (nX , n Y ) small
min (nX , nY ) small
min (n X , nY ) large
N/A
N/A

25 / 25