4

(Maximum Likelihood Estimator)

(MLE) Bermoulli (normal)

MLE
4.1

2 (u )

(u )
x 1
y
y

(4.1)

x + u

E (u ) = 0
E (u u ) = 2
ui

u ~ iid

N 0, 2

E ( y i ) =
(pdf) y

f (y) = 2 2

1 / 2

1
exp
2

E ( yi )

(y

= 2

, < y <

(4.2)

y 4.1
(mean) (mode) (median) y

58

+ y = (inflection point) y
y N ( , 2 )
(covariance)
3 normal variable

4.1
(3.2) (covariance matrix)

12

2
2

2
n

( 2 , t
F ) (Judge et al., 1988: 51)
h-statistics
( ) u i
y i y i ( )
(central limit theorem)

y
( E (u ) = 0 )
(Judge et al., 1982: 39)

59

2
4.2 (Maximum Likelihood Estimation)
(Maximum Likelihood Estimation :
MLE)
y (Bernoulli) (Judge et al., 1988: 44, 63-65)
( p p

n = 3

y1 = 1 , y 2 = 1 , y 3 = 0

1
4

3
4

MLE p ()

p =

1
4

3
4

(maximum probability) y
f ( y p ) = p y (1 p ) y = 0 1
(probability) pdf y 1 , y 2 y 3
1 y

( y1

= 1 , y 2 = 1 , y3 = 0 )

f (1 , 1 , 0 )

i =1

yi

(1 p )

1 y i

p p (1 p )

(4.3)

(4.3) p y
(likelihood function) L ( p y ) pdf

() p

p =

p =

1
L
4

0.046

3
L
4

0.141

4
3
4

60

(probability likelihood)

( y1

~p

= 1 , y2 = 1 , y3 = 0)

3
4

MLE

y (Judge et al., 1988: 63)

2 Bernoulli
0-1 p
2
L ( p y ) = p (1 p ) p 4.2 L ( p y )
p p 0.67
pdf (global maximum)

p L ( p y )
(local maximum)
p

d L p y

2 p3p

(4.4)

d p
d

L p y
d p

~p

p
2
3

(4.5)

26 p

~p

= 0

~p

2
3

( )

4.2

L p

.67

)
1

4.2 (Likelihood function)

(MLE)
y ) joint pdf f ( y ) y1 , K , y n

61

L ( y ) f ( y ) L ( y )

y joint pdf f (y ) y
MLE L ( y )

~ y ~ = ~ ( y ) y = ( y1 , K , y n ) ~
(maximum likelihood estimator)
(natural logarithm)
L - (log-likelihood)
L = ln L ( y ) ln L ( y ) L ( y )
(Judge et al., 1988: 64)
1

y1 , K , y n

f yi

(Bernoulli) pdf

yi

(1 p )1 y , i = 1,, n 0

p 1

(4.6)

- (log-likelihood)
L

ln L p

ln p

y i ln ( p ) + (1 y i ) ln (1 p )

n
ln f y i
i =1

y i ( 1 p ) (1 y i

(4.7)

dL

d p

(1 y i )

yi

d L
d p

~p

yi

yi

d 2 L / d p 2
global maximum -

(4.8.1)

1 p

(1 y i )

(1 p )

(4.8.2)

(mean)
p

0 1 ~p

62

y1 , K , y n N ( , 2 )
= ( , 2 )
L

=
=

ln L

( ,

ln L

(4.9)

n
1 ( y i )2
1

ln
exp
2
2

2
i =1

n
1 ( y i )2

ln ( 2 ) 2 2 2 exp
2

1 n ( yi )
n
n
2

ln 2
ln

2
2
2 i =1
2

( )

( )

(4.10)

y = (2.7 , 4.3 , 5.0 ) 2 = 1 2.7 5.0

MLE 0 8
2
2
ln L ( ,
y 1 , y 2 , y 3 ) (4.10) = 1 , y = ( 2.7 , 4.3 , 5.0 )

ln L =

3
2

ln ( 2 )

2
2
2
1
( 2.7 ) + ( 4.3 ) + (5.0 )

0, 1, 2, , 8 ( 3 )

0
1
2
ln L -28.14 -17.64 -10.14

-5.64

-4.14

-5.64

-10.14

-17.64

-28.14

ln L 4 (-4.14) 2 = 1 2
2 2

2 () pdf y = 2
y pdf y = 4 y 3
4.3 4.3 maximum likelihood

63

4
f (y)

2.7

4 4.3

4.3 pdf 2 Maximum Likelihood

(4.10)
1
ln L =

ln 2

( )

ln

ln L

( y x ) ( y x )

1
2

x y x

(4.11.1)

ln L

n
2

( y x ) ( y x )

1
2

(4.11.1)

~2

(4.11.2)
: ~ 2

( x x )1 x y

(y x ~ ) ( y x )
n

(4.11.2)

(4.11.3)
(4.11.4)

u u
n

~ , ~ 2
(4.11.1) (4.11.2)

ln L

x x

(4.11.5)

64

ln L

ln L

(x ( y x ))

(y

x)

(4.11.6)

(y

x)

negative definite
et al., 1988: 66)

2
L
2

( )

x =1

~2

, ~ 2 (Judge

2
y

L
2

(4.11.7)

~4
2

~ 2 2
() ~ =
MLE
2 2 ~ 2 MLE
n 2 ~ 2
~

~ Best Linear
Unbiased Estimator (BLUE)

~ () (Jude et
al., 1988: 81) ~ 2 ~ 2
(efficient)

65

~
2


E

( ) =
2

n 1

)(

~
~2 = y x~

y x / n = ( yi y ) / n

2
~ 2 = n 1

E (~
) =

n 1
n

2 2

2
2 ~
( n 1) /
(3.25) 2

( y x ) ( y x ) =
n 1

u u
n 1

u M u
n 1

(4.12)

(4.12) (quadratic form)

= 0 (covariance matrix) = 2 M
(idempotent) ( M 2 = M ) trace = r () u M u 2
u M u

~ (2r )

(2r ) - = r
-
z ~ N (0 , 1) z 2 - 1 (2r )
z = ( z1 , K , z r ) N (0 , 1) z z
- (2r ) (2r ) = r = 2 r
2 4.4 pdf

2
2

2 2 / (n
2

E
2

(
1)

/ ( n 1 ) 2 ( n 1)

(4.13)

E (2n 1 )
n 1

(4.14)

66

( n 1)
n 1

2 var

( ) =
2

( n 1 )2

( : a b y ,
b )
(4.15) ~ 2

= 2

~ 2 / n (2n 1 )

2 (n 1)

(4.15)

var ( a y + b ) = a 2 var ( y )

(n 1)

(n 1)

( )

f 2

r = 1
r = 5

r = 10

4.4 -
Cramer-Rao
, , E ( ) =
~
~
var (
) var ( ) Cramer-Rao
(Cramer-Rao inequality)
y pdf f ( y ) y1 K y n
joint pdf = n
f

(y )

y1 , K , y n

f
i =1

(y )

67

f ( y ) joint pdf y i y i
y i L ( y ) L ( ) natural log
Cramer-Rao inequality

()

var

d 2 L ( )

E
d2

1 / E (d 2 L / d 2 )
(lower bound) Cramer-Rao lower bound (CRLB)
var ()
(sufficient but not necessary condition)
CRLB
y ~ N ( , 2 ) MLE,

(~ ) = var (~ ) = ~ 2

2
n

Cramer-Rao lower bound

MLE

(~ )

var

=
=

n
var y i / n
i =1

1
n

var ( y i

1
d 2L
E
d 2

1
n y

i
2 var i
=1

n
2
2

n
=
2
n
n

2
n

~ CRLB ~
best unbiased estimator 2 (Judge et
al., 1988: 75) ( (Judge et al., 1982: 48)
CRLB
CRLB

68

(LS)
Maximum Likelihood Estimator (MLE)

() MLE LS

MLE LS
(variance) (biased)
LS
~ 2

n 1

=
n

~ 2 2 MLE

LS
~ 2 2

MLE LS 3

( y )