'

"

:3
3.1
3.2
3.3
3.4

: .

- ()
3.4.1
3.4.2

3.5

- ( )

(* = ):
3.6

(* )Gaussian Mixture

DHS(2001) : ( 3.1-3.2 ) ( 4.1-4.3 -),

) HTF(2001 ( 8.5.1 ).

 :3

(")

3.1
, ()
.
( ,)supervised learning
. {xi , yi }in1
:

.1 ) p X ,Y ( x, y
.2 ) ( pY | X ( | x .) x X
:

) p X ( x , X n
D {xk }kn 1 .

( ) ,

. x

p X ,Y ) . ( X ,Y
( , ):

- ("")
, -
.
, "
" . " " , "
" ( ,)046201 , ,
. .

 :3

(")

3.2 :
, ) p X ( x ,
,

) pX ( x) p X ( x |
,

, P )T

. (1 ,

" ."
} , { p X ( | ), ,
.
,/
.
:

)N ( ,

X .

d (

.) X

)N ( ,

, X - .

( . d (d 1) / 2 ,
).

) exp(

,X

()

1 x /
,) x 0 ( p X ( x ) e .

) Bern(

X : X . P{X 1} , P{X 0} 1 :

]. [0,1
, . D
) (parameter estimation ,
. \ . D

 :3

(")

3.3
-:

, ( ) ,
) . p(

- , ,
. .

- ), p( | D) (posterior distribution
D . , :

)E ( | D

MMSE

E (( )2 | D) min
.Minimum Mean Square Error (MMSE) Estimator

:)MAP( Maximum a-Posteriori

)arg max p( | D

MAP

.:

: ) p( | D ) E ( | D , .

: ) p( ,
.

 :3

(")

3.4 -
- () , .
, :

) arg max p( D |

MLE

,
.

3.4.1 ()MLE, Maximum Likelihood Estimator

,:
.1 ( ) .
.2
.3 ( )
.

: :

D ,

) L(

) p( D |

( : .)Likelihood Function ,
.

) log L( ) log p( D |

) MLE arg max L(

}) arg max{log L(

 :3

(")

: D {xk }kn 1 ( )
. X :

) p( D | ) p( x1, , xn | ) nk 1 p X ( xk |

) L(

) MLE arg max k 1 p X ( xk |

n

) arg max k 1 log p X ( xk |

n

1 n
) log p X ( xk |
n k 1

arg max

, .

:
.1 ( ):

)N ( ,

X - d , ,

. " :
n

)) exp( ( xk )T 1 ( xk

L( ) p X ( xk | )

1
d
/2
) k 1 (2
||1/2

k 1

) log L( ) C k 1 ( xk )T 1 ( xk
n

:
n

xk

n k 1

MLE

.2 :
.
():

 :3

(")

1 n

xk , MLE ( xk )( xk )T
n k 1

n k 1

.3 Bern( ) :

MLE

, X . 0 1

. p( X 1) , p( X 0) 1 , ():
n

xk

n k 1

MLE

.4 ()Discr( p1,, pk ) :

.X

) p ( p1,, pJ j p j 1 :

X J , X {v1 ,, vJ } : . P( X v j ) p j

. p j 0,

, p J J ( J
.) J 1
p :

)p MLE arg max L( p

pJ

():

)N ( j
n

I {xk v j }
n
k 1

[ p MLE ] j

: . ,
, .

( MLE ):
) g ( .
MLE MLE , ) g (MLE MLE .

, MLE
: MLE

. MLE
MLE

 :3

(")

3.4.2
.
. ( D) : () ,
.
. ) :(bias :

E () E (( D)) ( D) p( D | )dD
. E () :
:

E ( )

) b(

b( ) 0 - (.)unbiased estimator
. :
:

) E (( )2

) MSE(

. : :

) E (( b( ))2
:

) var(

) MSE( ) b( )2 var(

: . MMSE( ) ,
: ,
( , ) , ) b(
. .

:
( 1) : 1.
n

xk )

n k 1

( E ( MLE ) E

-. b( ) 0 :
8

 :3

(")

:
n

) xk ) 2

n
k 1

(( MSE( ) var( ) E

var( X )
n

1
n

...

( 2) : MLE .
:

n 1
n

E ( MLE | , )

(n 1) -
"" MLE
n

, n( xk )( xk )T :

n 1

1
. .

k 1

( 4) : .

 (")

3.5

:3

- ( )

- ,
.
.
, X
. D {xk }kn 1 . X
( )CDF , X , X
. X
, .

.
"

) . FX ( x ) P( X x

FX , x:

1 n

}F ( x ) I {xk x
n k 1
().

( : ?).
( ) . FX:

) E ( g ( X )) g ( x )dFX ( x

) g ( x)dF ( x) n k 1 g ( xi
n

. :

1 n
} I {xk A
n k 1

P( X A)

10

 :3

(")

1 n
xk
n k 1
1 n
( xk )2 ( x )2

1
n

E( X )

Var ( X ) E ( X 2 ) E ( X )2

'.

.
) p X ( x
.X
:

1 n
) ( x xk
n k 1

p X ( x )

( ) .
. , "" .
, .
.1.
, X , X :

J
j 1 R j

.X

( R j )bin :

, x Rj

N (R j ) / n
) V (R j

p X ( x )

) N ( R j dx( d) , R j

)(1

R j dx

V ( R j ) ,

n - .

( x
) , .
11

 :3

(")

, -
. R j .

.
,
, ) p X ( x ( ) . n ,
. n .

, .

.2. ))Kernel density estimation

:

1 n
) K ( x xk
n k 1

p ( x )

) K ( x , () . ) K ( x
, ,0 . -
. () , , ,.
.
, :

1
x
) (K
d
h
h

Kh ( x )

1 n
Kh ( x xk ),
n k 1

p h ( x )

:
.1 K , ) p ( x .
.2
( ).
.3 .
: , () , h
( ) k x . ,k-NN
. ) . k O( n
12

 :3

(")

3.6 *
) p X ( x - , .
) ,(Mixture
):(Gaussian Mixture
J

) p X ( x | ) w j N ( x; j , j
j 1

)) exp( 12 ( x )T 1 ( x

1
d
/2
) (2
||1/2

p X - j w j 1 :

N ( x; , )
. wj 0 ,

: - : x
( )1 , j . w j
( )2 , x ) . N ( j , j
: . {w j , j , j }Jj 1 :
) (MLE :
n

) MLE arg max p( D | ) arg max log p X ( xk |

k 1

, . ,
. , ) L( ) p( D |
,
.
- ,.

:1

.1 } j , j , j
. {w
.2 : xk j ,:

) jk arg max w j N ( xk ; j , j
1 j J

.3 : j , j , j
w j
:
13

 :3

(")

}where n j I { jk j

nj

k 1

i ni

w j

1
nj

k 1 I { jk j}xk
n

I { jk j}( xk j )( xk j )T
k 1

1
j
nj

.4 2,3 ( : .)2

:
.1 2 ) (MAP xk , { j }Jj 1

. p( x | j ) N ( x; j , j ) , P( j ) w j
.2 ( ) .
, " .
.3 ( K-means )K=J ,
. k I
:(EM) 2 1 xk
, jk . , 2 :

) ) ( p( j | xk

) w j N ( xk ; j , j
) j w j N ( xk ; j , j

qkj

where n j k 1 qkj
n

nj

i ni

w j

1
nj

k 1 qkj xk
n

qkj ( xk j )( xk j )T
k 1

1
j
nj

:
.1 )Expectation-Maximization( EM
MLE .
.2 .
.3 , DHS .10.4

14