yi = h ( xi )

y = 0 + 1  x
 y1   x1  1 x1 
  =  +    =    0  y = X
  0 1      
 yN   xN  1 xN   1 

y =  0 + 1  x +  2  x 2 + +p  xp
 0 
 y1  1 x1 x  
p

  1 
1
 = y = X
    
 y N  1 xN xNp   
 p 
 

d=2

yi =  0 + 1  x1i +  2  x2i + +  d  xdi = xTi 

 0 
 y1  1 x   
T
x2
  1 
1
x1  = y = X
    
 yN  1 xTN   
 d 
 

d=2

x2

x1

yi = h (xi ) i = 1,..., N
 ˆ = arg min L ( y1 ,..., yN , x1 ,..., x N ;  )



1
L ( y1 ,..., y N , x1 ,..., x N ;  ) = ( yi − h (xi ) )
2

N i =1
 N

 ( y − x )
1
L ( y1 ,..., yN , x1 ,..., xN ;  ) = T 2
i i
N i =1

1
= ( y − X ) ( y − X )
T

  ( y − X ) ( y − X ) =   ( y T y − y T X − T XT y + T XT X )
1 1
 L =
T

N N
2 T 2 T
= − X y + X X = 
N N
ˆ = ( XT X )−1 XT y

Pseudoinverse of X
 k +1 =  k − h   L  =
k

 .
h
h h
L

2

1
 N N

  ( y − h (x ))  h (x )
1 2
 L = ( yi − h ( xi ) ) =−
2
 i  i   i
N i =1
N i =1

 L −2 ( yi − h ( xi ) )  h ( xi )
test test
  


 

Learning
Model
algorithm
 a
N p

 ( yi − h ( xi ) ) 
1
L ( y1 ,..., yN , x1 ,..., xN ;  ) = + i2
2

N i =1 2 i =1

h ( xi ) = xTi 

ˆ = ( XT X + a I )−1 XT y

a
 N p

 ( y − h (x )) 
1
L ( y1 ,..., yN , x1 ,..., xN ;  ) = +a
2
i  i i
N i =1 i =1
Generates a 3rd order model
 

2 2

1 1
 (1 − r )
N p p

 ( yi − h ( xi ) )  
1
L ( y1 ,..., yN , x1 ,..., xN ;  ) = + ra i + a i2
2

N i =1 i =1 2 i =1

a
 N
yi − h ( xi )

1
M=
N i =1
yi

 2
R 2 = 1 − 2  = y − h ( x)
y
  2
R = 1− 2
2
 = y − h ( x)
y

ŷ = y ŷyˆ = h ( x)

y y
N

 ( y − h ( x) )
N

( y − y )
1 1
 =
2
 y2 =
2 2
i i 
N i =1
N i =1
 cov ( y, yˆ )
r r= yˆ = h ( x)
 y yˆ
ŷ

y and yˆ , R 2 = r 2 = 1
MIC = I ( X ; Y ) = H ( X ) − H ( X | Y )
f ( x, y )
=  f ( x, y ) log
f ( x) f ( y)
dxdy
Quiz