Welcome to Scribd. Sign in or start your free trial to enjoy unlimited e-books, audiobooks & documents.Find out more
Standard view
Full view
of .
0 of .
Results for:
P. 1
Best Regress 11

# Best Regress 11

Ratings:
(0)
|Views: 16|Likes:

### Availability:

See more
See less

11/02/2011

pdf

text

original

1
Eight more ClassicMachine Learningalgorithms
Andrew W. Moore Associate ProfessorSchool of Computer ScienceCarnegie Mellon University
www.cs.cmu.edu/~awmawm@cs.cmu.edu412-268-7599
Note to other teachers and users of theseslides. Andrew would be delighted if youfound this source material useful in givingyour own lectures. Feel free to use theseslides verbatim, or to modify them to fityour own needs. PowerPoint originals areavailable. If you make use of a significantportion of these slides in your own lecture,please include this message, or thefollowing link to the source repository of  Andrew’s tutorials:http://www.cs.cmu.edu/~awm/tutorials.Comments and corrections gratefullyreceived.

8:
Polynomial Regression
So far we’ve mainly been dealing with linear regression
:::311723
Y X
2
X
1
::1123:37
X
=
y
=
1
=(3,2)..y
1
=7..

13::1121:37
Z
=
y
=
1
=(1,3,2)..
=(1,x
k1
,x
k2
)y
1
=7..
β
=(
T
)
-1
(
T
)y
est
=
β
0
+
β
1
x
1
+
β
2
x
2

2
It’s trivial to do linear fits of fixed nonlinear basis functions
:::311723
Y X
2
X
1
::1123:37
X
=
y
=
1
=(3,2)..y
1
=7..

12191613::1141:37
Z
=
y
=
=(1 , x
1
, x
2
, x
12
, x
1
x
2
,x
22,
)
β
=(
T
)
-1
(
T
)y
est
=
β
0
+
β
1
x
1
+
β
2
x
2
+
β
3
x
12
+
β
4
x
1
x
2
+
β
5
x
22
It’s trivial to do linear fits of fixed nonlinear basis functions
:::311723
Y X
2
X
1
::1123:37
X
=
y
=
1
=(3,2)..y
1
=7..

12191613::1141:37
Z
=
y
=
=(1 , x
1
, x
2
, x
12
, x
1
x
2
,x
22,
)
β
=(
T
)
-1
(
T
)y
est
=
β
0
+
β
1
x
1
+
β
2
x
2
+
β
3
x
12
+
β
4
x
1
x
2
+
β
5
x
22

Each component of a z vector is called a term.Each column of the Z matrix is called a term columnHow many terms in a quadratic regression with
m
inputs?•1 constant term•m linear terms•(m+1)-choose-2 = m(m+1)/2 quadratic terms(m+2)-choose-2 terms in total
= O(m
2
)
Note that solving
β
=(
T
)
-1
(
T
)
is thus
O(m
6
)

3
Q
th
-degree polynomial Regression
:::311723
Y X
2
X
1
::1123:37
X
=
y
=
1
=(3,2)..y
1
=7..

12191613:11:37
Z
=
y
=
=(all products of powers of inputs inwhich sum of powers is q or less
,
)
β
=(
T
)
-1
(
T
)y
est
=
β
0
+
β
1
x
1
+…
m inputs, degree Q: how many terms?
= the number of unique terms of the form
Qq x x x
miiqmqq
m
=
121
where...
21
Qq x x x
miiqmqqq
m
=
=
021
where...1
210
= the number of unique terms of the form= the number of lists of non-negative integers
[q
0
,q
1
,q
2
,..q
m
]
in which
Σ
q
i
= Q
= the number of ways of placing Q red disks on a row of squares of length Q+m = (Q+m)-choose-QQ=11, m=4
q
0
=2q
2
=0q
1
=2q
3
=4q
4
=3