Machine Learning & Pattern Recognition

Linear Regression –
Polynomial Curve fitting
 Linear Regression - Polynomial Curve Fitting

• Given a training data set comprising N observations {xn}, where n = 1,...,N

together with corresponding target values {tn}. M is the order of the
polynomial and xj denotes the x raised to the power of j.

Although the function y(x,w) is a non-linear function of x, it is a linear function of coefficient of w.

Thushari Silva, PhD Essentials of Machine Learning
 Linear Regression - Polynomial Curve Fitting

• Model Selection - How to choose M?

when M = 0 and M = 1, polynomials give rather poor fit to the data.

Thushari Silva, PhD Essentials of Machine Learning

 Linear Regression - Polynomial Curve Fitting

• Model Selection - How to choose M?

when M = 3 polynomial seems to give a better fit to the data. And when and M = 9, it gives
an excellent fit to the data. In fact polynomial passes exactly each data point and E(w∗) = 0
Thushari Silva, PhD Essentials of Machine Learning
 Linear Regression - Polynomial Curve Fitting

• Over-fitting can be overcome by increasing the size of the data set.

Over fitting problem can also be avoided by introducing Bayesian approach.

Thushari Silva, PhD Essentials of Machine Learning

 Linear Regression - Polynomial Curve Fitting

• Model Selection - How to choose M?

when M is small,ERMS takes relatively larger values. When M is between 3 and 8,ERMS
takes relatively smaller values. And when M = 9, ERMS goes to Training error goes zero, but
the Testing Error is very large (i.e Polynomial with larger M has tuned to the random noise).
Thushari Silva, PhD Essentials of Machine Learning
 Linear Basis Function Models

• The simplest linear model for regression is:

Y ( X ,W ) = w0 + w1 x1 + w2 x2 + ... + wD xD , where X = ( x1 , x2 ,..., xD )
• This simplest can be extended to cater many class of models by:

• In the simplest case, we use linear basis functions:

Thushari Silva, PhD Essentials of Machine Learning
 Linear Basis Function Models

• Examples for basis functions:

Polynomial basis functions

• These are global. A small change in X

affects all basis functions.

Thushari Silva, PhD Essentials of Machine Learning

 Linear Basis Function Models

• Examples for basis functions:

Gaussian basis functions

• These are local. A small change in X

only affects nearby basis functions.
µj and s control location and scale
(width) respectively.

Thushari Silva, PhD Essentials of Machine Learning

 Linear Basis Function Models

• Examples for basis functions:

Sigmoidal basis functions

where

• These are local. A small change in X

only affects nearby basis functions.
µj and s control location and scale
(slope) respectively.

Thushari Silva, PhD Essentials of Machine Learning

 Likelihood Function -Intuition

Likelihood Function
• The probability of observing the given data as a function parameters.
• The Likelihood function for{xi }in=1 which are independent and identically
distributed with P( xi ,q ) and{ti }in=1 are the corresponding targets.

• This approach essentially chooses the value for the parameter, θ ,that
maximizes the likelihood of seeing the sample data that is collected
• These parameters are known as Maximum Likelihood Estimators (MLE)

Thushari Silva, PhD Essentials of Machine Learning

 Gaussian Distribution – Uni-variate vs. Multivariate

Thushari Silva, PhD Essentials of Machine Learning

 Linear Regression – Maximum Likelihood and least squares

• Assume observations from a deterministic function with added Gaussian

noise:

• which can be equally said as:

• The likelihood function for { xi }in=1 which are iid

Thushari Silva, PhD Essentials of Machine Learning

 Gaussian Distribution – Uni-variate vs. Multivariate

Thushari Silva, PhD Essentials of Machine Learning

 Linear Regression – Maximum Likelihood and least squares

Thushari Silva, PhD Essentials of Machine Learning

 Linear Regression – Maximum Likelihood Estimators

• Computing the gradient and setting it to zero yields

Thushari Silva, PhD Essentials of Machine Learning

 Linear Regression – Maximum Likelihood Estimators
• Computing the gradient and setting it to zero yields

Thushari Silva, PhD Essentials of Machine Learning

 Linear Regression
Multiple Outputs

Thushari Silva, PhD Essentials of Machine Learning

 Linear Regression – Summary

Thushari Silva, PhD Essentials of Machine Learning

 Linear Regression – Summary

Thushari Silva, PhD Essentials of Machine Learning