MATH 241

Problem Set 8 - Solution Key

Question 1-Sol.
(a) If y = [y1 , y2 , ..., ym ]T and θ = [θ1 , θ2 , ..., θn ]T , then model function f (x; θ) can be written
as;
f (x; θ) = θT g(x)
⎡ ⎤
g1 (x)
⎢ g2 (x) ⎥
⎢ ⎥
where g(x) = ⎢ .. ⎥ . If the summation in the cost function is considered as a matrix
⎣ . ⎦
gn (x)
multiplication, the cost function can be rewritten as:
J(θ) = (y − A θ)T (y − A θ) = y − A θ2
where A matrix can be defined as :
⎡ ⎤
g1 (x1 ) . . . gn (x1 )
⎢ g1 (x2 ) . . . gn (x2 ) ⎥
⎢ ⎥
A = ⎢ .. . .. ⎥.
⎣ . . . . ⎦
g1 (xm ) . . . gn (xm )
Then the cost function J(θ) can be minimized with respect to θ, by taking its derivative and
setting the derivative to zero. J(θ) can be rewritten as follows;

J(θ) = (y − A θ)T (y − A θ)
= (y T − (A θ)T )(y − A θ)
= y T y − 2(A θ)T y + (A θ)T (A θ)
= y T y − 2(A θ)T y + θT AT A θ
then the derivative of J(θ) can be written as:
d(J(θ))
d(θ)
= −2AT y + 2AT A θ = 0
then closed form solution for the optimal θ∗ can be found as:
d(J(θ))
d(θ)
= −2AT y + 2AT A θ = 0 ⇒ θ∗ = (AT A)−1 AT y.
(b) An example implementation of MATLAB can be as follows:
clear all
close all
clc

%
Obtain
theta
by
using
the
result
of
the
minimization

theta
opt
=
inv(A'*A)*A'*y;

% Obtain curve fitting result y s

y s = hypothesis(X,Mu,sigma)*theta opt;

% Plot the curve fit

plot(x,y,'o',x,y s,'--')
function out = hypothesis(x,Mu,sigma)
[m,n] = size(x);
MU mult = 1:1:n;

MU mult = Mu*MU mult;

MU mult = repmat(MU mult,m,1);
out = exp(-((x-MU mult).ˆ2)./(2*sigmaˆ2) );
end

2 example outputs of this MATLAB script are as follows: