CS 4104

APPLIED MACHINE LEARNING

Dr. Hashim Yasin

National University of Computer
and Emerging Sciences,
Faisalabad, Pakistan.
GRADIENT DESCENT
 Gradient Descent
3

Have some function

Want

Outline:
• Start with some
• Keep changing to reduce
until we hopefully end up at a minimum
Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Gradient Descent
4

Gradient descent algorithm

Correct: Simultaneous update Incorrect:

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
5

Gradient descent algorithm

Notice : α is the learning rate.

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
6

Gradient descent algorithm Linear Regression Model

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
7

J(0,1)

1
0
Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Gradient Descent
8

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
9

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
10

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
11

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
12

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
13

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
14

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
15

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
16

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Gradient Descent
17

Dr. Hashim Yasin Applied Machine Learning (CS4104)

COST FUNCTION
 Cost Function
19

(for fixed , this is a function of x) (function of the parameter )

3 𝐽(𝜃1) = 𝐽(0) =
1
σ𝑚 (h𝜃(𝑥 𝑖 ) − 𝑦 𝑖 )2
2𝑚 𝑖=1

2 1
= σ𝑚 (𝜃1 𝑥 𝑖 − 𝑦 𝑖 )2
y 2𝑚 𝑖=1

1 1
= [ 0−1 2 + 0−2 2 + 0 − 3 2]
2𝑚
0
0 1 2 3 1
x = [1 + 4 + 9]
2×3
ℎ1 = 0 × 1 = 0 1
= × 14 = 2.3
ℎ2 = 0 × 2 = 0 6

ℎ3 = 0 × 3 = 0
Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Cost Function
20

(for fixed , this is a function of x) (function of the parameter )

3
3
2
y 2
1
1
0
0 1 2 3 0
x
1 -0.5 0 0.5 1 1.5 2 2.5
J(0) = σ3𝑖=1 [12 +22+32]
2×3
1
= × 14 = 2.3
6

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Cost Function
21

(for fixed , this is a function of x) (function of the parameter )

3 1
J(𝜃1) =J(0.5) = σ𝑚
𝑖=1(h𝜃(𝑥 𝑖 ) − 𝑦 𝑖 )2
2𝑚
2 h𝜃(x(i)) 1
y = σ𝑚
𝑖=1(𝜃1 𝑥 𝑖 − 𝑦 𝑖 )2
y(i) 2𝑚
1
1 2 2
= 2𝑚
[ 0.5 − 1 + 1−2 + 1.5 − 3 2 ]
0
0 1 2 3 1
x =
2×3
[(−0.5)2 +(−1)2+(−1.5)2]
ℎ1 = 0.5 × 1 = 0.5
1
= ×(3.5) = 0.58
ℎ2 = 0.5 × 2 = 1 6

ℎ3 = 0.5 × 3 = 1.5
Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Cost Function
22

(for fixed , this is a function of x) (function of the parameter )

3 3

2 2
h𝜃(x(i))
y y(i)
1 1

0 0
0 1 2 3 -0.5 0 0.5 1 1.5 2 2.5
x
1
J(0.5) = σ3𝑖=1 [(0.5−1)2 +(1−2)2+(1.5−3)2] 𝐽 0.5 = 0.58
2×3
1
= ×(3.5) = 0.58
6

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Cost Function
23

1
(for fixed , this is a function of x) J(𝜃1) =J(1) = σ𝑚
𝑖=1(h𝜃(𝑥 𝑖 ) − 𝑦 𝑖 )2
2𝑚
3 1
= σ𝑚
𝑖=1(𝜃1 𝑥 𝑖 − 𝑦 𝑖 )2
2𝑚
2
1
y = [ 1−1 2 + 2−2 2 + 3 − 3 2]
2𝑚
1
1
= 02 + 02 + 02 =0
2𝑚
0
0 1 x 2 3
ℎ1 = 1 × 1 = 1
ℎ2 = 1 × 2 = 2
ℎ3 = 1 × 3 = 3
Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Cost Function
24

(for fixed , this is a function of x) (function of the parameter )

3 3

2 2
y
1 1

0 0
0 1 x 2 3 -0.5 0 0.5 1 1.5 2 2.5
1
J(𝜃1) =J(1) = σ𝑚𝑖=1(h𝜃(𝑥 𝑖 ) − 𝑦 𝑖 )2
2𝑚 𝐽 1 =0
1
= σ𝑚
𝑖=1 (𝜃1 𝑥 𝑖 − 𝑦 𝑖 )2
2𝑚
1
= 02 + 02 + 02 =0
2𝑚
Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Cost Function
25

(for fixed , this is a function of x). (function of the parameter )

0
-0.5 0 0.5 1 1.5 2 2.5

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Acknowledgement
26

Tom Mitchel, Russel & Norvig, Andrew Ng, Alpydin &

Ch. Eick.

Dr. Hashim Yasin Applied Machine Learning (CS4104)