Introduction To ML - AI PDF

Advanced Driver Assistance System
Dr. Shree Prasad M.

Asst. Prof., Dept. of E.E.E.,
BITS Pilani
O↵-Campus, Bangalore.
Advanced Driver Assistance System 1 / 53 BITS Pilani, Pilani Campus

Outline
1 Linear Regression.
2 Classification: Logistic Regression

Linear Regression
Area Cost
[Acre] [Crore]
0.54 2.57
0.81 4.78
2.16 9.65
4.32 16.73
5.50 19.25

Fitting a Straight Line
Independent variable “x”: Site Dimension.

Dependent variable “y ”: Cost
Matrix Method
2 3 2 3
2.57 1 0.54
ao + 0.54 ⇥ a1 = 2.57 6 4.78 7 61
6 7 6 0.817 
7 ao
ao + 0.81 ⇥ a1 = 4.78 6 9.65 7 = 61 2.167
6 7 6 7 a
416.735 41 4.325 1
ao + 2.61 ⇥ a1 = 9.65
19.25 1 5.50
ao + 4.32 ⇥ a1 = 16.73 (1)
y = Xa
ao + 5.50 ⇥ a1 = 19.25

Fitting a Straight Line: Determining “a”
Solving: y = X a
i Pre-multiply both sides by X T
XTy = XTXa
⇣ ⌘ 1
T
ii Pre-multiply by X X
⇣ ⌘ 1
T
a= X X XTy

Fitting a Straight Line Contd.
Figure 1: ao = 1.7495, and a1 = 3.3183
3 2
2.57
   6 4.78 7
a0 0.5743 0.1404 1 1 1 1 1 6 7
6 9.65 7
=
a1 0.1404 0.0527 0.54 0.81 2.16 4.32 5.50 6
416.735
7
19.25
Machine Learning: Logic
Parameters for the model ha (x) = ao + a1 x.

Parametric model for single “example” : ha (x[i]) = ao + a1 x[i]
M
1 X
J (ao , a1 ) = (ha (x [i]) y [i])2 (2)
2M
i=1
M: Number of traning examples.

Estimate of parameter a is represented as â
M
1 X
â = min (ha (x [i]) y [i])2 (3)
ao ,a1 2M
i=1

Cost Function
Consider the parametric model ha (x) = a1 x with training examples

(x [1] = 1, y [1] = 2), and (x [2] = 2, y [2] = 4)
Cost Function
2
X
J (a1 ) = (a1 x [i] y [i])2 = (1 · a1 2)2 + (2 · a1 2)2
i=1
J (a1 ) = 5a12 20a1 + 20

Minimizing Cost Function
Steepest Descent Gradient Agorithm

@J (a1 )
a1 [j + 1] = aj [j] ↵
@a1 a1 =a1 [j]
a1 [j + 1] = aj [j] ↵ (10a1 [j] 20)
SDG Algorithm: Illustration with ↵ = 0.001, and a1 [j] = 1.5

i. a1 [1] = 1.5 0.001 ⇥ (10 ⇥ 1.5 20) = 1.505
ii. a1 [2] = 1.505 0.001 ⇥ (10 ⇥ 1.505 20) = 1.50995
iii. a1 [3] = 1.50995 0.001 ⇥ (10 ⇥ 1.50995 20) = 1.51485
..
.

SDG: Impact of ↵
Figure 2: ↵ = 0.001; J1 (a1 ) = 1.5 Figure 3: ↵ = 0.25; J1 (a1 ) = 1.5

Cost function for two parameters
Cost function for Eqn. (1)

1 n
J (ao , a1 ) = (ao + 0.54a1 2.57)2 + (ao + 0.81a1 4.78)2 · · ·
10
o
2 2 2
+ (ao + 2.16a1 9.65) + (ao + 4.32a1 16.73) + (ao + 5.50a1 19.25)
(4)
J (ao , a1 ) = 5ao2 + 54.5257a12 105.96ao 408.5044a1 + 26.66ao a1 + 773.0312
â = min J (a0 , a1 ) (5)

ao ,a1

Machine Learning: Logic Contd.
SDG Update equations

ao [i] = ao [i 1] ↵ (10ao [i 1] + 26.66a1 [i 1] 105.96) (6)
| {z }
@J(ao ,a1 )/@ao
a1 [i] = a1 [i 1] ↵ (109.0514a1 [i 1] + 26.66ao [i 1] 408.5044) (7)

| {z }
@J(ao ,a1 )/@a1

Classification: Logistic Regression
Di↵erent from linear regression: Output takes discrete values.

Example:
* Email Spam/Not Spam.
* Fraudulent/Non Fraudulent transaction.
* Deficient/Not deficient.
Case Study: Vitamin D Level in nanogram/mililiter (ng/ml).
Question: Yes or No?
* Vit. D 2 {0, 30}. Normal? No. Hence output y = 0.
* Vit. D > 30. Normal? Yes. Hence output y = 1.

Logistic Regression Contd.
Method 1 : Using Linear Regression Approach

* Through training, determine the parameters a0 , and a1 of the straight
line y = a0 + a1 x.
* Generate discrete output (from continuous output) :
i If y > 0.5, Vit. D normal.
ii Elseif y < 0.5, Vit. D Not normal
* Output “y ” can take any value. However, for classification problem
“y ” should take only discrete values.
Method 2: Using “Sigmoid” function.
i For y = 1, z = a0 + a1 x
should be positive.
ii For y = 0, z = a0 + a1 x
should be negative.
Note: ŷ = f (z) value is restricted between 0, and 1.

Desired property of cost function:
i. If ŷ [i] = y [i], J (a0 , a1 ) should be minimum or zero (Correct
classification).
ii. If ŷ [i] 6= y [i], J (a0 , a1 ) should take large value (Misclassification).
Binary Entropy Cost Function for single training data point:
J (a0 , a1 )[i] = {y [i] ln (ŷ [i]) + (1 y [i]) ln (1 ŷ [i])} (8)

* Case i: If y [i] = 1, and ŷ [i] ⇡ 1, then J (a0 , a1 ) ⇡ 0
* Case ii: If y [i] = 0, and ŷ [i] ⇡ 0, then J (a0 , a1 ) ⇡ 0
* Case iii: If y [i] 6= ŷ [i], then J (a0 , a1 ) is large.

Logistic Regression Cost Function

M
1 X
J (a0 , a1 ) = {y [i] ln (ŷ [i]) + (1 y [i]) ln (1 ŷ [i])} (9)
M
i=1

Traffic Sign Detection: Two Class
Figure 4: Sigmoid Acitvation Function

Traffic Sign Detection: Two Class
Figure 5: Softmax Activation

One Hot Encoding

Traffic Sign Detection

Traffic Sign Detection Contd.




y0 = w0,0 · x0 + w1,0 · x1 + w2,0 · x2 + · · · + w2499,0 · x2499 + b0

y1 = w0,1 · x0 + w1,1 · x1 + w2,1 · x2 + · · · + w2499,1 · x2499 + b1
y2 = w0,2 · x0 + w1,2 · x1 + w2,2 · x2 + · · · + w2499,2 · x2499 + b2
y3 = w0,3 · x0 + w1,3 · x1 + w2,2 · x3 + · · · + w2499,3 · x2499 + b3
2 3 2 32 x 3 2 3
y0 w0,0 w1,0 . . . w2499,0 0 b0
6
6y1 7 6w0,1 w1,1 . . . w2499,1 7 6 x1 7 6b1 7 7
6 7=6 7 .. 7 + 6 7
4y2 5 4w0,2 w1,2 . . . w2499,2 5 6 4 . 5 4 b2 5
y3 w0,3 w1,3 . . . w2499,3 x2499 b3
y = WTx + b

Multi Label Classification
2 3
w0,0 w0,1 w0,2 w0,3
6 w1,0 w1,1 w1,2 w1,3 7
6 7
6 w2,0 w2,1 w2,2 w2,3 7
W =6 7 (10)
6 .. .. .. .. 7
4 . . . . 5
w2499,0 w2499,1 w2499,2 w2499,3
[0] ⇥J [1]
W 2 RJ (11)
* J [0] = Length of feature vector.

* J [1] = Number of classes.
* Each matrix element = wPixel no.⇥Class no.

Output Layer:
y = WTx + b (12)
c = Softmax (y ) (13)
e yi
ci = Softmax (yi ) = K (14)
P y
e j
j=1
Cross Entropy Loss Function

K
X1
J= ck ln (cˆk ) (15)
k=0


Multi Label Classification Contd.



2 3
w0,0 w0,1 ... w0,9
6 w1,0 w1,1 ... w1,9 7
6 7
6 w2,0 w2,1 ... w2,9 7
W =6 7 (16)
6 .. .. .. .. 7
4 . . . . 5
w783,0 w783,1 . . . w783,9
[0] ⇥J [1]
W 2 RJ (17)
* J [0] = Length of feature vector.

* J [1] = Number of classes.
* Each matrix element = wPixel no.⇥Class no.

Deep Neural Networks
Single Neuron
i Linear combination:
z =w ·x +b
z = w 0 x0 + w 1 x1 + · · · + w L 1 xL 1 +b
ii Activation Function:
y = f (z)
Types of Activation Function
i Sigmoid.
ii REctified Linear Unit (RELU).
iii Leaky RELU.
iv Tan Hyperbolic.

Deep Neural Network

Deep Neural Network Contd.
⇢⇣ ⌘T
x [1]
=f W [1]
· x [0] + b [1] (18)
W [1] 2 RL⇥L1 ; b [1] 2 RL1 ⇥1

⇢⇣ ⌘T
[2]
x =f W [2]
· x [1] + b [2] (19)
W [1] 2 RL1 ⇥L2 ; b [1] 2 RL2 ⇥1

⇢⇣ ⌘T
[3]
x =f W [3]
· x [2] + b [3] (20)
W [2] 2 RL2 ⇥L3 ; b [1] 2 RL3 ⇥1

Convolutional Neural Networks
Figure 7: Output with no padding, and

Figure 6: Input and Kernel striding
Figure 9: Output with stride S = 2

Figure 8: Output with padding (Same
input and output size.)

CNN With Padding

CNN With Stride

Convolution Detailed Illustration







3D-Convolution Neural Networks

3D-Convolution Neural Networks Contd.

Applying activation function for the convolution output across each

filter.

Figure 10: Output dimension at next layer.

Deep CNN Problem 1

Deep CNN Problem 2

Introduction To ML - AI PDF

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Introduction To ML - AI PDF

Uploaded by

Copyright:

Available Formats

Advanced Driver Assistance System

Dr. Shree Prasad M.

Advanced Driver Assistance System 1 / 53 BITS Pilani, Pilani Campus

2 Classification: Logistic Regression

Advanced Driver Assistance System 2 / 53 BITS Pilani, Pilani Campus

Advanced Driver Assistance System 3 / 53 BITS Pilani, Pilani Campus

Independent variable “x”: Site Dimension.

Advanced Driver Assistance System 4 / 53 BITS Pilani, Pilani Campus

Advanced Driver Assistance System 5 / 53 BITS Pilani, Pilani Campus

Figure 1: ao = 1.7495, and a1 = 3.3183

Parameters for the model ha (x) = ao + a1 x.

M: Number of traning examples.

Advanced Driver Assistance System 7 / 53 BITS Pilani, Pilani Campus

Consider the parametric model ha (x) = a1 x with training examples

J (a1 ) = 5a12 20a1 + 20

Steepest Descent Gradient Agorithm

a1 [j + 1] = aj [j] ↵ (10a1 [j] 20)

SDG Algorithm: Illustration with ↵ = 0.001, and a1 [j] = 1.5

Advanced Driver Assistance System 9 / 53 BITS Pilani, Pilani Campus

Figure 2: ↵ = 0.001; J1 (a1 ) = 1.5 Figure 3: ↵ = 0.25; J1 (a1 ) = 1.5

Advanced Driver Assistance System 10 / 53 BITS Pilani, Pilani Campus

Cost function for Eqn. (1)

â = min J (a0 , a1 ) (5)

Advanced Driver Assistance System 11 / 53 BITS Pilani, Pilani Campus

SDG Update equations

a1 [i] = a1 [i 1] ↵ (109.0514a1 [i 1] + 26.66ao [i 1] 408.5044) (7)

Advanced Driver Assistance System 12 / 53 BITS Pilani, Pilani Campus

Di↵erent from linear regression: Output takes discrete values.

Advanced Driver Assistance System 13 / 53 BITS Pilani, Pilani Campus

Method 1 : Using Linear Regression Approach

Method 2: Using “Sigmoid” function.

Note: ŷ = f (z) value is restricted between 0, and 1.

J (a0 , a1 )[i] = {y [i] ln (ŷ [i]) + (1 y [i]) ln (1 ŷ [i])} (8)

Advanced Driver Assistance System 16 / 53 BITS Pilani, Pilani Campus

Logistic Regression Cost Function

Advanced Driver Assistance System 17 / 53 BITS Pilani, Pilani Campus

Figure 4: Sigmoid Acitvation Function

Advanced Driver Assistance System 18 / 53 BITS Pilani, Pilani Campus

Figure 5: Softmax Activation

Advanced Driver Assistance System 19 / 53 BITS Pilani, Pilani Campus

Advanced Driver Assistance System 20 / 53 BITS Pilani, Pilani Campus

Advanced Driver Assistance System 21 / 53 BITS Pilani, Pilani Campus

Advanced Driver Assistance System 22 / 53 BITS Pilani, Pilani Campus

Advanced Driver Assistance System 23 / 53 BITS Pilani, Pilani Campus

Advanced Driver Assistance System 24 / 53 BITS Pilani, Pilani Campus

Advanced Driver Assistance System 25 / 53 BITS Pilani, Pilani Campus

y0 = w0,0 · x0 + w1,0 · x1 + w2,0 · x2 + · · · + w2499,0 · x2499 + b0

Advanced Driver Assistance System 26 / 53 BITS Pilani, Pilani Campus

* J [0] = Length of feature vector.

Advanced Driver Assistance System 27 / 53 BITS Pilani, Pilani Campus

Cross Entropy Loss Function

Advanced Driver Assistance System 28 / 53 BITS Pilani, Pilani Campus

Advanced Driver Assistance System 29 / 53 BITS Pilani, Pilani Campus

Advanced Driver Assistance System 30 / 53 BITS Pilani, Pilani Campus

Advanced Driver Assistance System 31 / 53 BITS Pilani, Pilani Campus

Advanced Driver Assistance System 32 / 53 BITS Pilani, Pilani Campus

* J [0] = Length of feature vector.

Advanced Driver Assistance System 33 / 53 BITS Pilani, Pilani Campus

Advanced Driver Assistance System 35 / 53 BITS Pilani, Pilani Campus

Advanced Driver Assistance System 36 / 53 BITS Pilani, Pilani Campus

W [1] 2 RL⇥L1 ; b [1] 2 RL1 ⇥1

W [1] 2 RL1 ⇥L2 ; b [1] 2 RL2 ⇥1