CS446: Machine Learning

Lecture 16-17 (ML Models – Logistic Regression)

Instructor:
Dr. Muhammad Kabir
Assistant Professor
muhammad.kabir@umt.edu.pk

School of Systems and Technology

Department of Computer Science
University of Management and Technology, Lahore
 Previous Lectures…
 Linear Regression – intuition

 Linear Regression – Mathematical understanding

 Gradient Decent for Linear Regression

 Implementation of Linear Regression

 Today’s Lectures…
 Logistic Regression – intuition

 Logistic Regression – Mathematical understanding

 Gradient Decent for Logistic Regression

 Implementation of Logistic Regression

How Machine Learning Works….
 Logistic Regression - Intuition….
 Supervised Learning
model
 Classification model
 Best for binary
classification problem
 It uses sigmoid function.
 Binary cross entropy loss
function (or) Log loss.
 Logistic Regression - Intuition….
𝑌෠ − 𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑡ℎ𝑎𝑡 (𝑦 = 1)
𝑌෠ − 𝑃 𝑌 = 1 𝑋)
X – input features
w – weights (number of
weights is equal to the number
of input features in a dataset)
b – bias
𝑌෠ = 𝜎(𝑍)

1 Where
෠
𝑌= Large –ve values will be closer to
1 + 𝑒 −𝑍 Z = w.X + b
zero and large +ve values will be
Sigmoid function Suppose Z=5X+10 closer to 1
 Logistic Regression
Advantages
 Easy to implement.
 Perform well on data with linear relationship.
 Less prone to overfitting for low dimensional dataset.

Disadvantages
 High dimensional dataset causes overfitting.
 Difficult to capture complex relationship in a dataset.
 Sensitive to outliers.
 Needs a larger dataset.
 Logistic Regression – Example Calculation….

1
Suppose Z=5X+10 𝑌෠ = Sigmoid function
1+𝑒 −𝑍
 Logistic Regression - Inference….
If Z value is large positive
number, then
1
𝑌෠ =
1+0
𝑌෠ = 1

If Z value is large negative

number, then
1
𝑌෠ =
1+𝑙𝑎𝑟𝑔𝑒 𝑝𝑜𝑠𝑖𝑡𝑣𝑒 𝑛𝑢𝑚𝑏𝑒𝑟
1 𝑌෠ = 0
𝑌෠ =
1 + 𝑒 −𝑍
Sigmoid function
 Loss Function - Logistic Regression….
Loss Function: Loss function measures how far the estimated(predicted)
value is from the its true value. 𝑛
1 2
𝐿𝑜𝑠𝑠 = ෍ 𝑌𝑖 − 𝑌෡𝑖
𝑛
𝑖=1
 Loss Function - Logistic Regression….
Binary Cross Entropy Loss function (or) Log Loss:

Two possible values (0 or 1)

 Loss Function - Logistic Regression….
Binary Cross Entropy Loss function (or) Log Loss:

Two possible values (0 or 1)

 Loss Function - Logistic Regression….
Binary Cross Entropy Loss function (or) Log Loss:

Two possible values (0 or 1)

 Cost Function - Logistic Regression….
Loss function (L) mainly applies for a single training set as compared to the
cost function (J) which deals with a penalty for a number of training sets or
the complete batch.

Loss function (L)

‘m’ represents the number of data points in the training set Cost function (J)
Gradient Decent for Logistic Regression...
Working of Gradient Decent .
 Gradient Descent - Logistic Regression
- Gradient decent is an optimization algorithm for minimizing
the cost function in various ML algorithms.
- It is used for updating the parameters of the learning
model.
- Formula for updating w and b is:
- w --> weight
- b --> bias
- L --> Learning rate
- dw -->partial derivative of loss unction with respect to w.
- db --> partial derivate of loss function with respect to b.
 Logistic Regression

- Things required for Logistic

Regression

- Sigmoid function
- Updating weights through Gradient
decent
- Derivate to get dw and db.
 Chapter Reading

Chapter Chapter 01
Pattern Recognition and
Machine Learning
by
Machine Learning
by
Tom Mitchell
Christopher M. Bishop