Logistic Regression

Classification
 Logistic Regression
• An statistical approach that determine a Binary
outcome based on one or more independent
variables.

• Binary outcome:
» 1/0,
» Yes/No,
» True/False etc.
 Sigmoid Function (Logistic
Function)

  g
 Logistic Regression: Model Representation

• For Linear regression:  g

hθ(x) = θTx

• In Logistic Regression Model,

We want: 0 <= hθ(x) <= 1.

• The hypothesis function:

hθ(x) = g(θTx)
  g

Where, z= θTx
 Interpretation of Hypothesis
output
Hypothesis:
hθ(x) = “estimated probability that y= 1 on input x” = g(θTx)

Example: For a given tumor size = x

we get,
hθ(x) = 0.7
It means that 70% chance of tumor being
malignant.
 Mathematical representation of
the hypothesis output
hθ(x)= P(y=1|x; θ)
Probability that y=1, given x, parameterized by θ

(Probability that y=0) + (Probability that y=1) => 1

P(y=0|x; θ) + P(y=1|x; θ) = 1

P(y=0|x; θ) = 1- P(y=1|x; θ)
 Decision Boundary

Logistic Regression

hθ(x) = g(θTx) = P(y=1|x; θ)

  g

Suppose,
predict “y=1” if hθ(x) >= 0.5
predict “y= 0” if hθ(x) < 0.5
Example: Two Feature x1 and x2
Example: Two Feature x1 and x2
Let
In Logistic Regression
 Cost function for
Logistic Regression

Rewrite the above equation of cost function as

Multiclass Classification
 Multiclass Classification

y=1, y=2, y=3, y=4

y=1, y=2, y=3

y=1, y=2, y=3, y=4

 Use three different Binary classifications

y=1
y=2
y=3
Bias vs Variance: Overfitting Problem
Bias vs Variance: Overfitting Problem
Bias vs Variance: Overfitting Problem
Bias vs Variance: Overfitting Problem
 Addressing Overfitting
Options:
1. Reduce number of features
- Manually select which features to keep.
- Feature selection algorithm
(Feature Reduction approach like PCA etc.)