Machine Learning Department of Electrical Engineering

Summary of Last Lecture

• Bayes Classifiers to learn P(Y|X)

• MLE & MAP estimates for parameters

• Conditional Independence

• Naïve Bayes >>> makes Bayesian learning practical

Machine Learning Department of Electrical Engineering

 Naive Bayes in Nutshell

• Bayes Rule

• Assuming conditional independence among Xjs

Machine Learning Department of Electrical Engineering

 Training Naive Bayes (Example 1)

X1 =>Sunny, X2=>Temp, X3=>Humidity, X4=>Wind, X5=>Water, X6=>Forecast

Sunny =1 Warm = 1 High = 1 Strong = 1 Warm = 1 Same = 1
Rainy = 0 Cold = 0 Norm = 0 Mild = 0 Cool = 0 Change = 0

Machine Learning Department of Electrical Engineering

 Training Naive Bayes (Example 1)

Probabilities using Probabilities using

MLE MAP
P(sky = sunny | play)
P(sky = sunny | ~play)
P(Temp = warm | play)
P(Temp = warm | ~play)
P(hum = hi | play)
P(hum = hi | ~play)
P(wind = strong | play)
P(wind = strong | ~play)

Machine Learning Department of Electrical Engineering

 Training Naive Bayes (Example 1)

Probabilities using Probabilities using

MLE MAP
P(water = cool | play)
P(water = cool | ~play)
P(forecast = same | play)
P(forecast = same | ~play)

New Example:

Sky = Sunny >>> X1 = 1

Temp = Cold >>>X2 = 0
Humidity = High >>> X3 = 1
Wind = Strong >>>X4 = 1
Water = Cool >>>X5 = 0
Forecast = Change >>>X6 = 0

Machine Learning Department of Electrical Engineering

 Training Naive Bayes (Example 2)

Y = 1 (live in Islamabad) G = 1 (Shop at Centaurus)

D = 1 (Drive to EME by car/bike/bus) B = 1 (Visited Lake View Park)

P (Y = 1) = P (Y = 0) =

Probabilities Probabilities Probabilities Probabilities

using MLE using MAP using MLE using MAP
P(G|Y) P(~G|Y)
P(G| ~Y) P(~G| ~Y)
P(D|Y) P(~D|Y)
P(D| ~Y) P(~D| ~Y)
P(B|Y) P(~B|Y)
P(B| ~Y) P(~B| ~Y)

New Example
G = 1, D = 1, B = 0 P(Y=1) =?
Machine Learning Department of Electrical Engineering
 Naive Bayes: Another perspective

Machine Learning Department of Electrical Engineering

 What if Xis are Continuous?

Machine Learning Department of Electrical Engineering

 Gaussian (Normal) Distribution

Machine Learning Department of Electrical Engineering

 What if Xis are Continuous?

Machine Learning Department of Electrical Engineering

 Gaussian Naïve Bayes (GNB) Algorithm

Machine Learning Department of Electrical Engineering

 Estimating Parameters

jth training
example

δ ( ) = 1, if Yj = yk else 0
ith feature
kth class

Machine Learning Department of Electrical Engineering

 Gaussian Pdf with Topographic Image

Machine Learning Department of Electrical Engineering

 GNB: Big Picture

Example: Y = Play Basket Ball (Boolean), X1 = Height

X2 = GPA

Machine Learning Department of Electrical Engineering

 Logistic Regression

Key Idea:

• Naïve Bayes allows computing P(Y) and P(Y|X) by

learning P(Y) and P(X|Y).

• Why not P(Y|X) directly?

Machine Learning Department of Electrical Engineering

 Logistic Regression

Machine Learning Department of Electrical Engineering

 Derivation of Logistic Regression

Machine Learning Department of Electrical Engineering